chore: Remove flaky lowcode connector validation tests#732
Merged
Aaron ("AJ") Steers (aaronsteers) merged 3 commits intoJul 31, 2025
Merged
Conversation
The test_nocode_connectors_setup and test_expected_hardcoded_failures tests were causing unnecessary CI flakiness due to CDK version compatibility issues. These tests were not providing reliable value and were eternally-flaky. Removed: - test_nocode_connectors_setup function - test_expected_hardcoded_failures function - Hardcoded failure lists (_LOWCODE_CONNECTORS_FAILING_VALIDATION, etc.) - Associated imports and references This cleanup makes the test suite more reliable by removing tests that don't provide consistent value. Co-Authored-By: AJ Steers <aj@airbyte.io>
Co-Authored-By: AJ Steers <aj@airbyte.io>
Contributor
Original prompt from AJ Steers |
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
👋 Greetings, Airbyte Team Member!Here are some helpful tips and reminders for your convenience. Testing This PyAirbyte VersionYou can test this version of PyAirbyte using the following: # Run PyAirbyte CLI from this branch:
uvx --from 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1753934781-remove-flaky-lowcode-test' pyairbyte --help
# Install PyAirbyte from this branch for development:
pip install 'git+https://github.com/airbytehq/PyAirbyte.git@devin/1753934781-remove-flaky-lowcode-test'Helpful ResourcesPR Slash CommandsAirbyte Maintainers can execute the following slash commands on your PR:
Community SupportQuestions? Join the #pyairbyte channel in our Slack workspace. |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR removes flaky integration tests for low-code connector validation that were causing CI failures due to CDK version compatibility issues. The tests were deemed "eternally-flaky" and providing unreliable value.
- Removed two test functions that validated connector initialization and expected failures
- Eliminated hardcoded failure lists that tracked problematic connectors
- Updated connector filtering logic to include all YAML connectors without exclusions
Reviewed Changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tests/integration_tests/test_lowcode_connectors.py | Removed flaky test functions and cleaned up unused imports |
| airbyte/sources/registry.py | Removed hardcoded failure lists and exclusion filtering from connector retrieval |
Contributor
📝 WalkthroughWalkthroughThis change removes exclusion lists and related filtering logic for low-code connectors in Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
💤 Files with no reviewable changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (6)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
PyTest Results (Fast Tests Only, No Creds)256 tests - 473 256 ✅ - 473 3m 28s ⏱️ - 14m 15s Results for commit 99caa5a. ± Comparison against base commit 1266018. This pull request removes 473 tests.♻️ This comment has been updated with latest results. |
PyTest Results (Full)318 tests - 473 304 ✅ - 473 17m 28s ⏱️ - 12m 6s Results for commit 99caa5a. ± Comparison against base commit 1266018. This pull request removes 473 tests. |
Aaron ("AJ") Steers (aaronsteers)
deleted the
devin/1753934781-remove-flaky-lowcode-test
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Remove flaky lowcode connector validation tests
Summary
This PR removes two flaky test functions (
test_nocode_connectors_setupandtest_expected_hardcoded_failures) from the PyAirbyte integration test suite, along with their associated hardcoded failure lists. These tests were causing unnecessary CI failures due to CDK version compatibility issues (specifically with source-airtable ValidationError) and were determined to be "eternally-flaky" without providing reliable value.Key Changes:
test_nocode_connectors_setupfunction that was failing with ValidationError for source-airtabletest_expected_hardcoded_failuresfunction that validated hardcoded failure lists_LOWCODE_CONNECTORS_FAILING_VALIDATION,_LOWCODE_CONNECTORS_UNEXPECTED_ERRORS,_LOWCODE_CDK_FILE_NOT_FOUND_ERRORS, and_LOWCODE_CONNECTORS_EXCLUDEDget_available_connectors()function to remove filtering by the deleted exclusion listReview & Testing Checklist for Human
get_available_connectors(install_type="yaml")still returns expected connectors - The exclusion filtering was removed, so all YAML connectors should now be returnedDiagram
%%{ init : { "theme" : "default" }}%% graph TD TestFile["tests/integration_tests/<br/>test_lowcode_connectors.py"]:::major-edit Registry["airbyte/sources/<br/>registry.py"]:::major-edit GetSource["get_source()"]:::context AvailableConnectors["get_available_connectors()"]:::context TestFile -->|"used hardcoded lists for<br/>validation testing"| Registry Registry -->|"provides connector<br/>metadata"| AvailableConnectors TestFile -->|"tested connector<br/>initialization"| GetSource subgraph "Removed Components" RemovedTests["test_nocode_connectors_setup<br/>test_expected_hardcoded_failures"]:::major-edit RemovedLists["_LOWCODE_CONNECTORS_FAILING_VALIDATION<br/>_LOWCODE_CONNECTORS_UNEXPECTED_ERRORS<br/>_LOWCODE_CDK_FILE_NOT_FOUND_ERRORS<br/>_LOWCODE_CONNECTORS_EXCLUDED"]:::major-edit end subgraph Legend L1["Major Edit"]:::major-edit L2["Minor Edit"]:::minor-edit L3["Context/No Edit"]:::context end classDef major-edit fill:#90EE90 classDef minor-edit fill:#87CEEB classDef context fill:#FFFFFFNotes
config_normalization_rulesthat wasn't supported by PyAirbyte's current CDK version (6.56.5)Link to Devin run: https://app.devin.ai/sessions/56f5fa20d3794c3c833114086785608c
Requested by: Aaron ("AJ") Steers (@aaronsteers) (AJ Steers)
Summary by CodeRabbit
Refactor
Tests
Important
Auto-merge enabled.
This PR is set to merge automatically when all requirements are met.