Fix: Auto-use manifests when declared in the connector registry

[Aaron ("AJ") Steers (aaronsteers)]

Summary by CodeRabbit

• New Features

  • Introduced a cross-platform function to locate executables, enhancing system compatibility.
  • Improved flexibility in handling parameters for connector installation methods.

• Bug Fixes

  • Refined error handling for unregistered connectors to ensure users are informed of installation failures.
  • Enhanced error messages for connector registration issues, providing clearer guidance.

• Tests

  • Expanded test coverage for connector types, ensuring expected behavior for Docker-based connectors.
  • Added tests for handling non-existing connectors with local executables.

[coderabbitai]

Walkthrough

Walkthrough

The changes involve updates to several functions and error handling within the Airbyte codebase. Key modifications include adjustments to function parameters to allow for None values, improving flexibility in handling connector installation methods. The logic for counting installation methods has been refactored for clarity, and new functions have been introduced to enhance cross-platform executable path handling.

Changes

Files	Change Summary
airbyte/_executors/util.py	Updated get_connector_executor function to allow None as a default for docker_image and source_manifest. Refactored installation method counting and error handling for clarity.
airbyte/_util/meta.py	Introduced which function for cross-platform executable path resolution; updated is_docker_installed to utilize it.
airbyte/sources/registry.py	Redefined constants for clarity, updated error handling lists, and incorporated new error categories. Updated get_available_connectors to allow None as a valid install_type.
tests/integration_tests/test_source_test_fixture.py	Enhanced error handling in tests and added new test cases for non-existing connectors and Docker-only connectors.
tests/integration_tests/fixtures/registry.json	Updated source definition attributes to reflect rebranding and simplification of dependencies.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant Executor
    participant Metadata
    participant ErrorHandler

    User->>Executor: Request connector execution
    Executor->>Metadata: Retrieve connector metadata
    Metadata-->>Executor: Return metadata
    Executor->>Executor: Check installation methods
    alt Multiple methods found
        Executor->>ErrorHandler: Raise PyAirbyteInputError
    else No methods found
        Executor->>ErrorHandler: Log failure and raise exception
    end
    ErrorHandler-->>User: Provide error guidance

Loading

Would you like to delve deeper into any specific areas of the changes or perhaps add more details to the sequence diagram?

---

Recent review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 94b9cab and bfc6646.

Files selected for processing (1)

• tests/integration_tests/test_source_test_fixture.py (4 hunks)

Files skipped from review as they are similar to previous changes (1)

• tests/integration_tests/test_source_test_fixture.py

---

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

• X
• Mastodon
• Reddit
• LinkedIn

Tips

Chat

There are 3 ways to chat with CodeRabbit:

‼️ IMPORTANT
Auto-reply has been disabled for this repository in the CodeRabbit settings. The CodeRabbit bot will not respond to your replies unless it is explicitly tagged.

• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai generate unit testing code for this file.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai generate interesting stats about this repository and render them as a table.
  • @coderabbitai show all the console.log statements in this repository.
  • @coderabbitai read src/utils.ts and generate unit testing code.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai or @coderabbitai title anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[coderabbitai]

Actionable comments posted: 1

Outside diff range, codebase verification and nitpick comments (1)

airbyte/exceptions.py (1)

54-55: Enhancements to error handling and documentation links

The restructuring of DOCS_URL to use DOCS_URL_BASE simplifies maintenance and is a neat improvement. Also, the expanded guidance in AirbyteConnectorNotRegisteredError is more user-friendly. However, could we ensure that all links provided in errors are regularly checked to avoid dead links, perhaps through automated tests? Wdyt?

Also applies to: 250-255

[coderabbitai]

Actionable comments posted: 11

Outside diff range, codebase verification and nitpick comments (1)

tests/integration_tests/fixtures/registry.json (1)

Line range hint 47-80: Removal of PyPI Object

The entire "pypi" object has been removed from the "remoteRegistries" section. This indicates that the source no longer has a dependency or configuration related to PyPI. Could you elaborate on the reason behind this removal? Is it to streamline the configuration or due to other strategic reasons?

[coderabbitai]

Actionable comments posted: 2