Add configurable request timeout for the Airbyte API - #71092
Open
filipeaaoliveira wants to merge 2 commits into
Open
Add configurable request timeout for the Airbyte API#71092filipeaaoliveira wants to merge 2 commits into
filipeaaoliveira wants to merge 2 commits into
Conversation
Since apache-airflow-providers-airbyte 6.0.0 the provider uses airbyte-api 1.x, which switched from requests to httpx and applies a 5-second default request timeout. On self-hosted Airbyte deployments job creation can take far longer than that under load, so every trigger fails - and because the create-job call is not idempotent, the timed-out request still creates a job server-side, causing 409 errors and duplicate syncs on retry. Allow users to raise the timeout via a hook parameter or a connection extra.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hello everyone.
While upgrading Airflow to 3.3.0 yesterday, our Airbyte DAGs started failing on every trigger even though the syncs themselves were starting in Airbyte.
The cause is the provider's move to airbyte-api 1.x, which replaced requests with httpx and with it picked up httpx's 5-second default request timeout. Provider 6.0.0 requires the 1.x SDK, but 5.5.1 hits it too: its airbyte-api>=0.52.0 pin has no upper bound (#69081 added the <1.0.0 cap after 5.5.1 shipped), and constraints-3.3.0 resolves it to airbyte-api==1.0.1.
On self-hosted deployments, job creation can take far longer than 5 seconds under load — we measured POST /v1/jobs at 30–130 seconds with 9 connections triggered in parallel. Every trigger task then fails with
httpx.ReadTimeoutraised fromAirbyteHook.submit_sync_connection. And because create-job is not idempotent, the timed-out request still creates the job server-side, so retries get 409 "A sync is already running" and the attempt that eventually succeeds starts a duplicate sync.The hook currently exposes no way to change the timeout. This PR lets users set it through an
AirbyteHookparameter or a connection extra, passing it to the SDK's existingtimeout_ms. With neither set, the session is built exactly as before, so default behavior is unchanged.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code - Fable 5 following the guidelines