Skip to content

Fix DbtCloudRunJobOperator having false failures during deferred polling - #70581

Merged
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:worktree-fix-dbtcloud-async-connection-70398
Jul 30, 2026
Merged

Fix DbtCloudRunJobOperator having false failures during deferred polling#70581
amoghrajesh merged 2 commits into
apache:mainfrom
astronomer:worktree-fix-dbtcloud-async-connection-70398

Conversation

@amoghrajesh

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes

closes: #70398

Why?

Deferred DbtCloudRunJobOperator tasks failed within seconds of deferring with a misleading DbtCloudJobRunException: Job run <id> has failed, even though the dbt Cloud job kept running and completed successfully hours later.

What

DbtCloudHook.get_headers_tenants_from_connection() and get_job_details() resolved the connection through the hook's synchronous connection cached_property while running inside the triggerer's live event loop. That
path masks the connection's secret via a synchronous comms call, which asgiref rejects when called from a thread already running an event loop the resulting RuntimeError propagated up through the trigger and was surfaced by the operator as a job failure instead of the real (transient) error it was.

Both call sites now resolve the connection through the existing async safe get_async_connection() helper, sharing a cache slot with the sync connection cached_property so neither path performs a duplicate connection lookup. Also switched the file's Connection type import to the airflow.providers.common.compat.sdk compat shim (matching what
get_connection()/get_async_connection() actually return under Airflow 3), which let a stale # type: ignore[return-value] on the connection property be removed.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread providers/dbt/cloud/src/airflow/providers/dbt/cloud/hooks/dbt.py
@amoghrajesh
amoghrajesh requested a review from Lee-W July 29, 2026 12:09
@amoghrajesh
amoghrajesh merged commit 13d7df7 into apache:main Jul 30, 2026
78 checks passed
@amoghrajesh
amoghrajesh deleted the worktree-fix-dbtcloud-async-connection-70398 branch July 30, 2026 04:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred DbtCloudRunJobOperator (and similar triggers) fail spuriously via mask_secret() cross-event-loop RuntimeError in _get_connection()

3 participants