Skip to content

Normalize BigQuery DTS sensor expected statuses after rendering - #70528

Open
mitre88 wants to merge 1 commit into
apache:mainfrom
mitre88:fix-bigquery-dts-sensor-template
Open

Normalize BigQuery DTS sensor expected statuses after rendering#70528
mitre88 wants to merge 1 commit into
apache:mainfrom
mitre88:fix-bigquery-dts-sensor-template

Conversation

@mitre88

@mitre88 mitre88 commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Part of the template-field validation burn-down tracked in #70296.

BigQueryDataTransferServiceTransferRunSensor lists expected_statuses in template_fields but converts it to TransferState members in __init__ via _normalize_state_list, so passing a Jinja expression raised KeyError at parse time. The sensor now stores the raw value and normalizes it in poke, after rendering.

Added a test constructing the sensor with a templated expected_statuses that passes once the field holds the rendered value — it fails against the previous implementation. The class is removed from the exemption list and the validate-operators-init check passes locally.

Per the discussion in #70505 this is a genuine value transformation (not an argument-provision check), so it belongs at execute time.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Fable 5)

Generated-by: Claude Code (Fable 5) following the guidelines

@boring-cyborg boring-cyborg Bot added area:dev-tools area:providers backport-to-v3-3-test Backport to v3-3-test provider:google Google (including GCP) related issues labels Jul 27, 2026
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 28, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — correct, and this one is the clearest case of the pattern. expected_statuses is in template_fields, so normalising it in __init__ ran against the raw "{{ ... }}" string: _normalize_state_list would try TransferState["{{ VAR.VALUE.EXPECTED_STATUS }}"] and blow up at parse time before the value ever existed. Deferring to poke is the only way this can work.

A side benefit worth noting: self.expected_statuses now holds the rendered value rather than a set[TransferState], which makes the rendered-template view in the UI show what the user actually wrote instead of an enum set.

One observation inline about what a bad value now looks like.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

message = f"Transfer {self.run_id} did not succeed"
raise AirflowException(message)
return run.state in self.expected_statuses
return run.state in self._normalize_state_list(self.expected_statuses)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small consequences of moving the normalisation here, neither blocking:

Error quality. _normalize_state_list does TransferState[state.upper()], so a typo'd status now surfaces as a bare KeyError: 'SUCEEDED' from inside the sensor on every poke, rather than at parse time where the traceback at least pointed at the Dag file. That was survivable when it failed fast at parse; as a runtime failure it's worth wrapping in a ValueError naming the offending value and the valid ones. Arguably out of scope here, but this PR is what makes it user-visible.

Repeated work. Normalisation now runs on every poke rather than once. It's cheap and a sensor is I/O-bound, so this is genuinely negligible — mentioning it only so it's a conscious choice rather than an oversight. If you'd rather avoid it, normalising once into a cached attribute on first poke would do it.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Rebased onto current main. Validate expected_statuses after template
rendering so templated values are checked at runtime.
@mitre88
mitre88 force-pushed the fix-bigquery-dts-sensor-template branch from 8637f8f to dc4e9b5 Compare August 1, 2026 02:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers backport-to-v3-3-test Backport to v3-3-test provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants