Skip to content

Delay BigQueryToMsSqlOperator source table parsing - #70493

Merged
potiuk merged 2 commits into
apache:mainfrom
fat-catTW:fix-bigquery-to-mssql-template-init
Aug 1, 2026
Merged

Delay BigQueryToMsSqlOperator source table parsing#70493
potiuk merged 2 commits into
apache:mainfrom
fat-catTW:fix-bigquery-to-mssql-template-init

Conversation

@fat-catTW

Copy link
Copy Markdown
Contributor

Move BigQueryToMsSqlOperator parsing of source_project_dataset_table out of __init__ so templated values are rendered before validation and dataset/table extraction.

This also removes the corresponding validate_operators_init exemption and adds regression coverage for rendered source table values.

related: #70296

Was generative AI tooling used to co-author this PR?
  • [X] Yes (please specify the tool below)

Generated-by: [Codex] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Template fields are rendered after operator construction, so parsing the BigQuery source table in the constructor rejects valid Jinja expressions before tasks can run.

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — and the test here is the strongest in this whole batch. test_execute_uses_rendered_source_project_dataset_table calls operator.render_template_fields({...}) and then execute, asserting on the resulting list_rows and BigQueryTableLink.persist arguments. That exercises the real rendering path rather than simulating it by assigning attributes, so it would catch a field being dropped from template_fields — several of the sibling PRs in this batch fake the rendering step instead. I've pointed #70530 at this as the pattern to copy.

Extracting _get_source_project_dataset_table_parts() rather than duplicating the split in two places is right, and keeping the ValueError with from None preserves the clean error message.

One question inline about the placeholder.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

The MSSQL transfer derives source table parts from a templated project.dataset.table value at execution time, so initializing the shared base class with synthetic table parts can expose confusing internal placeholders before rendering completes.
@potiuk
potiuk merged commit e6341f3 into apache:main Aug 1, 2026
86 checks passed
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker e6341f3 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers backport-to-v3-3-test Backport to v3-3-test provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants