Delay BigQueryToMsSqlOperator source table parsing - #70493
Conversation
Template fields are rendered after operator construction, so parsing the BigQuery source table in the constructor rejects valid Jinja expressions before tasks can run.
potiuk
left a comment
There was a problem hiding this comment.
Thanks — and the test here is the strongest in this whole batch. test_execute_uses_rendered_source_project_dataset_table calls operator.render_template_fields({...}) and then execute, asserting on the resulting list_rows and BigQueryTableLink.persist arguments. That exercises the real rendering path rather than simulating it by assigning attributes, so it would catch a field being dropped from template_fields — several of the sibling PRs in this batch fake the rendering step instead. I've pointed #70530 at this as the pattern to copy.
Extracting _get_source_project_dataset_table_parts() rather than duplicating the split in two places is right, and keeping the ValueError with from None preserves the clean error message.
One question inline about the placeholder.
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
The MSSQL transfer derives source table parts from a templated project.dataset.table value at execution time, so initializing the shared base class with synthetic table parts can expose confusing internal placeholders before rendering completes.
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker e6341f3 v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
Move
BigQueryToMsSqlOperatorparsing ofsource_project_dataset_tableout of__init__so templated values are rendered before validation and dataset/table extraction.This also removes the corresponding
validate_operators_initexemption and adds regression coverage for rendered source table values.related: #70296
Was generative AI tooling used to co-author this PR?
[X]Yes (please specify the tool below)Generated-by: [Codex] following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.