Skip to content

Check GCSToAzureBlobStorageOperator match_glob support after template… - #70574

Open
bramhanandlingala wants to merge 3 commits into
apache:mainfrom
bramhanandlingala:fix/70296-6
Open

Check GCSToAzureBlobStorageOperator match_glob support after template…#70574
bramhanandlingala wants to merge 3 commits into
apache:mainfrom
bramhanandlingala:fix/70296-6

Conversation

@bramhanandlingala

@bramhanandlingala bramhanandlingala commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Fixes the microsoft/azure provider's GCSToAzureBlobStorageOperator entry from the #70296 exemption-list burn-down.

match_glob is a template field, but __init__ validated it against the installed apache-airflow-providers-google version and raised immediately if unsupported — so this check ran on the un-rendered Jinja expression instead of the actual rendered value.

Moved the compatibility check from __init__ into execute(), right before match_glob is used to build the GCS list call. __init__ now only does a plain assignment. _is_match_glob_supported itself stays computed in __init__, since it depends on the installed provider version, not on any template field.

  • Removed the raise from __init__; self.match_glob = match_glob is now a plain assignment.
  • Added the same check in execute(), guarded by _is_match_glob_supported.
  • Updated the existing test to construct the operator successfully and assert the error on .execute() instead of construction.
  • Removed the operator's entry from validate_operators_init_exemptions.txt.

Verified locally that scripts/ci/prek/validate_operators_init.py reports zero findings for this class after the change, and that the existing unit tests in test_gcs_to_wasb.py pass with the updated test.

Related to #70296


Gen-AI disclosure: I used a generative AI tool to help identify the root
cause, write tests, and draft the PR description. I reviewed, tested, and
verified all changes locally before submitting.

Was generative AI tooling used to co-author this PR?
  • Yes - Claude

Generated-by: Claude following the guidelines

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — same fix as #70723 does for the Amazon GCSToS3Operator, and correct for the same reason: match_glob is in template_fields, so checking it in __init__ validated the un-rendered Jinja string and turned a bad value into a Dag parse failure rather than a task failure. Keeping the _is_match_glob_supported probe in __init__ is right, since that depends on the installed Google provider rather than anything templated.

Using ValueError here is the better choice — I've suggested on #70723 that it match yours rather than the other way round, since Airflow is trying to reduce direct AirflowException raises. Worth the two of you syncing so the sibling changes land consistent.

One test nit inline, not blocking.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

container_name=CONTAINER,
match_glob="**/*.csv",
)
op.execute(context=None)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

context=None works only because the raise happens before anything touches the context. It's a bit fragile as documentation of intent — if the guard later moves below something that reads context, this fails with an unrelated AttributeError rather than the assertion you meant.

mock.MagicMock() (as #70529 uses for the same situation) or {} would be more robust. Also worth adding mock_gcs_hook.return_value.list.assert_not_called()#70723's equivalent test does that, and it's what actually proves the operator bailed out before doing any work rather than just raising somewhere.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retracting my earlier approval — my mistake, and I'm sorry for the churn.

I approved this before reading #70296, which sets out the rule for this burn-down, and by that rule this change moves the wrong kind of check.

A check that only asks whether an argument was passed … belongs in __init__ and must not be moved. Fix these by rewriting in place, not by moving. Use the is not None polarity.

The reasoning in the issue is sound and I'd missed both halves of it:

  • With render_template_as_native_obj=True, a field that was provided can render to None, so the same check in execute() reports a supplied argument as missing.
  • Raising in the constructor surfaces a static authoring mistake as a Dag import error, rather than once per task instance and per retry on a worker.

if not self._is_match_glob_supported and match_glob: asks whether match_glob was passed (combined with an environment capability that __init__ can already answer). It's a provision check, so it should be rewritten in place rather than relocated:

# in __init__ — keep it here
if not self._is_match_glob_supported and match_glob is not None:
    raise ValueError(
        "The 'match_glob' parameter requires 'apache-airflow-providers-google>=10.3.0'."
    )

Note the is not None polarity matters: if match_glob: is a truthiness test on the un-rendered Jinja string, which is a third question that matches neither intent. You already use ValueError here, which is right.

This is the twin of #70723 — worth the two of you landing the same shape.

Good news on mechanics: #70505 (which narrows the hook to allow provision checks written with is not None) merged on 28 July, so the rewrite below passes validate-operators-init and you can still remove the exemption-file entry in this PR — the burn-down goal is unaffected.

There's an active follow-up, #70503, cataloguing already-merged PRs that made exactly this move so they can be put back. Fixing it here saves this PR from joining that list.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@bramhanandlingala

Copy link
Copy Markdown
Contributor Author

@potiuk Thanks for catching this — restored the check to init using
match_glob is not None, removed the duplicate from execute(), and
updated the test to assert the raise on construction instead of execute().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:providers backport-to-v3-3-test Backport to v3-3-test provider:microsoft-azure Azure-related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants