Resolve GCSFileTransformOperator destination fallbacks after rendering - #70488
Resolve GCSFileTransformOperator destination fallbacks after rendering#70488mitre88 wants to merge 1 commit into
Conversation
boring-cyborg
Bot
added
area:dev-tools
area:providers
backport-to-v3-3-test
| if objects is None and prefix is None: | ||
| err_message = "(Task {task_id}) Either objects or prefix should be set. Both are None.".format( | ||
| **kwargs | ||
| ) | ||
| raise ValueError(err_message) | ||
| if objects is not None and prefix is not None: | ||
| err_message = "(Task {task_id}) Objects or prefix should be set. Both provided.".format(**kwargs) | ||
| raise ValueError(err_message) | ||
|
|
||
| super().__init__(**kwargs) | ||
|
|
There was a problem hiding this comment.
Please restore the changes here, as it will be flagged as false-positive (see ##70505 for more details)
| raise ValueError(f"(Task {self.task_id}) Either objects or prefix should be set. Both are None.") | ||
| if self.objects is not None and self.prefix is not None: | ||
| raise ValueError(f"(Task {self.task_id}) Objects or prefix should be set. Both provided.") | ||
|
|
There was a problem hiding this comment.
Please revert these changes
| providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocCreateClusterOperator | ||
| providers/google/src/airflow/providers/google/cloud/operators/dataproc.py::DataprocSubmitJobOperator | ||
| providers/google/src/airflow/providers/google/cloud/operators/functions.py::CloudFunctionDeployFunctionOperator | ||
| providers/google/src/airflow/providers/google/cloud/operators/gcs.py::GCSDeleteObjectsOperator |
There was a problem hiding this comment.
Please restore this line
mitre88
changed the title
|
Done — reverted the Drafted-by: Claude Code (Fable 5) (no human review before posting) |
destination_bucket/destination_object are template fields, so resolving their 'or source_*' fallbacks in __init__ reads the unrendered value. Move the fallback resolution to execute() where the rendered values are available, keep get_openlineage_facets_on_complete resolving fallbacks locally (it can run outside execute), and remove the operator from the validate_operators_init exemptions list. Related: apache#70296
6963c2d to
9784313
Compare
|
@shahar1 Done — the PR is now scoped to To confirm the remaining change is consistent with the #70505 distinction:
Everything else from the earlier revision ( Drafted-by: Claude Code (Fable 5) (no human review before posting) |
3 participants
Part of the template-field validation burn-down tracked in #70296.
GCSFileTransformOperatorresolvesdestination_bucket/destination_objectfallbacks (destination_bucket or source_bucket) in__init__, where the source fields still hold un-rendered Jinja expressions. This moves the fallback resolution toexecute()(and mirrors it inget_openlineage_facets_on_start, which runs beforeexecute), so the fallback applies to the rendered values.Scope notes from review:
delimiterdeprecation warning inGCSListObjectsOperatorstays in__init__— deprecation applies to any usage of the argument, not the rendered value.objects/prefixprovision checks inGCSDeleteObjectsOperatorstay in__init__— they are false positives under the relaxed validator definition proposed in Allow argument-provision checks of templated fields in operator __init__ #70505. Both classes keep their exemption entries; only theGCSFileTransformOperatorentry is removed.Note:
TestGCSDeleteObjectsOperator::test_get_openlineage_facets_on_start[objects]fails on a pristinemaincheckout as well — pre-existing and unrelated to this change.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines