Build DataprocCreateClusterOperator cluster_config from legacy kwargs after rendering - #70922
Build DataprocCreateClusterOperator cluster_config from legacy kwargs after rendering#70922bujjibabukatta wants to merge 2 commits into
Conversation
boring-cyborg
Bot
added
area:dev-tools
area:providers
backport-to-v3-3-test
| @mock.patch(DATAPROC_PATH.format("DataprocHook")) | ||
| def test_deprecated_kwargs_cluster_config_built_in_execute(self, mock_hook, to_dict_mock): | ||
| mock_hook.return_value.create_cluster.result.return_value = None | ||
| op = DataprocCreateClusterOperator( |
There was a problem hiding this comment.
I think you need to wrap the operator construction like this in both tests to fix failing CI:
with pytest.warns(AirflowProviderDeprecationWarning):
op = DataprocCreateClusterOperator(...)
| mock_hook.return_value.create_cluster.result.return_value = None | ||
| op = DataprocCreateClusterOperator( | ||
| task_id=TASK_ID, region=GCP_REGION, project_id=GCP_PROJECT, | ||
| cluster_name="cluster_name", num_workers=2, zone="zone", |
There was a problem hiding this comment.
The formatting of these arguments should be fixed by the linter.
61df2d0 to
724860b
Compare
82398f0 to
e12ee19
Compare
|
Hi @SameerMesiah97 thanks for catching both — fixed. Wrapped the operator Separately: Static checks keeps failing on validate-operators-init, flagging |
| **kwargs, | ||
| ) -> None: | ||
| # TODO: remove one day | ||
| self._legacy_cluster_kwargs: dict | None = None |
There was a problem hiding this comment.
The kwargs functionality is deprecated and scheduled for deletion, so what the point bringing them into templated fields, the original issue #70296 states
Template fields are rendered after an operator's constructor runs. Any logic applied to a template-field parameter's value inside init — validation, type checks, transformation, string interpolation — therefore operates on the un-rendered Jinja expression, not the real value.
So whats the point to bringing them into templating fields?
There was a problem hiding this comment.
@olegkachur-e Good point. They're deprecated but still around, so they should still work right until they're actually removed. Right now they don't — cluster_config gets built in init before rendering happens, so any templated value just doesn't get filled in. This fix takes care of that in the meantime.
If you'd rather I just remove the kwargs path instead of fixing it, I'm okay with that too.
3 participants
Summary
DataprocCreateClusterOperator's deprecated keyword-based cluster
construction built cluster_config in init, using individually
templated kwargs (zone, cluster_name, project_id, ...) before rendering.
Root cause
ClusterGenerator(**kwargs).make() ran in init. These kwargs aren't
declared as template fields, so Jinja expressions passed this way never
got rendered before being baked into cluster_config.
Fix
Stash the raw kwargs in a new template field (_legacy_cluster_kwargs)
instead of building cluster_config immediately. Being a template field, it
survives DAG serialization and gets rendered automatically. cluster_config
is now built in execute(), after rendering.
closes: #70296
Was generative AI tooling used ?
Generated-by: Claude following the guidelines