Skip to content

Fix PsrpOperator option checks and drop the cmdlet task_id default - #70347

Open
1fanwang wants to merge 11 commits into
apache:mainfrom
1fanwang:fix-psrp-init
Open

Fix PsrpOperator option checks and drop the cmdlet task_id default#70347
1fanwang wants to merge 11 commits into
apache:mainfrom
1fanwang:fix-psrp-init

Conversation

@1fanwang

@1fanwang 1fanwang commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

command, powershell and cmdlet are mutually exclusive, and arguments / parameters are only valid alongside the latter two. All five are template fields, and every check was written as a truthiness test:

  • exactly_one was handed a set literal, so passing the same value to two options deduped to one and passed validation.
  • A provided-but-empty option (command="", arguments=[]) read as absent, so valid combinations were rejected and invalid ones accepted.
  • execute() dispatched on truthiness too, so a command that renders to "" fell through and ran as powershell.

The checks stay in __init__. Per #70505 the constructor is the only place that can answer "was this argument passed?", so they are rewritten with the is None polarity the docs now prescribe. The example added there is this operator's guard. Only the dispatch in execute() moves to is not None, where the rendered value is finally known.

Breaking change, flagged for maintainers: this drops the cmdlet-derived task_id default, the last value read in the constructor and the only thing still holding PsrpOperator on the exemption list (log 4 below). task_id is identity, so it cannot move to execute(); pass it explicitly when using cmdlet. Say the word and I'll revert that part and restore the entry.

related: #70296, #70505

Testing Done
# Scenario Result
1 New/changed cases against the pre-fix operator 6 fail, 3 pass (the 3 are the pre-existing guards this must not regress)
2 Full operator suite after the fix 41 passed
3 validate-operators-init on the operator, exemption entry removed exit 0
4 Same hook with the task_id default restored fails on the two cmdlet lines — the entry cannot be cleared while it stays

Raw logs

  1. Red — new tests run against upstream/main's psrp.py:
$ git show upstream/main:providers/.../operators/psrp.py > providers/.../operators/psrp.py
$ uv run --project providers/microsoft/psrp pytest providers/microsoft/psrp/tests/unit/microsoft/psrp/operators/test_psrp.py \
    -q -k "invalid_option_combination or empty_option_counts or dispatches_as_command"
...
E   AssertionError: expected call not found.
E   Expected: add_script("cmd.exe /c @'\n\n'@")
E   Actual: add_script(None)
=========================== short test summary info ============================
FAILED ...::test_invalid_option_combination[two-options-one-empty]
FAILED ...::test_invalid_option_combination[two-equal-options]
FAILED ...::test_empty_option_counts_as_provided[empty-command]
FAILED ...::test_empty_option_counts_as_provided[arguments]
FAILED ...::test_empty_option_counts_as_provided[parameters]
FAILED ...::test_command_rendering_to_empty_dispatches_as_command
============ 6 failed, 3 passed, 32 deselected, 1 warning in 3.14s =============
  1. Green — full suite on the fixed operator:
$ uv run --project providers/microsoft/psrp pytest providers/microsoft/psrp/tests/unit/microsoft/psrp/operators/test_psrp.py -q
======================== 41 passed, 1 warning in 3.63s =========================
  1. Hook green with the exemption entry removed:
$ prek run validate-operators-init --files providers/microsoft/psrp/src/airflow/providers/microsoft/psrp/operators/psrp.py
Running hooks for `providers`:
Validate templated fields in operator __init__...........................Passed
exit=0
  1. The same hook with if cmdlet: kwargs.setdefault("task_id", cmdlet) put back, showing what the removal buys:
Validate templated fields in operator __init__...........................Failed
- hook id: validate-operators-init
- exit code: 1

  microsoft/psrp/src/airflow/providers/microsoft/psrp/operators/psrp.py
  PsrpOperator's constructor applies logic to template fields. ...
    line 115: if cmdlet:  (cmdlet)
    line 116: kwargs.setdefault("task_id", cmdlet)  (cmdlet)

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: GitHub Copilot CLI following the guidelines

command, powershell, cmdlet, arguments and parameters are template fields,
rendered after __init__ runs. The constructor validated their combination and
derived task_id from cmdlet, all reading the un-rendered Jinja expressions. Move
the validation into execute(), which runs after rendering.

This drops the cmdlet-derived task_id default (a construction-time read of a
template field that cannot move): task_id must now be passed explicitly when
using cmdlet.

related: apache#70296
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang marked this pull request as draft July 24, 2026 05:47
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang
1fanwang marked this pull request as ready for review July 24, 2026 06:23
@shahar1 shahar1 removed the backport-to-v3-3-test Backport to v3-3-test label Jul 24, 2026
1fanwang added 3 commits July 24, 2026 11:32
Signed-off-by: 1fanwang <1fannnw@gmail.com>

# Conflicts:
#	scripts/ci/prek/validate_operators_init_exemptions.txt
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>
command is a template field, so validating it in __init__ checked the raw
Jinja expression, not the rendered value. Add a test that renders a command
resolving to an empty string, then executes: it fails on the pre-fix source
(the empty command slips past __init__ and reaches execution) and passes with
validation in execute().

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Comment thread providers/microsoft/psrp/docs/changelog.rst Outdated
Comment thread providers/microsoft/psrp/src/airflow/providers/microsoft/psrp/operators/psrp.py Outdated
exactly_one deduped equal values before counting them, so passing the same value to two of command/powershell/cmdlet passed validation. Pass the options positionally so each is counted.

Drop the manual changelog note; the provider release manager regenerates the changelog from git log.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
The mutual-exclusivity and arguments/parameters checks are about which options the Dag author provided, not what the templates render to. Keying them off rendered truthiness dropped a provided option that rendered to a falsy value, and let a second option slip through when the first rendered empty. Check is-not-None (usage) instead, and dispatch the chosen option consistently.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Only the constructor can tell whether an option was passed: with
render_template_as_native_obj a provided field can render to None, so the same
check in execute() reports a supplied argument as missing. Compare against None
rather than truthiness, because a provided option can itself be empty.

related: apache#70505
Signed-off-by: 1fanwang <1fannnw@gmail.com>
@1fanwang 1fanwang changed the title Validate PsrpOperator parameters after rendering Fix PsrpOperator option checks and drop the cmdlet task_id default Jul 29, 2026
Signed-off-by: 1fanwang <1fannnw@gmail.com>

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — the core fix is correct and worth having. args = {command, powershell, cmdlet} built a set, so equal values collapsed into one entry, and exactly_one then counted truthiness rather than presence — meaning an empty-string command read as "not provided". Switching to explicit is not None checks fixes both problems, and carrying the same correction into execute() keeps it consistent.

One blocking concern: this PR also removes kwargs.setdefault("task_id", cmdlet), so a Dag that relied on the cmdlet supplying the task_id now gets a different task_id. That changes task identity — history, logs, XComs and UI links all key off it — so it isn't a refactor, it's a breaking change for those users.

Please either:

  1. split the task_id removal into its own PR so the (uncontroversial) validation fix can land immediately, or
  2. keep it here but add a newsfragment in airflow-core/newsfragments/ explaining that PsrpOperator no longer defaults task_id to cmdlet and what users must do — plus a line in the provider changelog.

For context: #70656 makes the same validation fix without the task_id change. It was opened six days after this one, so I'm closing it in favour of this PR — which means the validation fix now depends on this one moving. Option 1 would unblock it fastest.

These are judgement calls rather than mechanical fixes — I'd welcome your own reasoning in reply. My review was AI-assisted and shouldn't be treated as settled.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

:param cmdlet:
cmdlet to execute on remote host (templated). Also used as the default
value for `task_id`.
cmdlet to execute on remote host (templated).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This docstring edit is the visible half of the behaviour change — dropping "Also used as the default value for task_id" alongside removing the setdefault.

Documenting it here is right, but a docstring is not where users find out that their task_ids changed. That needs the newsfragment/changelog entry described in the review body.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants