Add XCom read access to callback supervisor comms channel#66611
Add XCom read access to callback supervisor comms channel#66611seanghaeli wants to merge 7 commits into
Conversation
|
What is the plan for plumbing the dag_id, run_id, etc through here? You don't seem to have addressed thr "motivation" part of your description. The reason I didn't include this one in the initial batch of comms channels was because GetXCom requires those and I didn't want users to have to manually provide those like |
|
Good question. The plan is for PR #66608 to provide the context plumbing — once a callback has access to So this PR gives the mechanism (XCom comms channel), and #66608 gives the identifiers. Once #66608 lands, we can add a convenience wrapper that pre-fills |
90f3323 to
91e5a68
Compare
4a33ab3 to
454244a
Compare
cfdf5dc to
94b8c53
Compare
94b8c53 to
cccbd89
Compare
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
85d8671 to
4b1a928
Compare
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
9873c1b to
7e7248f
Compare
ferruzzi
left a comment
There was a problem hiding this comment.
In #66608, you and Kaxil discussed and changed the scope of token:workload and this looks like it might need to be re-applied here? I'm not sure have a look and make sure the security concerns you two discussed over there aren't being undone here. I think it'll inherit the fix from over there, just make sure when you rebase onto that one that you test it.
That said, approved with the caveat of the follow-ups that were discussed and the fact that the CI is currently failing in main.
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
328513d to
4b4810d
Compare
apache#66611 makes token:workload effective on the connections read route (route_class=ExecutionAPIRoute), so deadline callback subprocesses can read connections. Update test_workload_scope_rejected_on_connections_endpoint -> test_workload_scope_accepted_on_connections_endpoint: a workload token now passes the token-type check and reaches the route (404 for a missing connection) rather than being rejected with 403. Verified locally: 1 passed.
This PR's callback comms only use GetDagRun/GetConnection/GetVariable, never GetXCom, so the token:workload scope changes on the XCom execution-API route (and their TestWorkloadTokenScopeEnforcement tests) had no consumer here — they are apache#66611's deliverable (callback XCom read access) and were riding along. Revert xcoms.py and test_xcoms.py to main so this PR stays scoped to callback context fetching; apache#66611 owns the XCom route + comms changes.
apache#66611 makes token:workload effective on the connections read route (route_class=ExecutionAPIRoute), so deadline callback subprocesses can read connections. Update test_workload_scope_rejected_on_connections_endpoint -> test_workload_scope_accepted_on_connections_endpoint: a workload token now passes the token-type check and reaches the route (404 for a missing connection) rather than being rejected with 403. Verified locally: 1 passed.
caa3ea7 to
f6b5e2c
Compare
Accept workload tokens on the get_xcom, connections, and variables execution API routes so a deadline callback subprocess can read XCom values (and connections/variables) via the supervisor comms channel, mirroring task access.
Callback supervisors carry a workload-scoped JWT, but the XCom read routes
only accepted execution-scoped tokens, so a callback's XCom read got 403
("Token type 'workload' not allowed"). Mirror the variables/connections
routes: set route_class=ExecutionAPIRoute on the router (so per-route
token:* scopes are honored) and add token:workload to the GET item/slice
and HEAD routes alongside the existing get_xcom route. POST/DELETE stay
execution-only.
…rs is pre-existing main breakage in amazon/login.py, unrelated)
(route_class=ExecutionAPIRoute), so deadline callback subprocesses can read connections. Update test_workload_scope_rejected_on_connections_endpoint -> test_workload_scope_accepted_on_connections_endpoint: a workload token now passes the token-type check and reaches the route (404 for a missing connection) rather than being rejected with 403. Verified locally: 1 passed.
f6b5e2c to
906c102
Compare
Conflict resolution when rebasing onto apache#66608 unioned two import lists and left BundleInfo imported but unused. Drop it to satisfy ruff.
|
Parking this as a draft pending a security-model decision raised on #66608. |
Add read-only XCom access to the callback supervisor IPC channel, allowing
callbacks to fetch XCom values from upstream tasks via the same mechanism
used for Connections and Variables.
Motivation
PR #65269 added Connection, Variable, and MaskSecret comms channels to the
callback supervisor. XCom was deferred because it requires explicit dag_id,
run_id, and task_id (callbacks have no implicit task context). This PR adds
that missing piece.
Changes
GetXComto theCallbackToSupervisorunion typehandle_get_xcomshared handler inrequest_handlers.pyGetXCommessages inCallbackSubprocess._handle_requestOnly read access (GetXCom) is exposed. SetXCom and DeleteXCom remain out of
scope for callbacks since they should not be producing XCom values.
related: #65269
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.6) following the guidelines