Document effect of clearing a task instance on task state store - #71356
Conversation
There was a problem hiding this comment.
IIUC, it only makes sense when the job is in the deferred state.
For the case of wait_for_termination=True + deferrable=False (watching the external job on worker instead of triggerer), we still execute the operator.on_kill hook, which result in killing the healthy external job and unable to connect back to the prior external job instance.
So perhaps we should mention the case of wait_for_termination=True + deferrable=True in the note? Just in case users interpret having ResumableJobMixin is same as "we can always connect to the external job after clear" but it's not.
|
@jason810496 thats a good point, I pushed an update. WDYT: f62d2d99fc? |
7e45110 to
d8570db
Compare
jason810496
left a comment
There was a problem hiding this comment.
@jason810496 thats a good point, I pushed an update. WDYT: f62d2d9?
LGTM now, thanks for the update.
cf23b41 to
4b570a3
Compare
Was generative AI tooling used to co-author this PR?
Note on existing behaviour. It's this way because, we should attempt to connect to a remote job instead of submitting fresh even when cleared but its worth making a note of this.
Came up during review of #71211
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.