Skip to content

Document effect of clearing a task instance on task state store - #71356

Merged
amoghrajesh merged 4 commits into
apache:mainfrom
astronomer:document-clear-behaviour-resumable-tasks
Aug 12, 2026
Merged

Document effect of clearing a task instance on task state store#71356
amoghrajesh merged 4 commits into
apache:mainfrom
astronomer:document-clear-behaviour-resumable-tasks

Conversation

@amoghrajesh

Copy link
Copy Markdown
Contributor

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Note on existing behaviour. It's this way because, we should attempt to connect to a remote job instead of submitting fresh even when cleared but its worth making a note of this.

Came up during review of #71211


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

Comment thread airflow-core/docs/core-concepts/resumable-tasks.rst Outdated
@amoghrajesh
amoghrajesh requested a review from Lee-W August 10, 2026 07:28
Comment thread airflow-core/docs/core-concepts/resumable-tasks.rst Outdated

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, it only makes sense when the job is in the deferred state.
For the case of wait_for_termination=True + deferrable=False (watching the external job on worker instead of triggerer), we still execute the operator.on_kill hook, which result in killing the healthy external job and unable to connect back to the prior external job instance.

So perhaps we should mention the case of wait_for_termination=True + deferrable=True in the note? Just in case users interpret having ResumableJobMixin is same as "we can always connect to the external job after clear" but it's not.

@amoghrajesh

Copy link
Copy Markdown
Contributor Author

@jason810496 thats a good point, I pushed an update. WDYT: f62d2d99fc?

@amoghrajesh
amoghrajesh force-pushed the document-clear-behaviour-resumable-tasks branch from 7e45110 to d8570db Compare August 11, 2026 09:43

@jason810496 jason810496 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jason810496 thats a good point, I pushed an update. WDYT: f62d2d9?

LGTM now, thanks for the update.

Comment thread airflow-core/docs/core-concepts/resumable-tasks.rst
@amoghrajesh
amoghrajesh force-pushed the document-clear-behaviour-resumable-tasks branch from cf23b41 to 4b570a3 Compare August 12, 2026 02:52
@amoghrajesh
amoghrajesh merged commit 1481de6 into apache:main Aug 12, 2026
68 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Done in Durable / Crash-Safe Execution Aug 12, 2026
@amoghrajesh
amoghrajesh deleted the document-clear-behaviour-resumable-tasks branch August 12, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Development

Successfully merging this pull request may close these issues.

4 participants