Skip to content

Fix repeated deletion attempts from adopted completed pods in kubernetes executor#68360

Closed
styndall wants to merge 4 commits into
apache:mainfrom
styndall:st-kub-sync-completed-drain
Closed

Fix repeated deletion attempts from adopted completed pods in kubernetes executor#68360
styndall wants to merge 4 commits into
apache:mainfrom
styndall:st-kub-sync-completed-drain

Conversation

@styndall

@styndall styndall commented Jun 10, 2026

Copy link
Copy Markdown

This PR fixes an issue where adopted completed pods are processed over and over without ever being removed from the completed pod set. We saw thousands of duplicate pod deletion attempts after adoption events.

The fix simply removes the completed pods' state change from the main result_queue processing loop, processing them after the result_queue has been drained, and then removes them as they get processed, requeueing any failures.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code

@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jun 10, 2026
@boring-cyborg

boring-cyborg Bot commented Jun 10, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@styndall styndall force-pushed the st-kub-sync-completed-drain branch from 1ad4dfd to abfd64a Compare June 11, 2026 12:42
@styndall styndall marked this pull request as ready for review June 11, 2026 12:42

@jscheffl jscheffl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for me - but unfortunately I lag a bit of understanding of K8sExecutor. Therefore another reviewer would be good not that I miss some side-effect. But change looks reasonable

@styndall

styndall commented Jun 16, 2026

Copy link
Copy Markdown
Author

Looks good for me - but unfortunately I lag a bit of understanding of K8sExecutor. Therefore another reviewer would be good not that I miss some side-effect. But change looks reasonable

Is there someone specific I can ping to have a K8sExecutor expert's eyes on this?

@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jun 17, 2026
@styndall

Copy link
Copy Markdown
Author

@potiuk Are you able to review, or can you suggest the appropriate person?

@styndall

styndall commented Jul 1, 2026

Copy link
Copy Markdown
Author

Nevermind, fixed by this PR while I was waiting for review.

@styndall styndall closed this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants