Skip to content

Prevent deferrable KubernetesPodOperator log parsing from blocking the triggerer event loop - #69661

Merged
potiuk merged 1 commit into
apache:mainfrom
aeroyorch:fix-kpo-trigger-loop-blocking
Jul 31, 2026
Merged

Prevent deferrable KubernetesPodOperator log parsing from blocking the triggerer event loop#69661
potiuk merged 1 commit into
apache:mainfrom
aeroyorch:fix-kpo-trigger-loop-blocking

Conversation

@aeroyorch

@aeroyorch aeroyorch commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Prevent deferrable KubernetesPodOperator log parsing from blocking the triggerer event loop when default_deferrable is True with get_logs and logging_interval set, which stales on verbose pods.
Offload that work to a thread so the loop stays responsive.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Claude Opus 4.8 for tests.


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jul 9, 2026
@aeroyorch
aeroyorch force-pushed the fix-kpo-trigger-loop-blocking branch from d537638 to 324277a Compare July 9, 2026 15:34
@aeroyorch

Copy link
Copy Markdown
Contributor Author

I believe the CI errors are unrelated to the current changes.

@aeroyorch
aeroyorch force-pushed the fix-kpo-trigger-loop-blocking branch from 324277a to d446fff Compare July 9, 2026 18:04
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Jul 11, 2026
@aeroyorch
aeroyorch force-pushed the fix-kpo-trigger-loop-blocking branch from d446fff to f8644d6 Compare July 22, 2026 21:23
@aeroyorch
aeroyorch force-pushed the fix-kpo-trigger-loop-blocking branch from f8644d6 to e4568ea Compare July 25, 2026 15:25

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — this is the right concern. The triggerer multiplexes many triggers onto a single event loop, so any CPU-bound work in an async def stalls every other trigger, not just this one. A large decode() + splitlines(), and the per-line parse/emit loop, are both real candidates.

The restructuring is clean: _split_log_bytes and _emit_container_logs extracted so they can be handed to to_thread, with return now correctly relocated ahead of the extraction so the function's contract is unchanged.

I checked the thing that usually bites when you move logging into a worker thread — contextvars. asyncio.to_thread copies the current context (contextvars.copy_context()), so the self.log.log(...) calls inside _emit_container_logs keep whatever task/dag context is bound. No log-attribution loss.

One question worth a thought rather than a change: to_thread isn't free — each call dispatches to the executor and round-trips through the loop. For a pod emitting a handful of log lines per poll, that overhead may exceed the decode it's avoiding, and the triggerer polls frequently across many pods. Did you look at whether it's worth gating on payload size (offload only above some threshold), or is the constant cost small enough not to matter at the polling rates involved? Not blocking — just that "always offload" optimises the large-log case at some cost to the common small-log one.


Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting

@potiuk
potiuk merged commit a6699a4 into apache:main Jul 31, 2026
108 checks passed
@aeroyorch
aeroyorch deleted the fix-kpo-trigger-loop-blocking branch July 31, 2026 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants