Skip to content

Add streaming task log support to KubernetesExecutor#69300

Merged
jason810496 merged 5 commits into
apache:mainfrom
jason810496:refactor/logging/kubernetes-executor-streaming-logs
Jul 22, 2026
Merged

Add streaming task log support to KubernetesExecutor#69300
jason810496 merged 5 commits into
apache:mainfrom
jason810496:refactor/logging/kubernetes-executor-streaming-logs

Conversation

@jason810496

@jason810496 jason810496 commented Jul 3, 2026

Copy link
Copy Markdown
Member

part of the streaming task log series

Why

KubernetesExecutor.get_task_log materializes the whole pod log in the API server. Implementing the new BaseExecutor.get_streaming_task_log yields lines lazily into the bounded LogStreamAccumulator: ~11.6x lower peak heap growth (+2093.9 MiB vs +179.9 MiB) serving a ~415 MB running-task log (full benchmark in #69299).

What

  • Implement get_streaming_task_log on KubernetesExecutor; get_task_log stays and delegates to it, so older cores keep working.
  • Advertise supports_streaming_logs on KubernetesExecutor, CeleryKubernetesExecutor, and LocalKubernetesExecutor (the wrappers route kubernetes-queue tasks).
  • Update log handler tests to assert the streaming read path for running tasks.

Was generative AI tooling used to co-author this PR?

@boring-cyborg boring-cyborg Bot added area:Executors-core LocalExecutor & SequentialExecutor area:logging area:providers provider:celery provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Jul 3, 2026
@jason810496
jason810496 force-pushed the refactor/logging/kubernetes-executor-streaming-logs branch 4 times, most recently from 33b7c4d to 0d8a6bf Compare July 7, 2026 04:59
@jason810496 jason810496 self-assigned this Jul 7, 2026
@jason810496
jason810496 force-pushed the refactor/logging/kubernetes-executor-streaming-logs branch from 0d8a6bf to 06b2ad1 Compare July 8, 2026 02:24
@jason810496
jason810496 marked this pull request as ready for review July 8, 2026 02:24
@jason810496
jason810496 force-pushed the refactor/logging/kubernetes-executor-streaming-logs branch 2 times, most recently from c34f6c8 to b902870 Compare July 9, 2026 10:14

@Miretpl Miretpl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. I'd like a second review since I'm not familiar with how tests are done for the executors code.

@jason810496
jason810496 force-pushed the refactor/logging/kubernetes-executor-streaming-logs branch from b548338 to 8c19704 Compare July 22, 2026 08:48
Fetching a running task's pod log materialized every line in the API
server before serving it. Streaming the lines lazily through the new
BaseExecutor.get_streaming_task_log interface lets the bounded log
accumulator cap resident memory while serving large logs.
BaseExecutor now detects streaming support by calling
get_streaming_task_log directly and catching NotImplementedError, so
this flag is no longer read anywhere.
Empty ti.hostname produced "pod  through" (double space) in the log
message; only add the surrounding space when a hostname is present.
Requested in review: an underscored static method is clearer than an
inner function since the generator does not capture any local state.
@jason810496
jason810496 force-pushed the refactor/logging/kubernetes-executor-streaming-logs branch from 8c19704 to 9612834 Compare July 22, 2026 09:46
@jason810496

Copy link
Copy Markdown
Member Author

IIRC, the only CI failure should be resolved by other PR, so the static check failure it not related.

Traceback (most recent call last):
  File "/opt/airflow/scripts/in_container/run_provider_yaml_files_check.py", line 1117, in <module>
    all_parsed_yaml_files: dict[str, dict] = _load_package_data(paths)
  File "/opt/airflow/scripts/in_container/run_provider_yaml_files_check.py", line 170, in _load_package_data
    with open(provider_yaml_path) as yaml_file:
FileNotFoundError: [Errno 2] No such file or directory: '/opt/airflow/providers/apache/provider.yaml'

I will merge this.

@jason810496
jason810496 merged commit 5cbb64b into apache:main Jul 22, 2026
152 of 153 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Executors-core LocalExecutor & SequentialExecutor area:logging area:providers provider:celery provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants