Skip to content

fix: aggregate ephemeral workload names to prevent series cardinality explosion#241

Merged
mayankpande88 merged 1 commit into
mainfrom
fix/aggregate-ephemeral-workloads
Apr 13, 2026
Merged

fix: aggregate ephemeral workload names to prevent series cardinality explosion#241
mayankpande88 merged 1 commit into
mainfrom
fix/aggregate-ephemeral-workloads

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Summary

  • Bare pods and standalone Jobs (e.g. Airflow KubernetesExecutor task pods) each get a unique src_workload_name in TCP metrics, causing O(pods × destinations) series growth. For one customer this produced 693k series / 11GB RSS from 2,608 unique Airflow task pod names.
  • Resolves a stable workload name from standard labels (app.kubernetes.io/name, app, app.kubernetes.io/component, component) for kind=pod and kind=Job. Falls back to <namespace>-ephemeral when no label matches.
  • Controlled by --aggregate-ephemeral-workloads / AGGREGATE_EPHEMERAL_WORKLOADS (default: true). Pods owned by Deployments, StatefulSets, DaemonSets, or CronJobs are unaffected.

Test plan

  • Unit tests for label priority resolution (TestResolveEphemeralWorkloadName)
  • Integration tests for bare pods, standalone Jobs, and verifying CronJob/Deployment-owned pods are NOT aggregated (TestEphemeralWorkloadAggregation)
  • All 25 existing + new tests pass
  • gofmt and go vet clean
  • Deploy to dev cluster and verify Airflow-like ephemeral pods collapse to a single workload name
  • Verify Prometheus series count drops as expected

… explosion

Bare pods and standalone Jobs (e.g. Airflow KubernetesExecutor task pods)
each get a unique workload name in TCP metrics, causing O(pods × destinations)
series growth. For one customer this produced 693k series / 11GB RSS from
2,608 unique Airflow task pod names.

Resolve a stable workload name from standard labels (app.kubernetes.io/name,
app, app.kubernetes.io/component, component) for kind=pod and kind=Job.
Falls back to <namespace>-ephemeral when no label matches.

Controlled by --aggregate-ephemeral-workloads / AGGREGATE_EPHEMERAL_WORKLOADS
(default: true). Pods owned by Deployments, StatefulSets, DaemonSets, or
CronJobs are unaffected.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements aggregation for ephemeral workloads, specifically bare pods and standalone Jobs, to reduce metric cardinality. It introduces the AggregateEphemeralWorkloads flag and logic to resolve stable workload names using standard Kubernetes labels, falling back to a namespace-specific ephemeral name when labels are missing. The implementation includes corresponding unit tests. A review comment identifies an inconsistency in string casing for workload type checks that could lead to fragility.

Comment thread common/ip_resolver.go
@mayankpande88 mayankpande88 merged commit 750bfdd into main Apr 13, 2026
1 check passed
@mayankpande88 mayankpande88 deleted the fix/aggregate-ephemeral-workloads branch April 13, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants