k8s-reporter chart: expose nodeSelector/tolerations/affinity and pod-template annotations (Karpenter node consolidation)

[AlexKantor87]

Summary

The k8s-reporter Helm chart (charts/k8s-reporter/) runs the environment reporter as a CronJob (default every 5 minutes). On EKS clusters using Karpenter, this can prevent node consolidation, and the chart does not expose the scheduling controls needed to avoid it (no nodeSelector, tolerations, affinity, or pod-template annotations).

Why this happens (corrected after verification against Karpenter sources)

The mechanism is not that Karpenter refuses to disrupt a node running a Job pod. Karpenter will evict a controller-owned Job pod during consolidation. The real cause is the consolidateAfter churn timer:

• Karpenter tracks the last pod scheduling/removal event on a node (lastPodEventTime) and only marks the node Consolidatable once it has been free of pod churn for the NodePool's consolidateAfter window.
• A reporter pod arriving every 5 minutes keeps resetting that timer, so any node it lands on whose consolidateAfter is greater than or equal to the reporter interval never becomes consolidatable.
• This is confirmed maintainer-intended behaviour: Support ignoring specific pods when calculating consolidateAfter kubernetes-sigs/karpenter#1921 (open, triage/accepted) is exactly this CronJob scenario. Severity depends on the customer's consolidateAfter: if it is shorter than the reporter interval, quick-completing reporter pods may leave enough quiet gaps.

Current gaps in the chart

• No nodeSelector, tolerations, or affinity in values.yaml or the pod template, so the reporter cannot be pinned to a stable, non-Karpenter-managed node group.
• podAnnotations is rendered onto the CronJob object's metadata.annotations (templates/cronjob.yaml), not the pod template, so there is no way to set pod-level annotations. Only podLabels reaches the pod template.

Proposed change

Add standard scheduling controls to the pod template so operators can keep the reporter off Karpenter-managed nodes:

1. nodeSelector, tolerations, affinity rendered into spec.jobTemplate.spec.template.spec. Primary value: pin the reporter to a stable managed node group (e.g. eks.amazonaws.com/nodegroup: <name>), which Karpenter never considers for consolidation, so the reporter stops interfering. AWS EKS best practices recommend keeping such workloads on a managed node group.
2. podTemplateAnnotations rendered into the pod template's metadata.annotations, for pod-level annotations. Keep the existing podAnnotations (CronJob-level) unchanged to avoid a breaking change, and tighten its comment.

Mitigations to document (so users avoid this)

Ranked for a Karpenter user:

1. Widen cronSchedule beyond the NodePool's consolidateAfter (already supported; cheapest; attacks the root cause directly, ref #1921).
2. Pin to a stable node group via the new nodeSelector/tolerations/affinity (when such a node group exists).
3. Run out-of-cluster (externally scheduled kosli snapshot k8s) for zero node footprint.
4. A long-running Deployment mode would also help (removes per-run pod events); deferred to a separate issue, as it needs a watch/loop mode in the CLI.

Corrections to earlier framing in this issue

• cluster-autoscaler.kubernetes.io/safe-to-evict is a Cluster Autoscaler annotation and is a no-op on Karpenter (Support the cluster-autoscaler.kubernetes.io/safe-to-evict annotation kubernetes-sigs/karpenter#703, closed not-planned). It is not a Karpenter mitigation.
• karpenter.sh/do-not-disrupt: "true" prevents disruption; it protects a mid-run pod from eviction but does not help consolidation (if anything it blocks it). It should not be presented as a consolidation-friendliness knob.

References

• Chart: charts/k8s-reporter/values.yaml, charts/k8s-reporter/templates/cronjob.yaml
• Karpenter consolidateAfter + CronJob churn (root cause): Support ignoring specific pods when calculating consolidateAfter kubernetes-sigs/karpenter#1921
• Karpenter disruption docs: https://karpenter.sh/docs/concepts/disruption/
• EKS best practices (pin workloads to managed node groups): https://aws.github.io/aws-eks-best-practices/karpenter/
• safe-to-evict not honoured by Karpenter: Support the cluster-autoscaler.kubernetes.io/safe-to-evict annotation kubernetes-sigs/karpenter#703

[mbevc1]

Option 3 would need more work as we currently run CLI on schedule