From 5d59d7a324514988ba9d2af9d10602655668f2c1 Mon Sep 17 00:00:00 2001 From: Alex Kantor Date: Wed, 1 Jul 2026 09:50:03 +0100 Subject: [PATCH] feat(k8s-reporter): add node scheduling controls and fix podAnnotations rendering Add nodeSelector, tolerations, affinity and podTemplateAnnotations to the k8s-reporter chart, rendered into the CronJob pod template, so the reporter can be pinned to a stable node group away from Karpenter-managed nodes. Also fix podAnnotations, which was rendered inline on the CronJob metadata and produced invalid YAML for any non-empty value. It is now rendered correctly and kept CronJob-scoped; pod-level annotations use the new podTemplateAnnotations. Document 'Running on EKS with Karpenter' in the README and Mintlify docs (root cause is the consolidateAfter churn timer, ref kubernetes-sigs/karpenter#1921), regenerate README, and bump the chart to 2.3.0. Refs #984 Co-Authored-By: Claude Opus 4.8 (1M context) --- charts/k8s-reporter/Chart.yaml | 2 +- charts/k8s-reporter/MINTLIFY.md.gotmpl | 2 + charts/k8s-reporter/README.md | 57 ++++++++++++++++++- charts/k8s-reporter/README.md.gotmpl | 2 + .../k8s-reporter/_mintlify_templates.gotmpl | 53 +++++++++++++++++ charts/k8s-reporter/_templates.gotmpl | 51 +++++++++++++++++ charts/k8s-reporter/templates/cronjob.yaml | 21 ++++++- charts/k8s-reporter/values.yaml | 14 ++++- 8 files changed, 197 insertions(+), 5 deletions(-) diff --git a/charts/k8s-reporter/Chart.yaml b/charts/k8s-reporter/Chart.yaml index f6db3aa15..36f289623 100644 --- a/charts/k8s-reporter/Chart.yaml +++ b/charts/k8s-reporter/Chart.yaml @@ -15,7 +15,7 @@ type: application # This is the chart version. This version number should be incremented each time you make changes # to the chart and its templates, including the app version. # Versions are expected to follow Semantic Versioning (https://semver.org/) -version: 2.2.1 +version: 2.3.0 # This is the version number of the (CLI) application being deployed. This version number should be # incremented each time you make changes to the application. They should reflect the version the diff --git a/charts/k8s-reporter/MINTLIFY.md.gotmpl b/charts/k8s-reporter/MINTLIFY.md.gotmpl index ec59b9af9..27d2cae92 100644 --- a/charts/k8s-reporter/MINTLIFY.md.gotmpl +++ b/charts/k8s-reporter/MINTLIFY.md.gotmpl @@ -20,6 +20,8 @@ description: A Helm chart for installing the Kosli K8S reporter as a cronjob. {{ template "mintlify.customCA" . }} +{{ template "mintlify.karpenter" . }} + {{ template "mintlify.configurations" . }} --- diff --git a/charts/k8s-reporter/README.md b/charts/k8s-reporter/README.md index b4c4684d0..186773f82 100644 --- a/charts/k8s-reporter/README.md +++ b/charts/k8s-reporter/README.md @@ -4,7 +4,7 @@ title: Kubernetes Reporter Helm Chart # k8s-reporter -![Version: 2.2.1](https://img.shields.io/badge/Version-2.2.1-informational?style=flat-square) +![Version: 2.3.0](https://img.shields.io/badge/Version-2.3.0-informational?style=flat-square) A Helm chart for installing the Kosli K8S reporter as a CronJob. The chart allows you to create a Kubernetes cronjob and all its necessary RBAC to report running images to Kosli at a given cron schedule. @@ -149,9 +149,59 @@ Both options use `secret`-backed volumes, which are permitted under the Pod Secu If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute a corporate CA bundle into a well-known ConfigMap in every namespace, point `extraVolumes` / `extraVolumeMounts` at that ConfigMap instead of creating a per-namespace Secret. +## Running on EKS with Karpenter (or another node autoscaler) + +By default the reporter runs as a CronJob every 5 minutes. On clusters that use [Karpenter](https://karpenter.sh) for node autoscaling, this frequent scheduling can prevent nodes from being **consolidated** (scaled down). + +The cause is Karpenter's `consolidateAfter` timer: Karpenter only consolidates a node once it has seen no pod scheduling activity on it for the configured window. A reporter pod arriving every 5 minutes keeps resetting that timer, so any node whose `consolidateAfter` is longer than the reporter interval never becomes eligible for consolidation (see [karpenter#1921](https://github.com/kubernetes-sigs/karpenter/issues/1921)). This is Karpenter working as designed, not a reporter bug. + +There are three good ways to avoid it, in order of preference. + +### 1. Widen the report interval + +The simplest fix. Set `cronSchedule` longer than your NodePool's `consolidateAfter` so nodes get quiet windows long enough to consolidate. Environment snapshots rarely need 5-minute freshness. + +```yaml +cronSchedule: "*/15 * * * *" +``` + +### 2. Pin the reporter to a stable node group + +If you run a stable managed node group that Karpenter does not manage, schedule the reporter there so it never disturbs Karpenter-managed nodes. Use `nodeSelector`, and `tolerations` if that node group is tainted: + +```yaml +nodeSelector: + eks.amazonaws.com/nodegroup: system # your managed node group + +tolerations: + - key: dedicated + operator: Equal + value: system + effect: NoSchedule +``` + +To steer the reporter *away* from Karpenter-managed nodes instead, use `affinity` (a plain `nodeSelector` cannot express "not on these nodes"): + +```yaml +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist +``` + +### 3. Run the reporter out of the cluster + +For zero footprint on cluster nodes, run `kosli snapshot k8s` on a schedule outside the cluster (for example a CI cron job) with kubeconfig access. See the [Kubernetes environment reporting tutorial](https://docs.kosli.com/tutorials/report_k8s_envs/). + +> Note: `karpenter.sh/do-not-disrupt: "true"` is **not** a fix here. It prevents Karpenter from disrupting the pod, which protects a mid-run report from interruption but makes consolidation of that node *less* likely, not more. Likewise `cluster-autoscaler.kubernetes.io/safe-to-evict` only affects the Kubernetes Cluster Autoscaler and is ignored by Karpenter. + ## Configurations | Key | Type | Default | Description | |-----|------|---------|-------------| +| affinity | object | `{}` | affinity rules for scheduling the reporter pod. Supports nodeAffinity, podAffinity and podAntiAffinity. | | concurrencyPolicy | string | `"Replace"` | specifies how to treat concurrent executions of a Job that is created by this CronJob | | cronSchedule | string | `"*/5 * * * *"` | the cron schedule at which the reporter is triggered to report to Kosli | | customCA | object | `{"enabled":false,"key":"ca.crt","secretName":""}` | convenience wrapper for mounting a corporate / custom CA bundle. See the "Running behind a TLS-inspecting proxy" section of the README for usage. | @@ -170,8 +220,10 @@ If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/t | kosliApiToken.secretKey | string | `"key"` | the name of the key in the secret data which contains the Kosli API token | | kosliApiToken.secretName | string | `"kosli-api-token"` | the name of the secret containing the kosli API token | | nameOverride | string | `""` | overrides the name used for the created k8s resources. If `fullnameOverride` is provided, it has higher precedence than this one | -| podAnnotations | object | `{}` | any custom annotations to be added to the cronjob | +| nodeSelector | object | `{}` | node labels for scheduling the reporter pod. On EKS with Karpenter, use this to pin the reporter to a stable managed node group (e.g. `eks.amazonaws.com/nodegroup: `) so it does not interfere with node consolidation. See the "Running on EKS with Karpenter" section of the README. | +| podAnnotations | object | `{}` | annotations to add to the CronJob object itself. For pod-level annotations (added to each reporter pod), use `podTemplateAnnotations` instead. | | podLabels | object | `{}` | custom labels to add to pods | +| podTemplateAnnotations | object | `{}` | annotations to add to the reporter pod template (applied to each Job pod that the CronJob creates) | | reporterConfig.dryRun | bool | `false` | whether the dry run mode is enabled or not. In dry run mode, the reporter logs the reports to stdout and does not send them to kosli. | | reporterConfig.environments | list | `[]` | List of Kosli environments to report to. Each entry has required 'name' and optional namespace selectors. Use one entry to report a single environment; use multiple entries to report to multiple environments with different selectors. Per entry: name (required), namespaces, namespacesRegex, excludeNamespaces, excludeNamespacesRegex (optional). Leave namespace fields unset for an entry to report the entire cluster to that environment. | | reporterConfig.httpProxy | string | `""` | the http proxy url | @@ -188,6 +240,7 @@ If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/t | serviceAccount.name | string | `""` | the name of the service account to use. If not set and create is true, a name is generated using the fullname template | | serviceAccount.permissionScope | string | `"cluster"` | specifies whether to create a cluster-wide permissions for the service account or namespace-scoped permissions. allowed values are: [cluster, namespace] | | successfulJobsHistoryLimit | int | `3` | specifies the number of successful finished jobs to keep | +| tolerations | list | `[]` | tolerations for scheduling the reporter pod, e.g. to run on a dedicated or tainted node group. | ---------------------------------------------- Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2) diff --git a/charts/k8s-reporter/README.md.gotmpl b/charts/k8s-reporter/README.md.gotmpl index c6c82ee37..41f526d6d 100644 --- a/charts/k8s-reporter/README.md.gotmpl +++ b/charts/k8s-reporter/README.md.gotmpl @@ -19,6 +19,8 @@ title: Kubernetes Reporter Helm Chart {{ template "extra.customCA" . }} +{{ template "extra.karpenter" . }} + {{ template "extra.valuesHeader" . }} {{ template "chart.valuesTable" . }} diff --git a/charts/k8s-reporter/_mintlify_templates.gotmpl b/charts/k8s-reporter/_mintlify_templates.gotmpl index a50ed33d8..99a330a4f 100644 --- a/charts/k8s-reporter/_mintlify_templates.gotmpl +++ b/charts/k8s-reporter/_mintlify_templates.gotmpl @@ -123,6 +123,59 @@ Both options use `secret`-backed volumes, which are permitted under the Pod Secu If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute a corporate CA bundle into a well-known ConfigMap in every namespace, point `extraVolumes` / `extraVolumeMounts` at that ConfigMap instead of creating a per-namespace Secret. {{- end }} +{{ define "mintlify.karpenter" -}} +## Running on EKS with Karpenter (or another node autoscaler) + +By default the reporter runs as a CronJob every 5 minutes. On clusters that use [Karpenter](https://karpenter.sh) for node autoscaling, this frequent scheduling can prevent nodes from being **consolidated** (scaled down). + +The cause is Karpenter's `consolidateAfter` timer: Karpenter only consolidates a node once it has seen no pod scheduling activity on it for the configured window. A reporter pod arriving every 5 minutes keeps resetting that timer, so any node whose `consolidateAfter` is longer than the reporter interval never becomes eligible for consolidation (see [karpenter#1921](https://github.com/kubernetes-sigs/karpenter/issues/1921)). This is Karpenter working as designed, not a reporter bug. + +There are three good ways to avoid it, in order of preference. + +### 1. Widen the report interval + +The simplest fix. Set `cronSchedule` longer than your NodePool's `consolidateAfter` so nodes get quiet windows long enough to consolidate. Environment snapshots rarely need 5-minute freshness. + +```yaml +cronSchedule: "*/15 * * * *" +``` + +### 2. Pin the reporter to a stable node group + +If you run a stable managed node group that Karpenter does not manage, schedule the reporter there so it never disturbs Karpenter-managed nodes. Use `nodeSelector`, and `tolerations` if that node group is tainted: + +```yaml +nodeSelector: + eks.amazonaws.com/nodegroup: system # your managed node group + +tolerations: + - key: dedicated + operator: Equal + value: system + effect: NoSchedule +``` + +To steer the reporter away from Karpenter-managed nodes instead, use `affinity` (a plain `nodeSelector` cannot express "not on these nodes"): + +```yaml +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist +``` + +### 3. Run the reporter out of the cluster + +For zero footprint on cluster nodes, run `kosli snapshot k8s` on a schedule outside the cluster (for example a CI cron job) with kubeconfig access. See the [Kubernetes environment reporting tutorial](/tutorials/report_k8s_envs). + + +`karpenter.sh/do-not-disrupt: "true"` is **not** a fix here. It prevents Karpenter from disrupting the pod, which protects a mid-run report from interruption but makes consolidation of that node *less* likely, not more. Likewise `cluster-autoscaler.kubernetes.io/safe-to-evict` only affects the Kubernetes Cluster Autoscaler and is ignored by Karpenter. + +{{- end }} + {{ define "mintlify.versionInfo" -}} This reference applies to **chart version {{ .Version }}**, which defaults to CLI **{{ .AppVersion }}** via `appVersion`. Override with `image.tag`. diff --git a/charts/k8s-reporter/_templates.gotmpl b/charts/k8s-reporter/_templates.gotmpl index 2a14daa82..7cc260811 100644 --- a/charts/k8s-reporter/_templates.gotmpl +++ b/charts/k8s-reporter/_templates.gotmpl @@ -153,6 +153,57 @@ Both options use `secret`-backed volumes, which are permitted under the Pod Secu If you already run [cert-manager's trust-manager](https://cert-manager.io/docs/trust/trust-manager/) to distribute a corporate CA bundle into a well-known ConfigMap in every namespace, point `extraVolumes` / `extraVolumeMounts` at that ConfigMap instead of creating a per-namespace Secret. {{- end }} +{{ define "extra.karpenter" -}} +## Running on EKS with Karpenter (or another node autoscaler) + +By default the reporter runs as a CronJob every 5 minutes. On clusters that use [Karpenter](https://karpenter.sh) for node autoscaling, this frequent scheduling can prevent nodes from being **consolidated** (scaled down). + +The cause is Karpenter's `consolidateAfter` timer: Karpenter only consolidates a node once it has seen no pod scheduling activity on it for the configured window. A reporter pod arriving every 5 minutes keeps resetting that timer, so any node whose `consolidateAfter` is longer than the reporter interval never becomes eligible for consolidation (see [karpenter#1921](https://github.com/kubernetes-sigs/karpenter/issues/1921)). This is Karpenter working as designed, not a reporter bug. + +There are three good ways to avoid it, in order of preference. + +### 1. Widen the report interval + +The simplest fix. Set `cronSchedule` longer than your NodePool's `consolidateAfter` so nodes get quiet windows long enough to consolidate. Environment snapshots rarely need 5-minute freshness. + +```yaml +cronSchedule: "*/15 * * * *" +``` + +### 2. Pin the reporter to a stable node group + +If you run a stable managed node group that Karpenter does not manage, schedule the reporter there so it never disturbs Karpenter-managed nodes. Use `nodeSelector`, and `tolerations` if that node group is tainted: + +```yaml +nodeSelector: + eks.amazonaws.com/nodegroup: system # your managed node group + +tolerations: + - key: dedicated + operator: Equal + value: system + effect: NoSchedule +``` + +To steer the reporter *away* from Karpenter-managed nodes instead, use `affinity` (a plain `nodeSelector` cannot express "not on these nodes"): + +```yaml +affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: karpenter.sh/nodepool + operator: DoesNotExist +``` + +### 3. Run the reporter out of the cluster + +For zero footprint on cluster nodes, run `kosli snapshot k8s` on a schedule outside the cluster (for example a CI cron job) with kubeconfig access. See the [Kubernetes environment reporting tutorial](https://docs.kosli.com/tutorials/report_k8s_envs/). + +> Note: `karpenter.sh/do-not-disrupt: "true"` is **not** a fix here. It prevents Karpenter from disrupting the pod, which protects a mid-run report from interruption but makes consolidation of that node *less* likely, not more. Likewise `cluster-autoscaler.kubernetes.io/safe-to-evict` only affects the Kubernetes Cluster Autoscaler and is ignored by Karpenter. +{{- end }} + {{ define "extra.valuesHeader" -}} ## Configurations {{- end }} diff --git a/charts/k8s-reporter/templates/cronjob.yaml b/charts/k8s-reporter/templates/cronjob.yaml index a5a11b813..71791821e 100644 --- a/charts/k8s-reporter/templates/cronjob.yaml +++ b/charts/k8s-reporter/templates/cronjob.yaml @@ -3,7 +3,10 @@ apiVersion: batch/v1 kind: CronJob metadata: name: {{ include "reporter.fullname" . }} - annotations: {{ toYaml .Values.podAnnotations }} + {{- with .Values.podAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} labels: {{- include "reporter.labels" . | nindent 4 }} @@ -17,12 +20,28 @@ spec: spec: template: metadata: + {{- with .Values.podTemplateAnnotations }} + annotations: + {{- toYaml . | nindent 12 }} + {{- end }} labels: {{- range $key, $value := .Values.podLabels }} {{ $key }}: {{ $value | quote }} {{- end }} spec: serviceAccountName: {{ include "reporter.serviceAccountName" . }} + {{- with .Values.nodeSelector }} + nodeSelector: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.affinity }} + affinity: + {{- toYaml . | nindent 12 }} + {{- end }} + {{- with .Values.tolerations }} + tolerations: + {{- toYaml . | nindent 12 }} + {{- end }} volumes: - name: environments-config configMap: diff --git a/charts/k8s-reporter/values.yaml b/charts/k8s-reporter/values.yaml index 119313c02..e6e0029fd 100644 --- a/charts/k8s-reporter/values.yaml +++ b/charts/k8s-reporter/values.yaml @@ -133,9 +133,21 @@ customCA: # -- key within the Secret that holds the PEM-formatted CA certificate (single cert or multi-cert PEM bundle) key: "ca.crt" -# -- any custom annotations to be added to the cronjob +# -- annotations to add to the CronJob object itself. For pod-level annotations (added to each reporter pod), use `podTemplateAnnotations` instead. podAnnotations: {} +# -- annotations to add to the reporter pod template (applied to each Job pod that the CronJob creates) +podTemplateAnnotations: {} + +# -- node labels for scheduling the reporter pod. On EKS with Karpenter, use this to pin the reporter to a stable managed node group (e.g. `eks.amazonaws.com/nodegroup: `) so it does not interfere with node consolidation. See the "Running on EKS with Karpenter" section of the README. +nodeSelector: {} + +# -- tolerations for scheduling the reporter pod, e.g. to run on a dedicated or tainted node group. +tolerations: [] + +# -- affinity rules for scheduling the reporter pod. Supports nodeAffinity, podAffinity and podAntiAffinity. +affinity: {} + resources: requests: # -- the memory request