[kube-prometheus-stack] Alertmanager statefulset not found by operator due to label mismatch

[toffiebotha]

Describe the bug a clear and concise description of what the bug is.

Using version 81.3.0 of the chart, after upgrading from 75.15.1
The prometheus operator is unable to find existing Alertmanager statefulsets which results in it trying to create the STS and logging errors stating:
failed: creating statefulset failed: statefulsets.apps \"alertmanager-prometheus-alertmanager\" already exists

This is due a change that was made in the prometheus operator since 0.85.0 where it looks for the STS based on the labels:

• app.kubernetes.io/name: alertmanager
• app.kubernetes.io/managed-by: prometheus-operator

The chart currently does not set the AlertManager resource's app.kubernetes.io/name label, and sets app.kubernetes.io/managed-by to HELM.

Workaround that has worked for me, is to configure values as follow:

alertmanager:
    enabled: true
    additionalLabels:
      app.kubernetes.io/name: alertmanager
      app.kubernetes.io/managed-by: prometheus-operator

What's your helm version?

3.19.5+g4a19a5b

What's your kubectl version?

v0.34.0

Which chart?

kube-prometheus-stack

What's the chart version?

81.3.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

No command, installed by argoCD

Anything else we need to know?

Related to this issue: prometheus-operator/prometheus-operator#8312

[AlexW00]

i fixed it (without helm chart values) by deleting the alertmanager statefulset - prometheus operator will re-create it with all the correct labels and can manage it again

[toffiebotha]

@AlexW00 Are you suggesting there is nothing in the chart to fix then?
Deleting the STS in production isn't always great, especially when there's persistent volumes involved.

Were these two mentioned labels added after you deleted and recreated the STS, or is it just that the operator then has a proper handle on the STS to know it exists without these two labels?

Think it would be better if the chart could update the STS and have the operator detect it without needing to delete the STS.

[AlexW00]

@AlexW00 Are you suggesting there is nothing in the chart to fix then? Deleting the STS in production isn't always great, especially when there's persistent volumes involved.

Were these two mentioned labels added after you deleted and recreated the STS, or is it just that the operator then has a proper handle on the STS to know it exists without these two labels?

Think it would be better if the chart could update the STS and have the operator detect it without needing to delete the STS.

afaik selector labels of statefulsets can't be updated since they are immutable - even if some helm chart changes would be made they wouldn't be able to update them.

before deleting the alertmanager sts, i checked there's no PVCs mounted; therefore it was save to delete.

however, if you don't want to delete the pods with it, you should delete it like so:

kubectl delete sts <sts-name> --cascade=orphan

this will keep all pods running - when the sts gets re-created it will "adopt" them again

note: i didn't do the --cascade=orphan method in my case since there were no PVCs involved - however I had to also restart the kube prom stack once so it re-created the sts...

i dont see any todos for the helm chart here

[toffiebotha]

The update to the labels is not the selectorLabels that targets the pods, it is the STS resource's main labels - See image below, those labels are applied to the STS as well - The workaround I applied didn't need recreation of the STS and resolved the prometheus operator's error messages as it only tries to find the STS, not the pods associated with the STS.

Our STS has PV and PVCs attached which adds to the headache of recreating the STS