Add support for `AlertmanagerConfig` CRDs

[pkarakal]

Component(s)

No response

Background

Currently, in order to update the Mimir Alertmanager configuration for a tenant, we need to have the Mimir Alertmanager exposed to the internet so we can push configuration changes via HTTP calls or mimirtool (which performs the HTTP requests for us). While Mimir alertmanager has support for firewall configuration, exposing it to the internet is not ideal.

To work around this limitation, I currently use a custom Lambda function that is triggered on S3 file uploads and pushes the configuration to the alertmanager within the same VPC, keeping everything private. However, this approach requires additional infrastructure and doesn't provide a native GitOps experience.

Proposal

Grafana Alloy already has excellent support for tracking PrometheusRule CRDs when deployed in Kubernetes through the mimir.rules.kubernetes component, which pushes rule configurations to Mimir Ruler. I propose adding a similar component for AlertManager configurations.

Proposed Component: mimir.alertmanager.kubernetes

This new component would:

• Track AlertmanagerConfig CRDs in the Kubernetes cluster
• Automatically push configuration changes to Mimir Alertmanager
• Provide the same seamless GitOps experience that currently exists for Mimir Ruler
• Keep the Mimir Alertmanager private and secure within the cluster

This would mirror the existing functionality of mimir.rules.kubernetes but for alertmanager configurations instead of recording/alerting rules. It would also eliminate the need to expose Mimir Alertmanager to the internet and provide a unified approach for managing both Mimir Ruler and Alertmanager configurations through Kubernetes CRDs, enabling true declarative GitOps workflows for both rules and alertmanager configurations.

This proposal is similar to a feature request I previously submitted to the Mimir project for file-based configuration loading. However, implementing this in Alloy would provide a more Kubernetes-native solution that integrates seamlessly with existing GitOps workflows. I don't have a huge preference on either proposal and I get behind either one of them.

Tip

_{React with 👍 if this issue is important to you.}

[ptodev]

Hi, @pkarakal! Would you mind if I close this as a duplicate of #504? Most of the development work has already been done, but I need to wait until both OpenTelemetry Collector and Prometheus have updated their Go versions and their k8s dependencies. I see that OTel already upgraded their Go, so I hope we can merge this feature soon.

[pkarakal]

No problem from my side. Sorry for the duplication.