ServiceMonitors broken in v1.12.0 with labeldrop rule

[eloo-abi]

Component(s)

prometheus.operator.servicemonitors

What's wrong?

Hi,
we have updated to alloy 1.12.0 and have discovered that our service monitors are not scraped anymore.

We get the following error:

"labeldrop action requires only \'regex\', and no other fields"

I have checked the version 1.11.3 which is working fine.
Also its kinda weird that podmonitors are working as expected with the same configuration.

As you can see in the config example it should work as expected. But as soon as there is at least one labeldrop rule included the reconcile of the servicemonitors is broken.

This leads to all metrics related to the service monitors not being scraped anymore and thus to an outage of the monitoring system at all.

Steps to reproduce

• Deploy alloy with a config like the mentioned one
• Add the same rule also for a podmonitor
• Deploy a service monitor and a podmonitor
• See podmonitor working
• See servicemonitor reconcile errors

System information

No response

Software version

Grafana Alloy 1.12.0

Configuration

prometheus.operator.servicemonitors "service_monitors" {
        clustering {
          enabled = true
        }
        scrape {
          default_scrape_interval = "60s"
          default_scrape_timeout = "10s"
        }

        rule {
          action = "labeldrop"
          regex  = "helm_sh_chart"
        }

        forward_to = argument.metrics_destinations.value
      }

Logs

Tip

_{React with 👍 if this issue is important to you.}

[kalleep]

Related to Prometheus update. Will try to figure out why this happens and it's strange that it only happens for servicemonitors

[kalleep]

Pretty sure I found the issue, seems like relabel rules are now validated when scrape config is validated. The issue we are hitting is this. In alloy default value for source_labels is not nil but an empty slice so we need to fix that

[eloo-abi]

@kalleep sounds good..

i will give it a try as soon as the release is available

thanks!