Micrometer metrics meter `events_received_total` is registered multiple times

[ThoSap]

Bug Report

What did you do?

Update to Quarkus 3.34.1 which also updates the JOSDK to 7.7.3

What did you expect to see?

That my tests run through.

What did you see instead? Under which circumstances?

I ran my test suite of https://github.com/aboutbits/postgresql-operator with Quarkus 3.34.1 (e.g. JOSDK 7.7.3 -> Java Operator SDK 5.3.2) and got the following error.
This is related to https://github.com/quarkusio/quarkus/wiki/Migration-Guide-3.34#micrometer-prometheus-registry

Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'events_received_total' containing tag keys [action, controller_name, event]. The meter you are attempting to register has keys [controller_name, event].
java.lang.IllegalArgumentException: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'events_received_total' containing tag keys [action, controller_name, event]. The meter you are attempting to register has keys [controller_name, event].
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$throwExceptionOnRegistrationFailure$19(PrometheusMeterRegistry.java:619)
	at io.micrometer.core.instrument.MeterRegistry.meterRegistrationFailed(MeterRegistry.java:1291)
	at io.micrometer.prometheus.PrometheusMeterRegistry.lambda$applyToCollector$18(PrometheusMeterRegistry.java:593)
	at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1956)
	at io.micrometer.prometheus.PrometheusMeterRegistry.applyToCollector(PrometheusMeterRegistry.java:579)
	at io.micrometer.prometheus.PrometheusMeterRegistry.newCounter(PrometheusMeterRegistry.java:200)
	at io.micrometer.core.instrument.MeterRegistry.lambda$counter$4(MeterRegistry.java:348)
	at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:725)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:652)
	at io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:347)
	at io.micrometer.core.instrument.Counter$Builder.register(Counter.java:148)
	at io.micrometer.core.instrument.Counter$Builder.register(Counter.java:144)
	at io.micrometer.core.instrument.composite.CompositeCounter.registerNewMeter(CompositeCounter.java:52)
	at io.micrometer.core.instrument.composite.CompositeCounter.registerNewMeter(CompositeCounter.java:23)
	at io.micrometer.core.instrument.composite.AbstractCompositeMeter.add(AbstractCompositeMeter.java:67)
	at java.base/java.lang.Iterable.forEach(Iterable.java:75)
	at java.base/java.util.Collections$SetFromMap.forEach(Collections.java:6060)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$0(CompositeMeterRegistry.java:67)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lock(CompositeMeterRegistry.java:189)
	at io.micrometer.core.instrument.composite.CompositeMeterRegistry.lambda$new$1(CompositeMeterRegistry.java:67)
	at io.micrometer.core.instrument.MeterRegistry.getOrCreateMeter(MeterRegistry.java:735)
	at io.micrometer.core.instrument.MeterRegistry.registerMeterIfNecessary(MeterRegistry.java:652)
	at io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:347)
	at io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:481)
	at io.micrometer.core.instrument.MeterRegistry.counter(MeterRegistry.java:459)
	at io.javaoperatorsdk.operator.monitoring.micrometer.MicrometerMetricsV2.incrementCounter(MicrometerMetricsV2.java:260)
	at io.javaoperatorsdk.operator.monitoring.micrometer.MicrometerMetricsV2.eventReceived(MicrometerMetricsV2.java:178)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor.handleEvent(EventProcessor.java:130)
	at io.javaoperatorsdk.operator.processing.event.source.timer.TimerEventSource$EventProducerTimeTask.run(TimerEventSource.java:129)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:572)
	at java.base/java.util.TimerThread.run(Timer.java:522)

Environment

Kubernetes cluster type: K3s

$ Mention java-operator-sdk version from pom.xml file
Java Operator SDK version: 5.3.2 (Quarkus 3.34.1 -> JOSDK 7.7.3)

$ java -version
Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 25.0.1+8.1 (build 25.0.1+8-LTS-jvmci-b01, mixed mode, sharing)

$ kubectl version
Client Version: v1.34.1+k3s1
Kustomize Version: v5.7.1
Server Version: v1.34.1+k3s1

Possible Solution

Register the Micrometer metrics only once by using MeterProvider to delay meter registration and avoid duplicating the registration of different 'events_received_total' tag keys [action, controller_name, event] and [controller_name, event].

Additional context

[metacosm]

A simple solution would be to add a specific tag for event that don't have any associated actions.

[csviri]

@ThoSap did you migrate to MicrometerMetricsV2 or still using MicrometerMetrics ?

[metacosm]

@ThoSap did you migrate to MicrometerMetricsV2 or still using MicrometerMetrics ?

@csviri If you look at the stack trace, you'll see this is coming from v2.

[csviri]

I see (sorry just hastily opened this), pls take a look if this makes sense:
#3256

I will also extend the E2E test to have these events covered.

[csviri]

@ThoSap could you pls test now on the snapshot version from main?

thank you!

[ThoSap]

@csviri I just built snapshot version 5.3.1-SNAPSHOT (commit c0eb78f), and I no longer get the exceptions in my tests with Quarkus 3.34.1.

Thanks guys ❤️

Subject: [PATCH] try Java Operator SDK c0eb78f SNAPSHOT
---
Index: build.gradle.kts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/build.gradle.kts b/build.gradle.kts
--- a/build.gradle.kts	(revision f0a650a6ac5c6b4a680a203c80a16f8baa1a4ca6)
+++ b/build.gradle.kts	(date 1774626384213)
@@ -68,7 +68,7 @@
         // https://mvnrepository.com/artifact/io.quarkus.platform/quarkus-bom
         implementation(enforcedPlatform("${quarkusPlatformGroupId}:${quarkusPlatformArtifactId}:${quarkusPlatformVersion}"))
         // https://mvnrepository.com/artifact/io.quarkus.platform/quarkus-operator-sdk-bom
-        implementation(enforcedPlatform("${quarkusPlatformGroupId}:quarkus-operator-sdk-bom:${quarkusPlatformVersion}"))
+        implementation(platform("${quarkusPlatformGroupId}:quarkus-operator-sdk-bom:${quarkusPlatformVersion}"))
 
         /**
          * NullAway
Index: operator/build.gradle.kts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/operator/build.gradle.kts b/operator/build.gradle.kts
--- a/operator/build.gradle.kts	(revision f0a650a6ac5c6b4a680a203c80a16f8baa1a4ca6)
+++ b/operator/build.gradle.kts	(date 1774626111375)
@@ -21,6 +21,16 @@
      */
     implementation("io.fabric8:generator-annotations")
     implementation("io.fabric8:crd-generator-api-v2")
+    implementation("io.javaoperatorsdk:operator-framework-core") {
+        version {
+            strictly("5.3.1-SNAPSHOT")
+        }
+    }
+    implementation("io.javaoperatorsdk:micrometer-support") {
+        version {
+            strictly("5.3.1-SNAPSHOT")
+        }
+    }
 
     /**
      * jOOQ
Index: gradle.properties
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>ISO-8859-1
===================================================================
diff --git a/gradle.properties b/gradle.properties
--- a/gradle.properties	(revision f0a650a6ac5c6b4a680a203c80a16f8baa1a4ca6)
+++ b/gradle.properties	(date 1774625928350)
@@ -7,9 +7,9 @@
 
 # Quarkus
 quarkusPluginId=io.quarkus
-quarkusPluginVersion=3.33.1
+quarkusPluginVersion=3.34.1
 # https://mvnrepository.com/artifact/io.quarkus.platform/quarkus-bom
 quarkusPlatformGroupId=io.quarkus.platform
 quarkusPlatformArtifactId=quarkus-bom
-quarkusPlatformVersion=3.33.1
+quarkusPlatformVersion=3.34.1
 systemProp.quarkus.analytics.disabled=true