[improve][pip] PIP-406: Introduce metrics related to dispatch throttled events

[shibd]

Motivation

• Discuss email: https://lists.apache.org/thread/w9pq1jp3f22prsr97mxzj1yhdoo1cnc4
• Vote thread: https://lists.apache.org/thread/pmgkf67ph98k5l3lr60rtg96octb397q
• Implatement PR: [improve][broker] PIP-406: Introduce metrics related to dispatch throttled events #23946

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[shibd]

hi, @RobertIndie @tjiuming @poorbarcode

The previous design made it difficult for users to identify where throttling was causing a high backlog. I have rewritten the PIP to separate the metrics at the broker, topic, and subscription levels. Please review it again.

Thanks.

[RobertIndie]

I think we don't need separate metrics for both messages throttled and bytes throttled. This could make it complex for users. Could we just name it pulsar_dispatch_throttled_events and only count the number of throttled events where the message count or bytes exceed the rate limits?

So the logic would be like:

     private boolean applyDispatchRateLimitsToReadLimits(DispatchRateLimiter rateLimiter,
                                                         MutablePair<Integer, Long> readLimits,
                                                         DispatchRateLimiter.Type limiterType) {
+        int originalMessagesToRead = readLimits.getLeft();
+        long originalBytesToRead = readLimits.getRight();
         // update messagesToRead according to available dispatch rate limit.
         int availablePermitsOnMsg = (int) rateLimiter.getAvailableDispatchRateLimitOnMsg();
         if (availablePermitsOnMsg >= 0) {
@@ -414,6 +416,12 @@ private boolean applyDispatchRateLimitsToReadLimits(DispatchRateLimiter rateLimi
         if (availablePermitsOnByte >= 0) {
             readLimits.setRight(Math.min(readLimits.getRight(), availablePermitsOnByte));
         }
+        if (readLimits.getLeft() < originalMessagesToRead || readLimits.getRight() < originalBytesToRead) {
+            rateLimiter.increaseDispatchThrottleMsgCount();
+        }

[lhotari]

LGTM, thank you @shibd

[shibd]

https://lists.apache.org/thread/pmgkf67ph98k5l3lr60rtg96octb397q

The vote passed with 3 +1 binding.

• Lari
• Zike
• Yubiao