[fix][broker][functions-worker] Ensure prometheus metrics are grouped by type (#8407, #13865)

[marksilcox]

Fixes #8407
Fixes #13865

Motivation

Current broker prometheus metrics are not grouped by metric type which causes issues in systems that read these metrics (e.g. DataDog).

Prometheus docs states "All lines for a given metric must be provided as one single group" - https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#grouping-and-sorting

Modifications
Updated the namespace and topic prometheus metric generators to group the metrics under the appropriate type header.
Updated function worker stats to include TYPE headers

Verifying this change

• Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

Added unit test to verify all metrics are grouped under correct type header

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (no)
• The public API: (no)
• The schema: (no)
• The default values of configurations: (no)
• The wire protocol: (no)
• The rest endpoints: (no)
• The admin cli options: (no)
• Anything that affects deployment: (no)

Documentation

Need to update docs?

• doc-not-needed
  Changes to match prometheus spec

[marksilcox]

@codelipenghui can you look at this?

[eolivelli]

Thanks for your first contribution

while I understand the goal of this patch and I think we should address this problem, we must find a better way to reduce memory allocations and the amount of memory we are using while generating metrics.

[marksilcox]

@eolivelli could you please approve the workflows and review again

[marksilcox]

/pulsarbot run-failure-checks

[marksilcox]

/pulsarbot run-failure-checks

[marksilcox]

/pulsarbot run cpp-tests

[Jason918]

branch-2.10 must be kept stable and we should cherry-pick only stuff that we are sure it is not going to break stability or compatibility.

I agree absolutely, there are over 190 commits included in 2.10.2.

[marksilcox]

@michaeljmarshall - DataDog seems to expect the prometheus metrics to be grouped together by name, this is usually under a #TYPE heading. If the metrics are not grouped then DataDog ignores any under a #TYPE header that do not have the name in that header.

[michaeljmarshall]

Thanks for clarifying @marksilcox. I misunderstood the initial issue. This seems like a bug fix, not an improvement @eolivelli.

[mattisonchao]

Hello @marksilcox
It looks like we got many conflicts when cherry-picking it to branch-2.9.
Would you mind pushing a PR to branch-2.9? (To avoid cherry-picking involving bugs)

[marksilcox]

@mattisonchao created #17618 to merge into branch-2.9.

Is similar needed for branch-2.10?

[liangyepianzhou]

@mattisonchao created #17618 to merge into branch-2.9.
Is similar needed for branch-2.10?

@marksilcox This PR does need to cherry-pick to branch-2.10.
I am trying to cherry-pick it to branch-2.10, but there are a lot of conflicts.
Could you please open a PR to cherry-pick this PR to branch-2.10 (to avoid including bugs)?

[liangyepianzhou]

I move this to 2.10.4. If you have any questions, feel free to ping me.

[asafm]

Just want to remind everyone that when a new PR was created to cherry-pick it from master to 2.9, a memory leak was introduced since the 2.9 metrics implementation is different a bit from the master branch. So we need to double-check that. See memory leak fixed

[marksilcox]

@liangyepianzhou I've created #18941 to merge into branch-2.10.
@asafm compared these changes against that PR to ensure the memory leak fix is in place

[Technoboy-]

#17419 relies on this, so cherry-picked to branch-2.11