Skip to content
This repository was archived by the owner on Jan 24, 2024. It is now read-only.

add prometheus metric exporter framework#389

Merged
BewareMyPower merged 11 commits into
streamnative:masterfrom
hangc0276:chenhang/add_prometheus_metric_exporter_framework
Apr 1, 2021
Merged

add prometheus metric exporter framework#389
BewareMyPower merged 11 commits into
streamnative:masterfrom
hangc0276:chenhang/add_prometheus_metric_exporter_framework

Conversation

@hangc0276

Copy link
Copy Markdown
Collaborator

Motivation

KOP doesn't expose prometheus metrics to monitor running stats, which will be hard the debug performance problem.

Changs

  1. add Prometheus metrics provider for kop and use individual port to expose kop metrics
  2. Due to BookKeeper Prometheus metric provider include basic metrics stats, such as jvm metrics, memory metrics, cpu metric etc, which will lead those metrics register twice. When run kop with pulsar broker, the prometheus metric thread will start failed.

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hangc0276 We can expose metrics through /metrics endpoint of the broker directly, I have add Raw Metrics Provider in Pulsar (apache/pulsar#9021), you can use it to expose additional metrics to the broker metrics endpoint.

@BewareMyPower

Copy link
Copy Markdown
Collaborator

The kafka-impl/conf won't be used anymore (see #387 ), please edit docs/configuration.md or add a new document for exposing metrics to Prometheus under docs directory and add a link in README.md.

@hangc0276
hangc0276 force-pushed the chenhang/add_prometheus_metric_exporter_framework branch from 84e0f20 to bc25497 Compare March 28, 2021 04:51
@hangc0276

Copy link
Copy Markdown
Collaborator Author

@hangc0276 We can expose metrics through /metrics endpoint of the broker directly, I have add Raw Metrics Provider in Pulsar (apache/pulsar#9021), you can use it to expose additional metrics to the broker metrics endpoint.

@codelipenghui Ok, I update the code and use pulsar raw metrics provider to expose kop metrics to pulsar metrics group. Please take a look. Thanks.

@hangc0276
hangc0276 requested a review from codelipenghui March 28, 2021 04:53
@hangc0276
hangc0276 requested a review from aloyszhang March 28, 2021 05:08
@BewareMyPower

Copy link
Copy Markdown
Collaborator

You should fix the code style first. Then please remove kafka-impl/conf/broker.conf. It's unused.

Regarding how to enable the monitor for prometheus, we can open a new PR to add the docs.

@jiazhai

jiazhai commented Mar 28, 2021

Copy link
Copy Markdown
Contributor

opened a new issue for the doc. @codelipenghui @BewareMyPower Would you please help review it again?

Comment on lines +24 to +25
String SERVER_STATUS = "SERVER_STATUS";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
String SERVER_STATUS = "SERVER_STATUS";

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unused variable

@BewareMyPower

Copy link
Copy Markdown
Collaborator

I've left a few comments, PTAL.

@hangc0276

Copy link
Copy Markdown
Collaborator Author

I've left a few comments, PTAL.

@BewareMyPower OK, i will fix it

@hangc0276
hangc0276 force-pushed the chenhang/add_prometheus_metric_exporter_framework branch from 987df52 to fd72cc5 Compare March 31, 2021 14:58
@hangc0276
hangc0276 requested a review from BewareMyPower March 31, 2021 15:00
@BewareMyPower
BewareMyPower merged commit 0643fae into streamnative:master Apr 1, 2021
BewareMyPower added a commit that referenced this pull request Nov 17, 2021
### Motivation

#389 exposed the metrics for KoP and added the docs. However, users cannot click into the metrics docs in the main web page.

### Modifications

Add the link of metrics docs to README.md. In addition, sync docs/kop.md with README.md so that upgrade.md and reference-metrics.md can both be navigated from SN hub. Then remove the **Upgrade** section because it's duplicated with upgrade.md.
BewareMyPower added a commit that referenced this pull request Nov 17, 2021
### Motivation

#389 exposed the metrics for KoP and added the docs. However, users cannot click into the metrics docs in the main web page.

### Modifications

Add the link of metrics docs to README.md. In addition, sync docs/kop.md with README.md so that upgrade.md and reference-metrics.md can both be navigated from SN hub. Then remove the **Upgrade** section because it's duplicated with upgrade.md.
BewareMyPower added a commit that referenced this pull request Nov 17, 2021
### Motivation

#389 exposed the metrics for KoP and added the docs. However, users cannot click into the metrics docs in the main web page.

### Modifications

Add the link of metrics docs to README.md. In addition, sync docs/kop.md with README.md so that upgrade.md and reference-metrics.md can both be navigated from SN hub. Then remove the **Upgrade** section because it's duplicated with upgrade.md.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants