Skip to content

Emit emitter/buffers/allocated/delta and emitter/buffers/failed/delta#6425

Merged
leventov merged 3 commits into
apache:masterfrom
metamx:emitter-delta-metrics
Oct 15, 2018
Merged

Emit emitter/buffers/allocated/delta and emitter/buffers/failed/delta#6425
leventov merged 3 commits into
apache:masterfrom
metamx:emitter-delta-metrics

Conversation

@leventov

@leventov leventov commented Oct 5, 2018

Copy link
Copy Markdown
Member

@gianm gauge emitter/buffers/failed and emitter/buffers/totalAllocated metrics proved to be very inconvenient in practice: they are not additive across a fleet of nodes, are randomly reset when a JVM restarts (that may be completely independent from metrics emitting issues). Regarding problem identification, emitter/buffers/dropped serves this well.

Currently in this PR I added /delta vs. /total metrics, but if there are no objections I would remove /total and so align emitter/buffers/allocated and /failed with /dropped and /events/emitted which are already delta-only.

@gianm gianm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. There is one thing that is technically incompatible, but I'm not sure it matters, since the old behavior was never documented. I will leave this open in case anyone else is interested in commenting.

{
int newTotalAllocatedBuffers = httpPostEmitter.getTotalAllocatedBuffers();
int allocatedBuffersDelta = newTotalAllocatedBuffers - lastTotalAllocatedBuffers;
emitter.emit(builder.build("emitter/buffers/allocated/total", newTotalAllocatedBuffers));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't compatible with the old name emitter/buffers/totalAllocated, but I think it's ok because we never documented the old names anyway.

@QiuMM QiuMM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lack of docs.

@leventov

leventov commented Oct 9, 2018

Copy link
Copy Markdown
Member Author

@QiuMM metrics emitted by monitors that are not enabled by default (HttpPostEmitterMonitor is not) are not documented.

Druid lacks an extensible system where each class or extension could define their own docs that are automatically published, see #4861.

@QiuMM

QiuMM commented Oct 9, 2018

Copy link
Copy Markdown
Member

Got that, then it looks good to me.

@nishantmonu51 nishantmonu51 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am in favor of removing total in this PR as it anyways changes the name of metrics in an incompatible way and it does not make much sense to have totals when we are emitting deltas.

@leventov

Copy link
Copy Markdown
Member Author

I removed /total and renamed emitter/buffers/dropped to emitter/buffers/dropped/delta and emitter/events/emitted to emitter/events/emitted/delta too.

@gianm gianm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@leventov leventov merged commit b57d5df into apache:master Oct 15, 2018
@leventov leventov deleted the emitter-delta-metrics branch October 15, 2018 20:30
@jon-wei jon-wei added this to the 0.14.0 milestone Feb 20, 2019
riovic918data pushed a commit to riovic918data/druid that referenced this pull request Jun 12, 2026
…apache#6425)

@gianm gauge `emitter/buffers/failed` and `emitter/buffers/totalAllocated` metrics proved to be very inconvenient in practice: they are not additive across a fleet of nodes, are randomly reset when a JVM restarts (that may be completely independent from metrics emitting issues). Regarding [problem identification](apache#4973 (comment)), `emitter/buffers/dropped` serves this well.

Currently in this PR I added `/delta` vs. `/total` metrics, but if there are no objections I would remove `/total` and so align `emitter/buffers/allocated` and `/failed` with `/dropped` and `/events/emitted` which are already delta-only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants