Expose msg_backlog_non_delayed metric

[codelipenghui]

Is your feature request related to a problem? Please describe.
Currently, metric msg_backlog contains delayed messages and un-delayed messages. However, users need the number of non-delayed messages in the backlog, which can use by too many un-delayed backlogs alerts. So it's better to add msg_backlog_non_delayed metric.

Discussion at slack channel: https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1581283048289800

[codelipenghui]

@sijie From the discussion at slack, you have mentioned that msg_backlog - msg_delayed can not be reliably used for measuring non-delayed messages. Does this mean we need an accurate way of measuring or just a reminder, we still expose an indicator msg_backlog_non_delayed = msg_backlog - msg_delayed?

[sijie]

I was thinking of finding a more accurate way to measure this.

[gmethvin]

Thanks @codelipenghui! Specifically, we want the number of messages that are ready to be consumed right now. In other words, we want the number of messages that either were not published with a deliverAt time, or have a deliverAt time that is on or before the current time.

For alerting purposes it should ideally be accurate within a minute or so. The problem we've seen is the number msg_backlog - msg_delayed can often be incorrect for a day or more so it is not sufficient for this purpose.

[codelipenghui]

Ok, I will take a look.

[sijie]

@gmethvin @codelipenghui

I created an issue #6045 before. msg_backlog is an estimated value. It is estimated by minus two cached numbers (numProduced - numConsumed).