Add key-shared consumer range to internal topic stats.

[MarvinCai]

Fixes #5752

Master Issue: #4077

Motivation

Expose consumer range information in topic internal stats.
sample output

"persistent://zxc-tenant/zxc-namespace/zxc-topicccc-p-partition-3" : {
      "entriesAddedCounter" : 5,
      "numberOfEntries" : 5,
      "totalSize" : 300,
      "currentLedgerEntries" : 5,
      "currentLedgerSize" : 300,
      "lastLedgerCreatedTimestamp" : "2020-11-14T21:12:08.753-08:00",
      "waitingCursorsCount" : 3,
      "pendingAddEntriesCount" : 0,
      "lastConfirmedEntry" : "138:4",
      "state" : "LedgerOpened",
      "ledgers" : [ {
        "ledgerId" : 138,
        "entries" : 0,
        "size" : 0,
        "offloaded" : false
      } ],
      "cursors" : {
        "zxc-sub" : {
          "markDeletePosition" : "138:4",
          "readPosition" : "138:5",
          "waitingReadOp" : true,
          "pendingReadOps" : 0,
          "messagesConsumedCounter" : 5,
          "cursorLedger" : 140,
          "cursorLedgerLastEntry" : 0,
          "individuallyDeletedMessages" : "[]",
          "lastLedgerSwitchTimestamp" : "2020-11-14T21:13:35.982-08:00",
          "state" : "Open",
          "numberOfEntriesSinceFirstNotAckedMessage" : 1,
          "totalNonContiguousDeletedMessagesRange" : 0,
          "consumerRange" : {
            "0--32768" : "cecee",
            "32769--65536" : "b514e"
          },
          "properties" : { }
        },
        "zxcc-sub" : {
          "markDeletePosition" : "138:-1",
          "readPosition" : "138:-1",
          "waitingReadOp" : false,
          "pendingReadOps" : 0,
          "messagesConsumedCounter" : 0,
          "cursorLedger" : 141,
          "cursorLedgerLastEntry" : 1,
          "individuallyDeletedMessages" : "[]",
          "lastLedgerSwitchTimestamp" : "2020-11-14T21:15:26.5-08:00",
          "state" : "Open",
          "numberOfEntriesSinceFirstNotAckedMessage" : 0,
          "totalNonContiguousDeletedMessagesRange" : 0,
          "properties" : { }
        }
      },
      "compactedLedger" : {
        "ledgerId" : -1,
        "entries" : -1,
        "size" : -1,
        "offloaded" : false
      }
    },

Modifications

Verifying this change

This change added tests and can be verified as follows:

• Add unit test to verify output expected range info.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): no
• The public API: no
• The schema: no
• The default values of configurations: no
• The wire protocol: no
• The rest endpoints: no
• The admin cli options: no
• Anything that affects deployment: no

Documentation

• Does this pull request introduce a new feature? no

[MarvinCai]

/pulsarbot run-failure-checks

[MarvinCai]

@codelipenghui can you please help take a look

[codelipenghui]

@MarvinCai I think it's better to add the consumer-related stats into the topics stats. The topics stats-internal expose the ledger and cursor stats, the topics stats expose the producer, subscription, and consumer stats. So topics stats seems more reasonable.

A consumer might be assigned to multiple hash range. So I think you can add a keyHashRanges in the consumer stats directly,

[MarvinCai]

Update to put this info in ConsumerStats and it's now part of topic-stats, example output:

{
  "msgRateIn" : 0.0,
  "msgThroughputIn" : 0.0,
  "msgRateOut" : 0.0,
  "msgThroughputOut" : 0.0,
  "bytesInCounter" : 0,
  "msgInCounter" : 0,
  "bytesOutCounter" : 0,
  "msgOutCounter" : 0,
  "averageMsgSize" : 0.0,
  "msgChunkPublished" : false,
  "storageSize" : 0,
  "backlogSize" : 0,
  "publishers" : [ ],
  "subscriptions" : {
    "zxcc-sub" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "bytesOutCounter" : 0,
      "msgOutCounter" : 0,
      "msgRateRedeliver" : 0.0,
      "chuckedMessageRate" : 0,
      "msgBacklog" : 0,
      "msgBacklogNoDelayed" : 0,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "msgDelayed" : 0,
      "unackedMessages" : 0,
      "type" : "Key_Shared",
      "msgRateExpired" : 0.0,
      "lastExpireTimestamp" : 0,
      "lastConsumedFlowTimestamp" : 1605508943123,
      "lastConsumedTimestamp" : 0,
      "lastAckedTimestamp" : 0,
      "consumers" : [ {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chuckedMessageRate" : 0.0,
        "consumerName" : "97a6d",
        "availablePermits" : 1000,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "179:0",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRange" : [ "32769--65536" ],
        "metadata" : { },
        "connectedSince" : "2020-11-15T22:42:18.625-08:00",
        "clientVersion" : "2.7.0-SNAPSHOT",
        "address" : "/127.0.0.1:58473"
      }, {
        "msgRateOut" : 0.0,
        "msgThroughputOut" : 0.0,
        "bytesOutCounter" : 0,
        "msgOutCounter" : 0,
        "msgRateRedeliver" : 0.0,
        "chuckedMessageRate" : 0.0,
        "consumerName" : "eb10d",
        "availablePermits" : 1000,
        "unackedMessages" : 0,
        "avgMessagesPerEntry" : 1000,
        "blockedConsumerOnUnackedMsgs" : false,
        "readPositionWhenJoining" : "179:0",
        "lastAckedTimestamp" : 0,
        "lastConsumedTimestamp" : 0,
        "keyHashRange" : [ "0--32768" ],
        "metadata" : { },
        "connectedSince" : "2020-11-15T22:42:23.118-08:00",
        "clientVersion" : "2.7.0-SNAPSHOT",
        "address" : "/127.0.0.1:58496"
      } ],
      "isDurable" : true,
      "isReplicated" : false,
      "consumersAfterMarkDeletePosition" : { }
    },
    "zxc-sub" : {
      "msgRateOut" : 0.0,
      "msgThroughputOut" : 0.0,
      "bytesOutCounter" : 0,
      "msgOutCounter" : 0,
      "msgRateRedeliver" : 0.0,
      "chuckedMessageRate" : 0,
      "msgBacklog" : 0,
      "msgBacklogNoDelayed" : 0,
      "blockedSubscriptionOnUnackedMsgs" : false,
      "msgDelayed" : 0,
      "unackedMessages" : 0,
      "msgRateExpired" : 0.0,
      "lastExpireTimestamp" : 0,
      "lastConsumedFlowTimestamp" : 0,
      "lastConsumedTimestamp" : 0,
      "lastAckedTimestamp" : 0,
      "consumers" : [ ],
      "isDurable" : true,
      "isReplicated" : false,
      "consumersAfterMarkDeletePosition" : { }
    }
  },
  "replication" : { },
  "deduplicationStatus" : "Disabled"
}

[codelipenghui]

The change looks good to me.

Is it better to use "[0, 65535]" to instead ''0--65535"? "[" or "]" usually understood to include. Of cause, if use "[]" the output topic stats looks like the followings

keyHashRanges: [[0, 123], [124, 65535]]

This is more like individuallyDeletedMessages in the topic stats-internal. This might be a good choice to keep consistent with individuallyDeletedMessages

[MarvinCai]

make sense, the key range is more like interval and it's better we clearly specify if it's open or closed interval, updated the string format.

[MarvinCai]

/pulsarbot run-failure-checks

[MarvinCai]

/pulsarbot run-failure-checks