[Enhancement]Added a metric for geo replication for tracking replicated subscriptions snapshot timeouts

[nikam14]

Fixes #21793

Motivation

Geo replication replicated subscriptions (PIP-33) snapshot creation might time out.
The code contains a debug log message when this happens
When this happens, the subscription state won't be reflected on the remote side and a backlog would build up.
There's no metric to detect this situation.

Modifications

Add a new metric pulsar_replicated_subscriptions_snapshot_timeouts which is a counter (that only resets when the broker restarts).

Verifying this change

• Make sure that the change passes the CI checks.

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[github-actions]

@nikam14 Please add the following content to your PR description and select a checkbox:

- [ ] `doc` <!-- Your PR contains doc changes -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [ ] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->

[dao-jun]

1. it's better to use OpenTelemetry
2. we need a proposal

[lhotari]

1. it's better to use OpenTelemetry
2. we need a proposal

@dao-jun we don't have Otel in use yet. Yes, we can handle this in a proposal.

[lhotari]

LGTM. I think that the lack of this metric could be considered a significant problem is replicated subscription observability and should be added to LTS version.

[lhotari]

Thanks for the contribution @nikam14 !

[codelipenghui]

Could you please help add an unit test to avoid the regression?

[asafm]

FYI @dragosvictor

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 75.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 74.33%. Comparing base (bbc6224) to head (5b69685).
Report is 662 commits behind head on master.

Files with missing lines	Patch %	Lines
.../persistent/ReplicatedSubscriptionsController.java	75.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #22381      +/-   ##
============================================
+ Coverage     73.57%   74.33%   +0.75%     
- Complexity    32624    34963    +2339     
============================================
  Files          1877     1952      +75     
  Lines        139502   147139    +7637     
  Branches      15299    16197     +898     
============================================
+ Hits         102638   109369    +6731     
- Misses        28908    29335     +427     
- Partials       7956     8435     +479     

Flag	Coverage Δ
inttests	27.58% <0.00%> (+2.99%)	⬆️
systests	24.36% <0.00%> (+0.04%)	⬆️
unittests	73.69% <75.00%> (+0.84%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../persistent/ReplicatedSubscriptionsController.java	70.54% <75.00%> (-2.05%)	⬇️

... and 634 files with indirect coverage changes