[cherry-pick][branch-2.9] Fix issue where leader broker information isn't available after 10 minutes - #18393
Closed
congbobo184 wants to merge 4 commits into
Closed
Conversation
…ite (apache#12788) ### Motivation When we're doing a write to the store from outside the `MetadataCache`, we are immediately invalidating the cache to ensure read-after-write consistency through the cache. The only issue is that the invalidation, will not trigger a reloading of the value. Instead it is relying on the next call to `cache.get()` which will see the cache miss and it will load the new value into the cache. This means that calls `cache.getIfCached()`, which is not triggering a cache load, will keep seeing the key as missing. ### Modification Ensure we're calling refresh on the cache to get the value automatically reloaded in background and make sure the `getIfCached()` will eventually return the new value, even if there are no calls to `cache.get()`. (cherry picked from commit 2bc4499)
Co-authored-by: Jiang Haiting <jianghaiting@didichuxing.com> (cherry picked from commit 2b939b7)
(cherry picked from commit 2e16b43)
congbobo184
added
doc-not-needed
congbobo184
requested review from
Technoboy-,
codelipenghui,
lhotari and
liangyepianzhou
November 9, 2022 03:41
Member
|
Already cherry-picked without a PR |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes issue where leader broker information isn't available and this warning gets logged:
WARN org.apache.pulsar.broker.namespace.NamespaceService - The information about the current leader broker wasn't available. Handling load manager decisions in a decentralized way.Possibly fixing:
The root cause of the problem is that currently leader election cache entries expire in 10 minutes. Entries are refreshed when there's a cache hit between 5 to 10 minutes after the entry was added to the cache or refreshed the last time. This expiration behavior cannot be configured currently in MetadataCache.
Modifications
Since leader election cache expiration leads to problems, it is better to disable expiration completely for leader election entries. The cache entry refreshing is not changed so a missed Zookeeper change notification won't cause the cache to serve stale values.
MetadataStoreMetadataCacheso that expiration can be disabled for leader election cachecache.refreshinstead of removing entries from the cache.Additional context
The value for the leader broker is currently read intentionally in a non-blocking way. There's an alternative fix [fix][broker] Fix intermittent issue where leader broker information is lost and causes lookup issues #17254 which retrieves the value also in this case. That approach is abandoned since the correct solution is to not expire the Leader election cache at all. Thank you, @michaeljmarshall, for recommending this approach where Leader election cache isn't expired at all.
doc-not-needed