[fix][meta][branch-4.0] Tolerate concurrent creation of the underreplication LAYOUT node - #26248
Merged
nodece merged 1 commit intoJul 27, 2026
Conversation
…ication LAYOUT node ### Motivation `PulsarLedgerUnderreplicationManager.checkLayout()` creates `<ledgersRoot>/underreplication/LAYOUT` with a check-then-act sequence: it tests `store.exists(layoutPath)` and, when the node is absent, writes it with expected version `-1` (create-if-absent). When several managers are constructed concurrently against the same metadata store, they all observe the node as absent and all attempt the create. Every loser receives a `BadVersionException`, which the bare `join()` propagated out of the constructor instead of retrying. Reported in apache#16455 as a bookie failing to start when several bookies are started at once: java.util.concurrent.CompletionException: MetadataStoreException$BadVersionException: KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /ledgers/underreplication/LAYOUT at ZKMetadataStore.handlePutResult(ZKMetadataStore.java:210) The same race makes `AuditorPeriodicBookieCheckTest` flaky on branch-4.0, where an `AuditorElector` and the test's own `LedgerUnderreplicationManager` race to create the node. ### Modifications Ignore `BadVersionException` from the create and let the enclosing `while (true)` loop run again: the next iteration observes the node, validates the layout type and version, and breaks. Any other failure is still rethrown. This ports the `checkLayout()` change already present on master and branch-4.2, where it landed as part of apache#25219 (PIP-454) rather than as a standalone fix, which is why it was never backported. branch-4.1 and branch-3.0 need the same change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes: #16455
Motivation
PulsarLedgerUnderreplicationManager.checkLayout()creates<ledgersRoot>/underreplication/LAYOUTwith a check-then-act sequence: it testsstore.exists(layoutPath)and, when the node is absent, writes it with expected version-1(create-if-absent).When several managers are constructed concurrently against the same metadata store, they
all observe the node as absent and all attempt the create. Exactly one wins; every loser
gets a
BadVersionException, which the barejoin()propagates out of the constructorinstead of retrying. The enclosing
while (true)loop was clearly meant to handle this,but it can never be reached.
This was reported in #16455 as a bookie failing to start when several bookies are
started at once:
The same race also makes
AuditorPeriodicBookieCheckTestflaky onbranch-4.0— theAuditorElectorstarted insetUp()and the test's ownLedgerUnderreplicationManagerrace to create the node:
Modifications
Ignore
BadVersionExceptionfrom the create and let the enclosingwhile (true)looprun again — the next iteration observes the node, validates the layout type and version,
and breaks. Any other failure is still rethrown.
This is byte-for-byte the change
masterandbranch-4.2carry, apart from theprotobuf builder expression, which differs on
branch-4.0only because of the laterBookKeeper 4.18 API migration (#25886). Keeping it identical avoids divergence between
the branches.
Verifying this change
This change is already covered by existing tests, such as
AuditorPeriodicBookieCheckTestand
LedgerUnderreplicationManagerTest, which exercisecheckLayout()— though onlyprobabilistically, since the defect is a race.
To verify it deterministically I wrote a throwaway test (not included in this PR, to keep
the backport identical to
master) that constructs 16 managers concurrently against onestore behind a
CyclicBarrier, run 5× against both ZooKeeper and MockZooKeeper:signature exactly:
ExecutionException -> CompletionException -> MetadataStoreException$BadVersionException -> KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /ledgers-.../underreplication/LAYOUTExisting tests on
branch-4.0with the fix applied:AuditorPeriodicBookieCheckTestLedgerUnderreplicationManagerTestPulsarLayoutManagerTestmvn checkstyle:checkonpulsar-metadata: 0 violations.Does this pull request potentially affect one of the following parts:
If the box was checked, please highlight the changes