[fix][metadata] Fix orphaned UR parent nodes not cleaned up with Oxia metadata backend - #26158
Conversation
|
Do we need to change the ZK NOTEMPTY mapping in this PR? The Oxia issue should be fixable by having Oxia throw MetadataStoreException.NotEmptyException and by making markLedgerReplicated() ignore both the new Oxia exception and the existing ZK shape. Changing ZK here also changes the exception shape for existing callers. For example, NamespacesBase still checks for KeeperException.NotEmptyException as the cause to retry namespace deletion. Could we keep the ZK mapping unchanged in this PR, and handle both exception forms in markLedgerReplicated()? If we want to normalize ZK NOTEMPTY too, it may be safer to do that separately with updates to unwrap() and existing catch sites. |
…nd Oxia shapes in markLedgerReplicated
|
@void-ptr974 Good catch, thank you. Agreed on all points.
|
…for Etcd - #26158 hasn't been implemented for Etcd
…for Etcd - #26158 hasn't been implemented for Etcd
Fixes #26157
Motivation
When using Oxia as the metadata backend,
markLedgerReplicated()inPulsarLedgerUnderreplicationManagerleaves orphaned intermediate nodes in the under-replication (UR) tree after every ledger is successfully replicated.OxiaMetadataStore.doStorePut()callscreateParents()to explicitly create up to 4 intermediate path entries for every UR leaf (Oxia is a flat key-value store; parent nodes must be created explicitly). However, the hierarchy cleanup inmarkLedgerReplicated()was gated on aninstanceofcheck that excludedOxiaMetadataStore, so those parent nodes were never deleted.Additionally, the inner catch block only handled ZK's
KeeperException.NotEmptyException. Oxia threw a plainMetadataStoreExceptionfor the same "node has children" condition, causing a spuriouslog.warn()on every successfully-replicated ledger.Over time, orphaned nodes accumulate proportionally to replication activity and cause
getLedgerToRereplicateFromHierarchy()to traverse empty subtrees on every replication poll, driving sustained elevated metadata store list ops.Modifications
MetadataStoreException.NotEmptyExceptionto the metadata API, consistent with the existingNotFoundException/BadVersionExceptionpatternNOTEMPTYerror code toNotEmptyExceptioninZKMetadataStore.getException()(was falling through todefault)NotEmptyExceptionfromOxiaMetadataStore.storeDelete()instead of a plainMetadataStoreException, so both backends surface the same exception typeOxiaMetadataStoreto theinstanceofguard inmarkLedgerReplicated()KeeperException.NotEmptyExceptioncheck with a cleaninstanceof MetadataStoreException.NotEmptyExceptioncheckVerifying this change
This change added tests and can be verified as follows:
testMarkReplicatedDeletesEmptyParentNodesto run against all distributed metadata backends (dataProvider = "distributedImpl": ZK + Oxia) instead of ZK-only (dataProvider = "zkImpls"). The test asserts that all 4 parent hierarchy nodes are deleted aftermarkLedgerReplicated()completes.Does this pull request potentially affect one of the following parts:
MetadataStoreException.NotEmptyException(new subclass, backward-compatible)