[WIP]Record unused schema ledgers to ZK for partitioned topics - #8258
Closed
BewareMyPower wants to merge 2 commits into
Closed
[WIP]Record unused schema ledgers to ZK for partitioned topics#8258BewareMyPower wants to merge 2 commits into
BewareMyPower wants to merge 2 commits into
Conversation
Contributor
Author
|
/pulsarbot run-failure-checks |
Contributor
Author
|
/pulsarbot run-failure-checks |
1 task
3 tasks
Contributor
Author
|
Close this PR first because it's a temporary solution for current schema design. After discussing with @codelipenghui , the schema storage design should change, then this PR may not be needed. |
BewareMyPower
deleted the
bewaremypower/record-unused-schema-ledgers
branch
September 16, 2022 07:41
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When creating schema for a partitioned topic, e.g.
my-topic, only one z-node/schemas/public/default/my-topicwould be created to record the schema's metadata. However, multiple ledgers would be created because each single partition would try to create the schema: first append a ledger, then write the same z-node.#2959 and #6683 solved the race condition of accessing the same z-node by calling
putSchema()again. However, the unused ledgers of other partitions were created and were not recorded in the z-node. So, there's no way to locate these ledgers from ZK, which means there's no way to delete a topic's all associated schema ledgers.Modifications
/schemas/public/default/my-topic/unusedLedgers/0for topicmy-topicand ledger 0.unusedLedgersz-node has these ledgers' id and these ledgers exist, after a partitioned topic's schema was created.BookkeeperSchemaStorage.delete()delete ledgers found from ZK, instead of recording ledgers in memory.Verifying this change
This change added tests and can be verified as follows:
Run test
PartitionedTopicsSchemaTest.testUnusedLedgers.