Skip to content

[fix][broker] Fix inconsistent topic policy - #21231

Merged
codelipenghui merged 31 commits into
apache:masterfrom
mattisonchao:fix/topic_policy_not_apply
Sep 26, 2023
Merged

[fix][broker] Fix inconsistent topic policy#21231
codelipenghui merged 31 commits into
apache:masterfrom
mattisonchao:fix/topic_policy_not_apply

Conversation

@mattisonchao

@mattisonchao mattisonchao commented Sep 21, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

Motivation

#7817 Introduced topic-level policy for a managed-config with a bug. Which can lead to an unexpected data retention policy If the pulsar topic policy service isn't initialized.

Modifications

  • Introduced some new pure async methods
    • CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesAsync(@Nonnull TopicName topicName, boolean isGlobal);
    • CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesAsync(@Nonnull TopicName topicName);
  • Make BrokerService use pure async method.

Verifying this change

  • Make sure that the change passes the CI checks.

Alternatives

  • Support lazy get retention-related managed ledger configuration.
  • Use TopicPoliciesService#CompletableFuture<Optional<TopicPolicies>> getTopicPoliciesAsyncWithRetry(TopicName topicName, final Backoff backoff, ScheduledExecutorService scheduledExecutorService, boolean isGlobal) method.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@mattisonchao mattisonchao added type/bug The PR fixed a bug or issue reported a bug area/broker category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost labels Sep 21, 2023
@mattisonchao mattisonchao self-assigned this Sep 21, 2023
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Sep 21, 2023
@mattisonchao mattisonchao added this to the 3.2.0 milestone Sep 21, 2023
@mattisonchao mattisonchao mentioned this pull request Sep 21, 2023
Closed
5 tasks
@mattisonchao
mattisonchao marked this pull request as draft September 22, 2023 04:01
@mattisonchao

mattisonchao commented Sep 22, 2023

Copy link
Copy Markdown
Member Author

I need do some changes for fixing performance issue.

@mattisonchao
mattisonchao marked this pull request as ready for review September 22, 2023 07:47
@mattisonchao mattisonchao reopened this Sep 22, 2023
@coderzc

coderzc commented Sep 22, 2023

Copy link
Copy Markdown
Member

Since getTopicPoliciesAsync has been introduced, should we consider using getTopicPoliciesAsync instead of getTopicPoliciesAsyncWithRetry and getTopicPolicies?

@mattisonchao

mattisonchao commented Sep 22, 2023

Copy link
Copy Markdown
Member Author

@coderzc Yep, you are right. I will do it in another PR.

@mattisonchao
mattisonchao requested review from 315157973, Technoboy-, codelipenghui, coderzc and hangc0276 and removed request for coderzc September 22, 2023 08:56
This was referenced Sep 26, 2023
Merged
Merged
Merged
Merged
@asafm

asafm commented Sep 26, 2023

Copy link
Copy Markdown
Contributor

The methods before were not async?

@lhotari lhotari mentioned this pull request Oct 7, 2023
Closed
2 tasks
poorbarcode pushed a commit that referenced this pull request Oct 7, 2023
@mattisonchao @poorbarcode
[fix][broker] Fix inconsistent topic policy (#21231)
a0e2104 
(cherry picked from commit afc9244)
poorbarcode added a commit that referenced this pull request Oct 7, 2023
@poorbarcode
Revert "[fix][broker] Fix inconsistent topic policy (#21231)"
01dfd3e 
This reverts commit a0e2104.
@mattisonchao

mattisonchao commented Oct 8, 2023

Copy link
Copy Markdown
Member Author

The methods before were not async?

Yes, it was used backoff to implement an async-like approach.

liangyuanpeng pushed a commit to liangyuanpeng/pulsar that referenced this pull request Oct 11, 2023
@mattisonchao @liangyuanpeng
[fix][broker] Fix inconsistent topic policy (apache#21231)
3af4ed6
Technoboy- pushed a commit that referenced this pull request Oct 19, 2023
@mattisonchao @Technoboy-
[fix][broker] Fix inconsistent topic policy (#21231)
2ed9a76
@mattisonchao mattisonchao mentioned this pull request Oct 26, 2023
Merged
5 tasks
codelipenghui pushed a commit that referenced this pull request Nov 8, 2023
@mattisonchao
[fix][broker] Fix namespace bundle stuck in unloading status (#21445)
428c18c 
### Motivation

PR #21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`.

It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout.


### Modifications

- Get the topic policy before loading.
Technoboy- pushed a commit that referenced this pull request Nov 10, 2023
@mattisonchao @Technoboy-
[fix][broker] Fix namespace bundle stuck in unloading status (#21445)
d39482f 
PR #21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`.

It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout.

- Get the topic policy before loading.
nborisov pushed a commit to nborisov/pulsar that referenced this pull request Nov 13, 2023
@mattisonchao Nikolai Borisov
[fix][broker] Fix namespace bundle stuck in unloading status (apache#…
faddf4d 
…21445)

### Motivation

PR apache#21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`.

It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout.


### Modifications

- Get the topic policy before loading.
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
@mattisonchao @nikhil-ctds
[fix][broker] Fix inconsistent topic policy (apache#21231)
3f9c5b6
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
@mattisonchao @nikhil-ctds
[fix][broker] Fix namespace bundle stuck in unloading status (apache#…
c57d9fb 
…21445)

PR apache#21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`.

It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout.

- Get the topic policy before loading.
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
@mattisonchao @srinath-ctds
[fix][broker] Fix inconsistent topic policy (apache#21231)
40e4dc2
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
@mattisonchao @srinath-ctds
[fix][broker] Fix namespace bundle stuck in unloading status (apache#…
20c1ff8 
…21445)

PR apache#21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`.

It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout.

- Get the topic policy before loading.
@Demogorgon314 Demogorgon314 mentioned this pull request Feb 20, 2024
Merged
15 tasks
bahetimansi pushed a commit to bahetimansi/pulsar that referenced this pull request Mar 23, 2025
@poorbarcode @bahetimansi
Revert "[fix][broker] Fix inconsistent topic policy (apache#21231)"
bd6a6bf 
This reverts commit a0e2104.

(cherry picked from commit 01dfd3e)
@TakaHiR07 TakaHiR07 mentioned this pull request Nov 13, 2025
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hangc0276 hangc0276 hangc0276 approved these changes
@codelipenghui codelipenghui codelipenghui approved these changes
@poorbarcode poorbarcode poorbarcode approved these changes
@coderzc coderzc coderzc approved these changes
@315157973 315157973 Awaiting requested review from 315157973
@Technoboy- Technoboy- Awaiting requested review from Technoboy-

Assignees

@mattisonchao mattisonchao

Labels

area/broker category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost cherry-picked/branch-2.10 cherry-picked/branch-2.11 cherry-picked/branch-3.0 cherry-picked/branch-3.1 doc-not-needed Your PR changes do not impact docs ready-to-test release/2.10.6 release/2.11.3 release/3.0.2 release/3.1.2 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

3.2.0

Development

Successfully merging this pull request may close these issues.

9 participants

@mattisonchao @coderzc @codecov-commenter @asafm @hangc0276 @codelipenghui @poorbarcode @Technoboy- @shibd