[fix][broker] Fix namespace bundle stuck in unloading status by mattisonchao · Pull Request #21445 · apache/pulsar

mattisonchao · 2023-10-26T02:14:05Z

Motivation

PR #21231 made user topic creation rely on system topic __change_event if the user is enabling topicLevelPoliciesEnabled.

It will introduce a race condition with namespace bundle unloading. All creating topics want to create __change_event reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout.

Modifications

Get the topic policy before loading.

Verifying this change

Make sure that the change passes the CI checks.

Documentation

doc
doc-required
doc-not-needed
doc-complete

Demogorgon314 · 2023-11-06T08:44:34Z

-                                });
-                    }
-                    return loadOrCreatePersistentTopic(tpName, createIfMissing, properties);
+                final CompletableFuture<Optional<TopicPolicies>> topicPoliciesFuture =


Do we have a test to cover the getTopic method and the TopicPolicies are loaded?

codelipenghui

The change looks good.
Just left a few minor comments.

codecov-commenter · 2023-11-08T02:37:22Z

Codecov Report

Attention: Patch coverage is 94.33962% with 9 lines in your changes missing coverage. Please review.

Project coverage is 73.24%. Comparing base (eebd821) to head (3bda31b).
Report is 1377 commits behind head on master.

Files with missing lines	Patch %	Lines
...rg/apache/pulsar/broker/service/BrokerService.java	95.45%	5 Missing and 2 partials ⚠️
.../service/SystemTopicBasedTopicPoliciesService.java	60.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #21445      +/-   ##
============================================
+ Coverage     73.23%   73.24%   +0.01%     
+ Complexity    32674    32648      -26     
============================================
  Files          1892     1892              
  Lines        140632   140645      +13     
  Branches      15467    15467              
============================================
+ Hits         102986   103010      +24     
+ Misses        29565    29519      -46     
- Partials       8081     8116      +35

Flag	Coverage Δ
inttests	`24.14% <69.18%> (-0.03%)`	⬇️
systests	`24.86% <64.77%> (+0.13%)`	⬆️
unittests	`72.52% <94.33%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
.../service/SystemTopicBasedTopicPoliciesService.java	`73.10% <60.00%> (-1.13%)`	⬇️
...rg/apache/pulsar/broker/service/BrokerService.java	`80.89% <95.45%> (-0.04%)`	⬇️

... and 89 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

PR #21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`. It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout. - Get the topic policy before loading.

…21445) ### Motivation PR apache#21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`. It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout. ### Modifications - Get the topic policy before loading.

…#21565) Cherry-pick #21445 to branch-3.0

…#21567) Co-authored-by: Jiwe Guo <technoboy@apache.org>

…#21566)

…21445) (apache#21565) Cherry-pick apache#21445 to branch-3.0

…21445) (apache#21567) Co-authored-by: Jiwe Guo <technoboy@apache.org>

…21445) PR apache#21231 made user topic creation rely on system topic `__change_event` if the user is enabling `topicLevelPoliciesEnabled`. It will introduce a race condition with namespace bundle unloading. All creating topics want to create `__change_event` reader but constantly fail by namespace-bundle inactive and retry mechanism. Unfortunately, the namespace-bundle unloading operation is waiting for all the topics to be completed and then release inactive status. Therefore, they will be stuck in a deadlock until one gets a timeout. - Get the topic policy before loading.

…licies before loading topics ### Motivation apache#21445 introduces an implicit requirement that if the topic policies cannot be loaded, `BrokerService#getTopic` will fail and then the client will retry loading the topic. It could break some existing usages like the tests in the C++ client: apache/pulsar-client-cpp#394 This change is applied only to avoid the race condition when unloading a namespace bundle. However, the client should not fail if the topic-level policies are not available. ### Modifications If the topic policies cannot be loaded due to a non-retryable error, we should not fail the `getTopic`. For retryable errors, the client will still retry until the broker gets the topic-level policies successfully after some attempts. Modify `TokenAuthenticatedProducerConsumerTest` to protect the change.

mattisonchao self-assigned this Oct 26, 2023

mattisonchao marked this pull request as draft October 26, 2023 02:14

mattisonchao added the area/broker label Oct 26, 2023

mattisonchao added this to the 3.2.0 milestone Oct 26, 2023

mattisonchao added release/2.11.3 release/2.10.6 release/3.1.2 release/3.0.3 labels Oct 26, 2023

mattisonchao changed the title ~~[fix][broker] Fix bundle stuck in unloading status~~ [fix][broker] Fix namespace bundle stuck in unloading status Oct 26, 2023

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Oct 26, 2023

mattisonchao added the category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost label Oct 26, 2023

shibd added release/2.11.4 and removed release/2.11.3 labels Oct 26, 2023

mattisonchao added the release/blocker Indicate the PR or issue that should block the release until it gets resolved label Oct 26, 2023

shibd added release/2.11.3 and removed release/2.11.4 labels Oct 26, 2023

mattisonchao marked this pull request as ready for review October 26, 2023 10:12

mattisonchao added the ready-to-test label Oct 26, 2023

mattisonchao closed this Oct 26, 2023

mattisonchao reopened this Oct 26, 2023

codelipenghui reviewed Oct 26, 2023

View reviewed changes

Comment thread pulsar-broker/src/test/java/org/apache/pulsar/broker/namespace/NamespaceUnloadingTest.java

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java Outdated

lhotari reviewed Nov 1, 2023

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/namespace/NamespaceService.java Outdated

mattisonchao marked this pull request as draft November 1, 2023 23:04

mattisonchao marked this pull request as ready for review November 2, 2023 11:14

mattisonchao closed this Nov 2, 2023

mattisonchao reopened this Nov 2, 2023

Demogorgon314 reviewed Nov 6, 2023

View reviewed changes

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java Outdated

Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java

Demogorgon314 reviewed Nov 6, 2023

View reviewed changes

Demogorgon314 approved these changes Nov 6, 2023

View reviewed changes

codelipenghui reviewed Nov 6, 2023

View reviewed changes

Fix checkstyle

e746b74

Technoboy- approved these changes Nov 7, 2023

View reviewed changes

Fix test

4edee21

poorbarcode approved these changes Nov 7, 2023

View reviewed changes

mattisonchao added 2 commits November 8, 2023 08:23

Fix falky test

4d45bff

Revert useless changes

3bda31b

poorbarcode mentioned this pull request Nov 8, 2023

[fix] [broker] Fix thousands orphan PersistentTopic caused OOM #21540

Merged

4 tasks

codelipenghui merged commit 428c18c into apache:master Nov 8, 2023

Technoboy- added the cherry-picked/branch-3.1 label Nov 10, 2023

poorbarcode added release/3.0.2 and removed release/3.0.3 labels Nov 10, 2023

mattisonchao deleted the fix/unloading_stuck branch November 10, 2023 13:12

mattisonchao mentioned this pull request Nov 13, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (#21445) #21565

Merged

1 task

mattisonchao added the cherry-picked/branch-3.0 label Nov 13, 2023

mattisonchao mentioned this pull request Nov 13, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (#21445) #21566

Merged

1 task

mattisonchao added the cherry-picked/branch-2.11 label Nov 13, 2023

mattisonchao mentioned this pull request Nov 13, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (#21445) #21567

Merged

1 task

poorbarcode pushed a commit that referenced this pull request Nov 14, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (#21445) (…

6b84514

…#21565) Cherry-pick #21445 to branch-3.0

Technoboy- added a commit that referenced this pull request Nov 16, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (#21445) (…

3431d2a

…#21567) Co-authored-by: Jiwe Guo <technoboy@apache.org>

mattisonchao added a commit that referenced this pull request Nov 20, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (#21445) (…

f59093c

…#21566)

coderzc pushed a commit to coderzc/pulsar that referenced this pull request Nov 21, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (apache#…

20264f2

…21445) (apache#21565) Cherry-pick apache#21445 to branch-3.0

nodece pushed a commit to ascentstream/pulsar that referenced this pull request Dec 2, 2023

[fix][broker] Fix namespace bundle stuck in unloading status (apache#…

7a488c2

…21445) (apache#21567) Co-authored-by: Jiwe Guo <technoboy@apache.org>

Technoboy- added the cherry-picked/branch-2.10 label Dec 4, 2023

BewareMyPower mentioned this pull request Feb 2, 2024

[fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics #22014

Closed

4 tasks

Navigation Menu

Uh oh!

[fix][broker] Fix namespace bundle stuck in unloading status - #21445

[fix][broker] Fix namespace bundle stuck in unloading status#21445
codelipenghui merged 12 commits into
apache:masterfrom
mattisonchao:fix/unloading_stuck

mattisonchao commented Oct 26, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Demogorgon314 Nov 6, 2023

Uh oh!

codelipenghui left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Nov 8, 2023 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Uh oh!

Conversation

mattisonchao commented Oct 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Verifying this change

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Demogorgon314 Nov 6, 2023

Choose a reason for hiding this comment

Uh oh!

codelipenghui left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov-commenter commented Nov 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

mattisonchao commented Oct 26, 2023 •

edited

Loading

codecov-commenter commented Nov 8, 2023 •

edited

Loading