Skip to content

[fix] [client] Messages lost due to TopicListWatcher reconnect - #21853

Merged
Technoboy- merged 12 commits into
apache:masterfrom
poorbarcode:fix/topic_watcher_2
Jan 8, 2024
Merged

[fix] [client] Messages lost due to TopicListWatcher reconnect#21853
Technoboy- merged 12 commits into
apache:masterfrom
poorbarcode:fix/topic_watcher_2

Conversation

@poorbarcode

@poorbarcode poorbarcode commented Jan 4, 2024

Copy link
Copy Markdown
Contributor

Motivation

Issue

  • Start a consumer subscribe with regexp public/default/*
  • Create a new topic public/default/tp-1
  • Send messages to the new topic
  • Consumers will not receive any messages, and the messages will be deleted due to there are no subscription.

Root cause

Case-1:

Time Thread start consumer Thread create topic
1 Start subscribe
2 Start to register TopicListWatcher Create new topic
3 Topic created
4 Since there are no listeners registered, there is no need to notice clients
5 TopicListWatcher registered, the consumer will not receive any messages from the new topic

Case-2:

Time Thread TopicListWatcher reconnect Thread create topic
1 Start reconnect
2 Create new topic
3 Topic created
4 Since there are no listeners registered, there is no need to notice clients
5 TopicListWatcher reconnected
6 The consumer will not receive any messages from the new topic

This issue only affects the releases >=3.1, because before 3.1 there is a supplementary mechanism scheduled recheckPatternTimeout, it makes the issues will not occur. This supplementary mechanism was removed at 3.1, see #20779

Modifications

  • After TopicListWatcher reconnected, re-check the topics list.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode self-assigned this Jan 4, 2024
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Jan 4, 2024
@poorbarcode poorbarcode added type/bug The PR fixed a bug or issue reported a bug category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost release/3.1.3 and removed doc-not-needed Your PR changes do not impact docs labels Jan 4, 2024
@poorbarcode poorbarcode added this to the 3.3.0 milestone Jan 4, 2024
@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Jan 4, 2024
RobertIndie
RobertIndie requested changes Jan 5, 2024
View reviewed changes
Comment thread pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java Outdated
Technoboy-
Technoboy- reviewed Jan 5, 2024
View reviewed changes
Comment thread pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java Outdated
private final CompletableFuture<TopicListWatcher> watcherFuture = new CompletableFuture<>();
protected NamespaceName namespaceName;
private volatile Timeout recheckPatternTimeout = null;
private volatile AtomicReference<Timeout> retryRecheckPatternTask = new AtomicReference<>();

@Technoboy- Technoboy- Jan 5, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

recheckPatternTask is better

lhotari
lhotari reviewed Jan 8, 2024
View reviewed changes
Comment thread pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/MockedPulsarServiceBaseTest.java Outdated
lhotari
lhotari approved these changes Jan 8, 2024
View reviewed changes

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Demogorgon314
Demogorgon314 reviewed Jan 8, 2024
View reviewed changes
Comment thread pulsar-broker/src/test/java/org/apache/pulsar/broker/auth/MockedPulsarServiceBaseTest.java Outdated
Comment thread pulsar-client/src/main/java/org/apache/pulsar/client/impl/PatternMultiTopicsConsumerImpl.java
@Technoboy-
Technoboy- merged commit 042e769 into apache:master Jan 8, 2024
Technoboy- pushed a commit that referenced this pull request Jan 8, 2024
@poorbarcode @Technoboy-
[fix] [client] Messages lost due to TopicListWatcher reconnect (#21853)
fd62df8
@Technoboy- Technoboy- modified the milestones: 3.3.0, 3.2.0 Jan 8, 2024
poorbarcode added a commit that referenced this pull request Jan 15, 2024
@poorbarcode
[fix] [client] Messages lost due to TopicListWatcher reconnect (#21853)
3fc7cb4 
(cherry picked from commit 042e769)
poorbarcode added a commit that referenced this pull request Jan 15, 2024
@poorbarcode
[fix] [client] Messages lost due to TopicListWatcher reconnect (#21853)
4ef5132 
(cherry picked from commit 042e769)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 1, 2024
@poorbarcode @mukesh-ctds
[fix] [client] Messages lost due to TopicListWatcher reconnect (apach…
de51eba 
…e#21853)

(cherry picked from commit 042e769)
(cherry picked from commit 3fc7cb4)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 6, 2024
@poorbarcode @mukesh-ctds
[fix] [client] Messages lost due to TopicListWatcher reconnect (apach…
202dc05 
…e#21853)

(cherry picked from commit 042e769)
(cherry picked from commit 3fc7cb4)
poorbarcode added a commit that referenced this pull request Jul 11, 2024
@poorbarcode
[fix] [client] Messages lost due to TopicListWatcher reconnect (#21853)
9658163 
(cherry picked from commit 042e769)
@shibd shibd mentioned this pull request Mar 21, 2025
Merged
4 tasks
@lhotari lhotari mentioned this pull request Nov 7, 2025
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Technoboy- Technoboy- Technoboy- approved these changes
@Demogorgon314 Demogorgon314 Demogorgon314 approved these changes
@lhotari lhotari lhotari approved these changes
@RobertIndie RobertIndie RobertIndie approved these changes
@coderzc coderzc coderzc approved these changes
@codelipenghui codelipenghui Awaiting requested review from codelipenghui
@shibd shibd Awaiting requested review from shibd
@liangyepianzhou liangyepianzhou Awaiting requested review from liangyepianzhou
@BewareMyPower BewareMyPower Awaiting requested review from BewareMyPower
@heesung-sohn heesung-sohn Awaiting requested review from heesung-sohn

Assignees

@poorbarcode poorbarcode

Labels

category/reliability The function does not work properly in certain specific environments or failures. e.g. data lost cherry-picked/branch-2.11 cherry-picked/branch-3.0 cherry-picked/branch-3.1 doc-not-needed Your PR changes do not impact docs release/2.11.5 release/3.0.3 release/3.1.3 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

3.2.0

Development

Successfully merging this pull request may close these issues.

6 participants

@poorbarcode @lhotari @Technoboy- @RobertIndie @Demogorgon314 @coderzc