Skip to content

[fix][broker] Fix calculate avg message per entry - #17046

Merged
codelipenghui merged 2 commits into
apache:masterfrom
coderzc:fix_avgMessagesPerEntry
Aug 17, 2022
Merged

[fix][broker] Fix calculate avg message per entry#17046
codelipenghui merged 2 commits into
apache:masterfrom
coderzc:fix_avgMessagesPerEntry

Conversation

@coderzc

@coderzc coderzc commented Aug 10, 2022

Copy link
Copy Markdown
Member

Motivation

Calculating avg message per entry using entries.size() is incorrect, because the entries may have null values.

Modifications

Calculate avg message per entry using the number of filter-out Non-null value total entries.

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@github-actions github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Aug 10, 2022
Technoboy-
Technoboy- reviewed Aug 11, 2022
View reviewed changes
Comment thread pulsar-broker/src/main/java/org/apache/pulsar/broker/service/Consumer.java
// Note
// Must ensure that the message is written to the pendingAcks before sent is first,
// because this consumer is possible to disconnect at this time.
if (pendingAcks != null) {

@Technoboy- Technoboy- Aug 11, 2022

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why move pendingAcks here ?

@coderzc coderzc Aug 11, 2022 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sure the loop is executed.

@coderzc
coderzc force-pushed the fix_avgMessagesPerEntry branch from 21e1859 to bfbfc3d Compare August 11, 2022 02:20
@coderzc
coderzc requested a review from Technoboy- August 11, 2022 02:22
@Technoboy- Technoboy- added this to the 2.11.0 milestone Aug 11, 2022
gaoran10
gaoran10 approved these changes Aug 12, 2022
View reviewed changes

@gaoran10 gaoran10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but I'm not sure why the entry could be null. Do you know the reason?

@coderzc

coderzc commented Aug 12, 2022

Copy link
Copy Markdown
Member Author

LGTM, but I'm not sure why the entry could be null. Do you know the reason?

Because some entries were set to null when the message was previously filtered.
Please see:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/AbstractBaseDispatcher.java

Line 98 in b1a29b5

public int filterEntriesForConsumer(Optional<MessageMetadata[]> optMetadataArray, int startOffset,

codelipenghui
codelipenghui reviewed Aug 13, 2022
View reviewed changes

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@coderzc The change looks good. Could you please help add a test? to avoid the regression in the future.

@coderzc
coderzc force-pushed the fix_avgMessagesPerEntry branch from 41efd83 to bfbfc3d Compare August 13, 2022 09:53
@coderzc

coderzc commented Aug 13, 2022

Copy link
Copy Markdown
Member Author

@coderzc The change looks good. Could you please help add a test? to avoid the regression in the future.

ok, I have already added a test.

eolivelli
eolivelli approved these changes Aug 13, 2022
View reviewed changes

@eolivelli eolivelli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

Good catch!

Please cherry pick to 2.10 branch

@eolivelli

eolivelli commented Aug 13, 2022

Copy link
Copy Markdown
Contributor

I don't think we need this in 2.9, filters don't exist there

@Technoboy-

Technoboy- commented Aug 15, 2022

Copy link
Copy Markdown
Contributor

I don't think we need this in 2.9, filters don't exist there

We need to cherry pick to 2.9 and 2.8, the same issue in sendMessages

@codelipenghui
codelipenghui merged commit 2c2b75e into apache:master Aug 17, 2022
Technoboy- pushed a commit that referenced this pull request Aug 17, 2022
@coderzc @Technoboy-
[fix][broker] Fix calculate avg message per entry (#17046)
72810a0
@mattisonchao

mattisonchao commented Aug 25, 2022

Copy link
Copy Markdown
Member

Hi, @coderzc
Would you like to push a PR to branch-2.9 to fix this problem?

@coderzc

coderzc commented Aug 25, 2022

Copy link
Copy Markdown
Member Author

Hi, @coderzc Would you like to push a PR to branch-2.9 to fix this problem?

Yes, I will push a PR to branch-2.9.

@coderzc coderzc mentioned this pull request Aug 25, 2022
Merged
4 tasks
@mattisonchao mattisonchao added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Sep 1, 2022
Jason918 pushed a commit that referenced this pull request Sep 4, 2022
@coderzc @Jason918
[fix][broker] Fix calculate avg message per entry (#17046)
4fda75b 
(cherry picked from commit 2c2b75e)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Sep 7, 2022
@coderzc @nicoloboschi
[fix][broker] Fix calculate avg message per entry (apache#17046)
ea426df 
(cherry picked from commit 2c2b75e)
@zymap

zymap commented Sep 15, 2022

Copy link
Copy Markdown
Member

Remove from release 2.8.5. Since the original code is introduced in 2.9.3.#14666

nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Sep 16, 2022
@coderzc @nicoloboschi
[fix][broker] Fix calculate avg message per entry (apache#17046)
241ada6 
(cherry picked from commit 2c2b75e)
(cherry picked from commit 4fda75b)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Technoboy- Technoboy- Technoboy- approved these changes
@eolivelli eolivelli eolivelli approved these changes
@AnonHxy AnonHxy AnonHxy approved these changes
@codelipenghui codelipenghui codelipenghui approved these changes
@gaoran10 gaoran10 gaoran10 approved these changes
@mattisonchao mattisonchao mattisonchao approved these changes
@BewareMyPower BewareMyPower Awaiting requested review from BewareMyPower

Assignees

@coderzc coderzc

Labels

area/broker cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 cherry-picked/branch-2.11 doc-not-needed Your PR changes do not impact docs release/2.9.4 release/2.10.2 type/bug The PR fixed a bug or issue reported a bug

Projects

None yet

Milestone

2.11.0

Development

Successfully merging this pull request may close these issues.

9 participants

@coderzc @eolivelli @Technoboy- @mattisonchao @zymap @AnonHxy @codelipenghui @gaoran10 @Jason918