Add a cache eviction policy：Evicting cache data by the slowest markDeletedPosition by lordcheng10 · Pull Request #14985 · apache/pulsar

lordcheng10 · 2022-04-01T09:54:55Z

Motivation

We found a problem: even if we adjusted the cache eviction size and time a lot, there would still be some misscaches when the client restarted.The corresponding configuration is as follows:
managedLedgerCacheSizeMB=40960
managedLedgerCacheEvictionTimeThresholdMillis=1800000

The current cache eviction strategy is to evict according to the read position, so there may be a miss cache when the data is pushed again.

In our scenario, there are tens of thousands of clients, and they are deployed on k8s. In order to save resources, they may dynamically expand or shrink, so that data re-pushing will happen from time to time:

Modifications

Therefore, to improve the cache hit rate , I think a new cache eviction strategy should be added: evict data according to the markdelete position.

Documentation

Check the box below or label this PR directly (if you have committer privilege).

Need to update docs?

doc-required

(If you need help on updating docs, create a doc issue)
no-need-doc

(Please explain why)
doc

(If this PR contains doc changes)

lordcheng10 · 2022-04-01T11:08:14Z

@eolivelli @Technoboy- @codelipenghui
PTAL,thanks!

eolivelli

Make sense, especially if you have some numbers to share.

Can you please add a test case ?

cc @merlimat @rdhabalia WDTY?

lordcheng10 · 2022-04-02T03:10:52Z

Make sense, especially if you have some numbers to share.

Can you please add a test case ?

cc @merlimat @rdhabalia WDTY?

OK ,I will fix!

Jason918

Make sense. IMO, markDeletedPosition is better than readPosition in normal case. Maybe we can use markDeletedPosition by default, and use readPosition iff overall cache usage is over some threshold (like 85%)?

lordcheng10 · 2022-04-02T03:50:55Z

markDeletedPosition is better than readPosition in normal case. Maybe we can use markDeletedPosition by default.

OK, I aggree.

aloyszhang · 2022-04-02T03:52:25Z

Make sense to me.
A test to cover this is better.

lordcheng10 · 2022-04-02T03:53:44Z

use readPosition iff overall cache usage is over some threshold (like 85%)?

In the scenario of message re-pushing, this way of dynamically adjusting the eviction policy is not suitable. I can reopen a discussion about dynamically adjusting the eviction policy later.

lordcheng10 · 2022-04-02T08:04:45Z

Make sense. IMO, markDeletedPosition is better than readPosition in normal case. Maybe we can use markDeletedPosition by default, and use readPosition iff overall cache usage is over some threshold (like 85%)?

Fixed. PTAL,thanks! @Jason918 @eolivelli @aloyszhang
1.add unit test;
2.fix evictionPos: use markDeletedPosition.next;

lordcheng10 · 2022-04-02T10:29:41Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T11:12:18Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T11:14:11Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T15:10:02Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T16:28:03Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T17:09:48Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T17:20:11Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T17:20:47Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T17:21:04Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T17:23:16Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-02T17:29:37Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-03T15:02:35Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-03T15:07:13Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-03T15:35:50Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-03T15:36:18Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-03T15:36:22Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-03T15:59:45Z

/pulsarbot run-failure-checks

lordcheng10 · 2022-04-05T16:36:29Z

@eolivelli PTAL,thanks!

lordcheng10 · 2022-04-06T04:35:07Z

ping

eolivelli

LGTM

…letedPosition (apache#14985)

…letedPosition (apache#14985) (cherry picked from commit 9b36dcd)

…t markDeletedPosition (apache#14985)" This reverts commit ee28018.

…letedPosition (apache#14985) (cherry picked from commit 9b36dcd)

…letedPosition (apache#14985)

…letedPosition (#14985) (cherry picked from commit 9b36dcd)

Evicting cache data by the slowest markDeletedPosition

afb3b9a

github-actions Bot assigned lordcheng10 Apr 1, 2022

github-actions Bot added the doc-not-needed Your PR changes do not impact docs label Apr 1, 2022

check style

d24a134

check style

5497474

eolivelli requested changes Apr 1, 2022

View reviewed changes

Jason918 approved these changes Apr 2, 2022

View reviewed changes

1.add unit test;

deeaec2

lordcheng10 requested a review from eolivelli April 2, 2022 08:05

1.use readposition by default;

4b976cd

aloyszhang approved these changes Apr 2, 2022

View reviewed changes

Jason918 reviewed Apr 2, 2022

View reviewed changes

Comment thread managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java Outdated

Comment thread managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerTest.java Outdated

use doCacheEviction instead of using sleep.

0afb77d

eolivelli approved these changes Apr 6, 2022

View reviewed changes

eolivelli merged commit 9b36dcd into apache:master Apr 6, 2022

lhotari mentioned this pull request Apr 6, 2022

lh fix race in persisting mark delete position lhotari/pulsar#64

Closed

codelipenghui added this to the 2.11.0 milestone Apr 6, 2022

codelipenghui added type/feature The PR added a new feature or issue requested a new feature area/broker labels Apr 6, 2022

aparajita89 pushed a commit to aparajita89/pulsar that referenced this pull request Apr 18, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

cf86dea

…letedPosition (apache#14985)

Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

dd69e00

…letedPosition (apache#14985)

This was referenced Jun 14, 2022

[ML] Fix thread safety issues in accessing ManagedCursorContainer.heap ArrayList #16049

Merged

Broker cache eviction evicts entries that haven't been read by active consumers #16054

Closed

sijie mentioned this pull request Jun 14, 2022

ISSUE-16054: Broker cache eviction evicts entries that haven't been read by active consumers streamnative/pulsar-archived#4375

Open

eolivelli mentioned this pull request Jul 7, 2022

Shared subscription + Batch messages: consumerFlow trigger reading the same messages from storage #16421

Closed

nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 7, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

ee28018

…letedPosition (apache#14985) (cherry picked from commit 9b36dcd)

nicoloboschi added a commit to datastax/pulsar that referenced this pull request Jul 8, 2022

Revert "Add a cache eviction policy：Evicting cache data by the slowes…

d0af0ac

…t markDeletedPosition (apache#14985)" This reverts commit ee28018.

nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 8, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

96666e1

…letedPosition (apache#14985) (cherry picked from commit 9b36dcd)

lhotari mentioned this pull request Aug 25, 2022

[fix][broker] Fix broker cache eviction of entries read by active cursors #17273

Merged

dragonls pushed a commit to dragonls/pulsar that referenced this pull request Oct 21, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

06114c5

…letedPosition (apache#14985)

congbobo184 pushed a commit that referenced this pull request Nov 17, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

f31a6fa

…letedPosition (#14985) (cherry picked from commit 9b36dcd)

congbobo184 pushed a commit that referenced this pull request Dec 7, 2022

Add a cache eviction policy：Evicting cache data by the slowest markDe…

7f12634

…letedPosition (#14985) (cherry picked from commit 9b36dcd)

liangyepianzhou mentioned this pull request Dec 19, 2022

[cherry-pick][branch-2.10]Fix broker cache eviction of entries read by active cursor #18980

Merged

14 tasks

liangyepianzhou added cherry-picked/branch-2.10 release/2.10.3 labels Dec 21, 2022

Uh oh!

Conversation

lordcheng10 commented Apr 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Documentation

Uh oh!

lordcheng10 commented Apr 1, 2022

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

Jason918 left a comment

Choose a reason for hiding this comment

Uh oh!

lordcheng10 commented Apr 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aloyszhang commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

Uh oh!

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 2, 2022

Uh oh!

lordcheng10 commented Apr 3, 2022

Uh oh!

lordcheng10 commented Apr 3, 2022

Uh oh!

lordcheng10 commented Apr 3, 2022

Uh oh!

lordcheng10 commented Apr 3, 2022

Uh oh!

lordcheng10 commented Apr 3, 2022

Uh oh!

lordcheng10 commented Apr 3, 2022

Uh oh!

lordcheng10 commented Apr 5, 2022

Uh oh!

lordcheng10 commented Apr 6, 2022

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lordcheng10 commented Apr 1, 2022 •

edited

Loading

lordcheng10 commented Apr 2, 2022 •

edited

Loading

lordcheng10 commented Apr 2, 2022 •

edited

Loading