[improve][broker] optimize the problem of subscription snapshot cache not hitting

[liudezhi2098]

Motivation

• When message acknowledgment confirmation is slower than message consumption rate, subscription cursor synchronization fails to complete. This occurs because:

1. Current Behavior
   • With large receiver queues (e.g., receiverQueueSize=1000), the cursor never synchronizes

    Consumer<String> consumer = client.newConsumer(Schema.STRING)
              .topic(topic)
              .subscriptionName("sub")
              .receiverQueueSize(1000)
               .subscribe();
     while (true) {
               Message<String> msg = consumer.receive();
                consumer.acknowledge(msg);
                Thread.sleep(100);
           }
   
   

   • With small queues (e.g., receiverQueueSize=1), synchronization works properly

    Consumer<String> consumer = client.newConsumer(Schema.STRING)
              .topic(topic)
              .subscriptionName("sub")
              .receiverQueueSize(1)
               .subscribe();
     while (true) {
               Message<String> msg = consumer.receive();
                consumer.acknowledge(msg);
                Thread.sleep(100);
           }
   
   

2. Root Cause:
   • The SnapshotCache updates too aggressively
   • When advancedMarkDeletePosition executes, valid cache entries are frequently unavailable
   • Current eviction strategy doesn't account for periodic synchronization needs
     https://github.com/apache/pulsar/blob/7be22eb2b23057bd5e09c361a43d6ccdcc0c8afd/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/ReplicatedSubscriptionSnapshotCache.java#L44C4-L59C6

Modifications

1. Cache Update Strategy:
   • modified the cache to maintain mapping relationships for remote clusters.
2. Eviction Policy Enhancement:
   • When cache reaches capacity (maxSnapshotToCache):
     • Allow subsequent snapshots to be added through periodic dynamic adjustment
     • The latest snapshot is used to replace the intermediate snapshot of the cache, and the update becomes slower as the difference between the latest snapshot time and the Mark Delete Position time increases.

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository:

[lhotari]

• When message acknowledgment confirmation is slower than message consumption rate, subscription cursor synchronization fails to complete.

@liudezhi2098 Regarding this scenario, is there a way to find out this happens?
Does the metric pulsar_replicated_subscriptions_timedout_snapshots added in #22381 help detecting problems?

[lhotari]

Great work @liudezhi2098! Added a comment about adding code comments. :)

[lhotari]

Since the publishTime information comes from Pulsar client, the logic could be brittle. In Pulsar, there's also an optional brokerPublishTime which is not enabled by default (requires specific configuration in all brokers).

Have you considered what could happen when publishTime values aren't in sync?

I've understood that the "PIP-33: Replicated subscriptions" algorithm rely on vector clocks so that clock sync doesn't become a problem. (earlier discussion)
The snapshots are the way how the vector clocks are synchronized, at least that's how I interpret it from one view point.

The changes in this PR don't currently make sense to me, mainly due to the use of publishTime.

I'd assume that in your problem scenario, the correct approach would be to tune replicatedSubscriptionsSnapshotFrequencyMillis, replicatedSubscriptionsSnapshotTimeoutSeconds and replicatedSubscriptionsSnapshotMaxCachedPerSubscription values.

pulsar/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

Lines 1444 to 1457 in 825b862

	@FieldContext(
	category = CATEGORY_SERVER,
	doc = "Frequency of snapshots for replicated subscriptions tracking.")
	private int replicatedSubscriptionsSnapshotFrequencyMillis = 1_000;
	@FieldContext(
	category = CATEGORY_SERVER,
	doc = "Timeout for building a consistent snapshot for tracking replicated subscriptions state. ")
	private int replicatedSubscriptionsSnapshotTimeoutSeconds = 30;
	@FieldContext(
	category = CATEGORY_SERVER,
	doc = "Max number of snapshot to be cached per subscription.")
	private int replicatedSubscriptionsSnapshotMaxCachedPerSubscription = 10;

Have you already done this?

Currently it is a problem that it's necessary to tune the values to fix issues and it's also hard to notice that the problem is occurring.

It looks like future improvements are needed too.

[lhotari]

Current Behavior

• With large receiver queues (e.g., receiverQueueSize=1000), the cursor never synchronizes

@liangyepianzhou Do you have a separate repro app where this could be observed with real brokers (let's say 2 Pulsar broker within docker-compose + some test app)? Creating a separate Git repo for such a repro would be one approach to share it. Having a runnable repro makes things easier for reviewers too.

[lhotari]

• The SnapshotCache updates too aggressively
• When advancedMarkDeletePosition executes, valid cache entries are frequently unavailable

@liudezhi2098 One thought here is that perhaps there could be interaction between ReplicatedSubscriptionsController and all ReplicatedSubscriptionSnapshotCache instances? Could there be a solution that when "updates too aggressively" that there's a solution in place that a snapshot would be completed every replicatedSubscriptionsSnapshotFrequencyMillis.
Since the ReplicatedSubscriptionSnapshotCache is an internal interface, we don't need to keep it as a "cache". It's possible that it doesn't make sense in the revisited solution.
Do you have a chance to try something in this area instead since I don't think that using publishTime in the solution makes sense.

[liudezhi2098]

Since the publishTime information comes from Pulsar client, the logic could be brittle. In Pulsar, there's also an optional brokerPublishTime which is not enabled by default (requires specific configuration in all brokers).

Have you considered what could happen when publishTime values aren't in sync?

I've understood that the "PIP-33: Replicated subscriptions" algorithm rely on vector clocks so that clock sync doesn't become a problem. (earlier discussion) The snapshots are the way how the vector clocks are synchronized, at least that's how I interpret it from one view point.

The changes in this PR don't currently make sense to me, mainly due to the use of publishTime.

I'd assume that in your problem scenario, the correct approach would be to tune replicatedSubscriptionsSnapshotFrequencyMillis, replicatedSubscriptionsSnapshotTimeoutSeconds and replicatedSubscriptionsSnapshotMaxCachedPerSubscription values.

pulsar/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java

Lines 1444 to 1457 in 825b862

	@FieldContext(
	category = CATEGORY_SERVER,
	doc = "Frequency of snapshots for replicated subscriptions tracking.")
	private int replicatedSubscriptionsSnapshotFrequencyMillis = 1_000;
	@FieldContext(
	category = CATEGORY_SERVER,
	doc = "Timeout for building a consistent snapshot for tracking replicated subscriptions state. ")
	private int replicatedSubscriptionsSnapshotTimeoutSeconds = 30;
	@FieldContext(
	category = CATEGORY_SERVER,
	doc = "Max number of snapshot to be cached per subscription.")
	private int replicatedSubscriptionsSnapshotMaxCachedPerSubscription = 10;

Have you already done this?

Currently it is a problem that it's necessary to tune the values to fix issues and it's also hard to notice that the problem is occurring.

It looks like future improvements are needed too.

@lhotari The generation of snapshots is completed through the exchange of snapshotRequest and snapshotResponse between two clusters. Ultimately, the ReplicatedSubscriptionsController writes Marker messages, and using publishTime is reliable because this behavior occurs within the same broker.

Of course, the topic may be transferred to another broker, but this is a low-frequency scenario, and its publishTime will not exhibit continuous jumps.

However, we can adopt a simpler approach that doesn't require using publishTime. Instead, we can record the current system time each time the snapshotCache，but there is a flaw that it cannot truly reflect the time difference between two messages. In some scenarios, it will cause the cache update frequency to decrease.

[lhotari]

The generation of snapshots is completed through the exchange of snapshotRequest and snapshotResponse between two clusters. Ultimately, the ReplicatedSubscriptionsController writes Marker messages, and using publishTime is reliable because this behavior occurs within the same broker.

Of course, the topic may be transferred to another broker, but this is a low-frequency scenario, and its publishTime will not exhibit continuous jumps.

However, we can adopt a simpler approach that doesn't require using publishTime. Instead, we can record the current system time each time the snapshotCache is updated and use this timestamp for dynamic adjustments.

Thanks for explaining that. I missed that point that the marker messages are originated from the same broker. publishTime would be fine due to that detail.

When looking at the current master branch code in ReplicatedSubscriptionSnapshotCache.addNewSnapshot, I'd assume that a potential solution to the problem could be that the current mark delete position is taken into account before purging entries.

It looks like the problem arrises when there's isn't at least one entry that is older than the current mark delete position in the cache.

I'd suggest to revisit the purging logic in this way:

• modify addNewSnapshot and add a 2nd parameter which is the current mark delete position
• always keep the newest entry that is before the current mark delete position when purging entries, all older entries can be purged
• if the cache remains full after after doing this, remove a single entry in the cache so that the new entry could be added.
  • keep the position of the last removed entry so that it's possible to continue the purging algorithm in subsequent calls
  • if there's no previous last removed entry, purge the 2nd entry (assuming that the first entry is the newest entry before current mark delete position)
  • if there's a previous last removed entry, continue purging from next entry after the last removed position by first skipping one entry and then removing the 2nd entry
  • if there's no more entries to remove, start removing from the beginning.

This purging logic should always result in making it possible to add a new entry. Since every 2nd entry is removed, it will result in "sampling" so that when the mark delete position finally advances, it advances to the most recent position.

There's a possibility to increase replicatedSubscriptionsSnapshotMaxCachedPerSubscription parameter to improve the resolution, if that's desirable.

Perhaps the intention of your timestamp based approach is already to achieve something similar?

[liudezhi2098]

When looking at the current master branch code in ReplicatedSubscriptionSnapshotCache.addNewSnapshot, I'd assume that a potential solution to the problem could be that the current mark delete position is taken into account before purging entries.

It looks like the problem arrises when there's isn't at least one entry that is older than the current mark delete position in the cache.

I'd suggest to revisit the purging logic in this way:

• modify addNewSnapshot and add a 2nd parameter which is the current mark delete position

• always keep the newest entry that is before the current mark delete position when purging entries, all older entries can be purged

• if the cache remains full after after doing this, remove a single entry in the cache so that the new entry could be added.

  • keep the position of the last removed entry so that it's possible to continue the purging algorithm in subsequent calls
  • if there's no previous last removed entry, purge the 2nd entry (assuming that the first entry is the newest entry before current mark delete position)
  • if there's a previous last removed entry, continue purging from next entry after the last removed position by first skipping one entry and then removing the 2nd entry
  • if there's no more entries to remove, start removing from the beginning.

This purging logic should always result in making it possible to add a new entry. Since every 2nd entry is removed, it will result in "sampling" so that when the mark delete position finally advances, it advances to the most recent position.

There's a possibility to increase replicatedSubscriptionsSnapshotMaxCachedPerSubscription parameter to improve the resolution, if that's desirable.

Perhaps the intention of your timestamp based approach is already to achieve something similar?

@lhotari The intention of timestamp based is to achieve this purpose，the key is when cache is full, how to update the cache,
there is no perfect algorithm to solve this problem.

I recommend using Median-based eviction for simplicity, try to make the cache an arithmetic progression in time, because for the shared mode subscription, there will be individual unconfirmed messages, presenting a very jumpy situation.

[lhotari]

The intention of timestamp based is to achieve this purpose，the key is when cache is full, how to update the cache, there is no perfect algorithm to solve this problem.

I recommend using Median-based eviction for simplicity, try to make the cache an arithmetic progression in time, because for the shared mode subscription, there will be individual unconfirmed messages, presenting a very jumpy situation.

You are right about this. The added comments in the code make it easier to understand the intention of the logic. My previous comment about taking the mark deletion position into account in adding snapshots didn't make much sense after rethinking.
I'll review again.

[lhotari]

Great work, mainly comments about comments. The 2nd rule in skipping to add entries, timeSinceLastSnapshot < timeWindowPerSlot, seems risky to add since it could have surprising consequences. I think it would be better to remove that.

[lhotari]

this is a bit hard to grasp, what the "time slot" concept is here.
Let's say, timeSinceFirstSnapshot is 25000 ms, snapshotFrequencyMillis is 1000ms and maxSnapshotToCache is 10, it would result in 2. What's the point of this?
With low values, this would be close to 0, I guess. This is also why I think this is just unnecessary complexity.

[liudezhi2098]

What if timeSinceFirstSnapshot is 25 minutes? The goal is that if timeSinceFirstSnapshot becomes longer, the update frequency should be lower.

[lhotari]

What if timeSinceFirstSnapshot is 25 minutes? The goal is that if timeSinceFirstSnapshot becomes longer, the update frequency should be lower.

This doesn't seem to be a realistic case. I disagree that the update frequency should become lower. That's exactly my point that if there's a long delay, it will be enforced going forward. I think it's easier to make progress in this PR by removing this rule and adding it later if there's a specific reason to do so.

[lhotari]

I recommend using Median-based eviction for simplicity, try to make the cache an arithmetic progression in time, because for the shared mode subscription, there will be individual unconfirmed messages, presenting a very jumpy situation.

I don't see how the median based eviction could make sense. After the cache is filled up, when the median entry is removed and a new entry is added, and this repeats, the result will be that only entries after the median entry will be evicted (assuming no other events happen in between). Eventually there will be a large gap between the 2 entries in the middle.

Since time is already considered in the algorithm, it seems that an alternative approach would be to evict the entry with the shortest time distance to it's adjacent entries. Makes sense?

[lhotari]

I recommend using Median-based eviction for simplicity, try to make the cache an arithmetic progression in time, because for the shared mode subscription, there will be individual unconfirmed messages, presenting a very jumpy situation.

I don't see how the median based eviction could make sense. After the cache is filled up, when the median entry is removed and a new entry is added, and this repeats, the result will be that only entries after the median entry will be evicted (assuming no other events happen in between). Eventually there will be a large gap between the 2 entries in the middle.

Since time is already considered in the algorithm, it seems that an alternative approach would be to evict the entry with the shortest time distance to it's adjacent entries. Makes sense?

@liudezhi2098 Just wondering if you are fine with the provided feedback on this PR? It would be great to address this issue in replicated subscriptions and get this PR to completion.

[lhotari]

@liudezhi2098 Are you planning to continue working on this? I think that this is a really great improvement to address a long time issue with replicated subscriptions.

[lhotari]

@liudezhi2098 There's also a long-standing issue #10054 which is addressed by #16651. I have updated PR 16651 description, rebased it and revisited it slightly. Please review

[liudezhi2098]

@liudezhi2098 Are you planning to continue working on this? I think that this is a really great improvement to address a long time issue with replicated subscriptions.

yes , I will continue working on this

[lhotari]

@liudezhi2098 Any updates on this PR?

[lhotari]

@liudezhi2098 Any updates on this PR?

[Ksnz]

It seems that the best way to store ReplicatedSubscriptionsSnapshots is in something like an increasing backoff queue:
the last one — the most recent,
the next — plus snapshotFrequencyMillis,
then snapshotFrequencyMillis^2,
then snapshotFrequencyMillis^3, and so on.
The oldest one (the head) should remain untouched until it is consumed.

It also seems that the solution could be simplified around ReplicatedSubscriptionSnapshotCache, and delayed messages would be handled much better.
If the PR is stale, I may reopen a new one :)

[lhotari]

It seems that the best way to store ReplicatedSubscriptionsSnapshots is in something like an increasing backoff queue: the last one — the most recent, the next — plus snapshotFrequencyMillis, then snapshotFrequencyMillis^2, then snapshotFrequencyMillis^3, and so on. The oldest one (the head) should remain untouched until it is consumed.

It also seems that the solution could be simplified around ReplicatedSubscriptionSnapshotCache, and delayed messages would be handled much better. If the PR is stale, I may reopen a new one :)

@Ksnz That could be useful if time would be the only contributing factor.

One detail that has been missed before (in the comments of this PR) is the number of entries between 2 different snapshots. Optimally, the cache would keep snapshots with equal number of entries between the snapshots. When there's a burst of messages to the topic, there could be a lot of messages between a single snapshot that happens once per second.
When the snapshot cache fills up, I'd assume that the snapshot entry with the shortest distance to it's adjacent entries should be evicted. WDYT?

[Ksnz]

WDYT?

There may be several strategies to address this. And the cache-eviction policy should probably be aware of the namespace, topic, or even individual subscription.

The main problem in my current project is that markDelete never advances on the delayed-message topic replica. In cluster A the markDelete cursor moves only rarely, at unpredictable moments, and over an unpredictable range. This causes the Cluster B replica backlog to grow.

To mitigate this, we want to keep at least the oldest snapshot to ensure that we do not lose any markDelete update.

The following strategies come to mind:

1. Equal time intervals (current behavior) — evict from the head. May be used as a default, to not break expected behaviour.
2. Equal distance intervals (new strategy) — evict from the head. Requires extra code outside of ReplicatedSubscriptionSnapshotCache, because Position itself is not trusted source to calculate distance.
3. Increasing time intervals (from tail to head) (new strategy) — evict from the middle, keep the head.
4. Increasing distance intervals (from tail to head) (new strategy) — evict from the middle, keep the head. Again extra code.
5. Increasing time intervals (from head to tail) (new strategy) — evict from the middle, keep the head.
6. Increasing distance intervals (from head to tail) (new strategy) — evict from the middle, keep the head. Requires additional code .

Policies 5 or 6 fit our needs for delayed-topics. But any distance-based strategies require a lot changes to be implemented.

For me the simple way is to implement 3 and 5 strategies.
3 for fast-consumers but sometimes get stuck due to burst of messages. To not loss sync at all.
5 for delayed-messages consumers, where markDelete moves intermittently and unpredictably.

And an option to pick strategy per namespace.
Ability to pick strategy on consumer configuration requires a cascade of code changes.

[lhotari]

The main problem in my current project is that markDelete never advances on the delayed-message topic replica. In cluster A the markDelete cursor moves only rarely, at unpredictable moments, and over an unpredictable range. This causes the Cluster B replica backlog to grow.

@Ksnz Yes, this seems like a problem case where the issue is caused by snapshot cache overflowing.

2. Equal distance intervals (new strategy) — evict from the head. Requires extra code outside of ReplicatedSubscriptionSnapshotCache, because Position itself is not trusted source to calculate distance.

There's existing org.apache.bookkeeper.mledger.ManagedLedger#getNumberOfEntries(com.google.common.collect.Range<org.apache.bookkeeper.mledger.Position>) available for the calculation. The reason why I think it's more useful than time based strategies is that the rate of consuming usually remains about the same while producing can have high bursts.

3. Increasing time intervals (from tail to head) (new strategy) — evict from the middle, keep the head.
4. Increasing distance intervals (from tail to head) (new strategy) — evict from the middle, keep the head. Again extra code.
5. Increasing time intervals (from head to tail) (new strategy) — evict from the middle, keep the head.
6. Increasing distance intervals (from head to tail) (new strategy) — evict from the middle, keep the head. Requires additional code .

"evict from the middle" sounds great, but it's not practical due to the variable producing speed. It could work for topics where producing speed remains constant.

And an option to pick strategy per namespace. Ability to pick strategy on consumer configuration requires a cascade of code changes.

Having too many options to pick from is not a great idea. It should be possible to solve this snapshot caching issue in a generic way that works for all cases sufficiently. Based on the previous experiences, I believe that it should be based on distances between positions and the cache should optimally keep entries with equal distances between the entries.
In addition, there might be a need to simply increase the default snapshot cache size, snapshot timeout time and/or make it configurable with namespace level policies.

btw. I've previously created a mermaidjs diagram to show the main interactions in replicated subscriptions: https://gist.github.com/lhotari/96fda511a70d7de93744d868b4472b92
That might be helpful to understand the flow. However, it seems that you have already found out how it works. :)

[Ksnz]

We could introduce an additional option similar to replicatedSubscriptionsSnapshotFrequencyMillis, but for message count – for example replicatedSubscriptionsSnapshotFrequencyMessageGap (the naming can be discussed later, along with the semantics of how the two properties interact), and send ReplicatedSubscriptionsSnapshotRequest not only based on time but also after a certain number of messages have been received.
This way, we could achieve intervals between snapshots that are equal in terms of message count.

However, I still don’t see how this would help when markDelete can be far beyond the horizon of the current cache.

An additional option to solution would be to store the ReplicatedSubscriptionSnapshotCache somewhere in an off-heap storage, for example in RocksDB, evicting entries that are older than the markDelete horizon.

This is the generic approach to keep both slow/delayed message consumers and fast backlog-tip consumers subscriptions aligned across clusters and to minimise inconsistency lag.

[lhotari]

We could introduce an additional option similar to replicatedSubscriptionsSnapshotFrequencyMillis, but for message count – for example replicatedSubscriptionsSnapshotFrequencyMessageGap (the naming can be discussed later, along with the semantics of how the two properties interact), and send ReplicatedSubscriptionsSnapshotRequest not only based on time but also after a certain number of messages have been received. This way, we could achieve intervals between snapshots that are equal in terms of message count.

Yes, a position interval based approach could be useful in ensuring that there would be more frequent snapshots when a lot of messages have been produced. One detail is that the snapshotting isn't deterministic so it wouldn't result in equal intervals between the snapshots.

• Since the snapshot require a request-response flow, it's not deterministic. The remote side might be offline or the geo-replication processing of the messages might be slow or vary. There wouldn't be a stable interval between the completed snapshots because of these differences.
• There could also be more than 2 clusters in geo-replication and that requires 2 rounds of request-response. This increases variance.

However, I still don’t see how this would help when markDelete can be far beyond the horizon of the current cache.

The cache would keep snapshots with equal position intervals. I don't see why it wouldn't help.
Increasing the snapshot cache size would also help. It's not that much memory that it requires. Optimizing the memory usage further would be useful. One optimization target would be to de-duplicate the java.lang.String instances of the cluster ids that are part of the snapshots. org.apache.pulsar.common.util.StringInterner class can be used for deduplication.

An additional option to solution would be to store the ReplicatedSubscriptionSnapshotCache somewhere in an off-heap storage, for example in RocksDB, evicting entries that are older than the markDelete horizon.

This shouldn't be needed since the amount of memory that the snapshots consume can be optimized. It should be possible to store millions of snapshots with fairly low heap memory usage after optimizing the data structures.

[Ksnz]

Okay, let’s go back to the square one. We have this description:

When you enable replicated subscriptions, you're creating a consistent distributed snapshot to establish an association between message ids from different clusters. The snapshots are taken periodically. The default value is 1 second. It means that a consumer failing over to a different cluster can potentially receive 1 second of duplicates. You can also configure the frequency of the snapshot in the broker.conf file.

In my opinion, a one-second window is an acceptable limitation, even if a burst of messages occurs during that second it is still one second.

The problem is that we evict old snapshots from the head of the queue (the oldest positions), so slow consumers can no longer move their markDelete position because the required snapshot is already gone.

Perhaps if we added an option to make the cache act as a buffer (skipping all new entries once it’s full) it would help slow subscriptions stay in sync. Moreover, it would be generic approach

After a successful advanced markDelete position, the buffer would be cleared up to some point and refilled with new ReplicatedSubscriptionsSnapshots.

The queue that is aligned to a markDelete position is more useful for synchronization than a queue that follows the tip of the topic.

A remaining issue is that a slow consumer can suddenly advance its markDelete position by a large amount, wiping the entire buffer. The next snapshots will then be captured far ahead of the consumer’s new markDelete, leaving a gap.

But we can combine the best of both worlds. By adding the
private final NavigableMap<Position, ReplicatedSubscriptionsSnapshot> tailSnapshots;
and
private final NavigableMap<Position, ReplicatedSubscriptionsSnapshot> headSnapshots;
to a ReplicatedSubscriptionSnapshotCache

It's not a perfect solution. But it's less evil than rewriting everything and still better than the current one.

[lhotari]

In my opinion, a one-second window is an acceptable limitation, even if a burst of messages occurs during that second it is still one second.

That's true, but the end result isn't that the distance between 2 different snapshots would be with 1 second difference in rate since it's not deterministic due to the request-response required for creating the snapshot.

The problem is that we evict old snapshots from the head of the queue (the oldest positions), so slow consumers can no longer move their markDelete position because the required snapshot is already gone.

I agree that it doesn't make sense to evict the head of the queue.

Perhaps if we added an option to make the cache act as a buffer (skipping all new entries once it’s full) it would help slow subscriptions stay in sync. Moreover, it would be generic approach

The problem with that approach is that there would later be problems in updating the mark delete position since there would be a long duration where there wouldn't be snapshots at all. That's the reason why I think that it makes more sense to have an eviction algorithm that keeps snapshots with similar distances between the first entry and the last entry in the queue. It's not very hard to implement such an eviction algorithm. Together with a much larger maximum cache size (or even dynamic max size), it's possible to address the problem that has been caused by the snapshot cache overflowing.

After a successful advanced markDelete position, the buffer would be cleared up to some point and refilled with new ReplicatedSubscriptionsSnapshots.

Yes, it will eventually catch up as long as the head of the queue isn't deleted. However, there's a potential to fix the complete problem of having a long lag in the update when the eviction is improved and there are a lot more snapshots available.

The queue that is aligned to a markDelete position is more useful for synchronization than a queue that follows the tip of the topic.

yes

A remaining issue is that a slow consumer can suddenly advance its markDelete position by a large amount, wiping the entire buffer. The next snapshots will then be captured far ahead of the consumer’s new markDelete, leaving a gap.

yes. that's what could be addressed with the improved eviction algorithm that would keep snapshots with about equal distances from the head to the tail.

But we can combine the best of both worlds. By adding the private final NavigableMap<Position, ReplicatedSubscriptionsSnapshot> tailSnapshots; and private final NavigableMap<Position, ReplicatedSubscriptionsSnapshot> headSnapshots; to a ReplicatedSubscriptionSnapshotCache

It's not a perfect solution. But it's less evil than rewriting everything and still better than the current one.

true

[lhotari]

@Ksnz I have created draft PR #25044 based on my previous comments. It's easier to continue the discussion on that PR when there's concrete code. I haven't yet done any performance optimization and it's not great yet. Adding 1 million snapshots in the unit test takes about 18 seconds on Mac M3, so there's a need to optimize it. I'll take a closer look later by adding a JMH benchmark.

[lhotari]

Replaced by #25044