[Feature] Introduce continuous offset for pulsar by aloyszhang · Pull Request #9039 · apache/pulsar

aloyszhang · 2020-12-23T07:59:21Z

Motivation

As described in PIP-70.
One of the use case for Broker entry metadata is providing continuous message sequence-Id for messages in one topic-partition which is useful for Protocol Hanlder like KOP.

This PR enable Pulsar to support continuous offset for message based on Broker entry metadata.

Modifications

Introduce a new field for broker entry metadta named offset;
Introduce a new interceptor type ManagedLedgerInterceptor which intercept entry in ManagedLedger；
Each partition will be assigned a ManagedLedgerInterceptor when ManagedLedger created;
Each Entry will be intercept for adding a monotone increasing offset in Broker entry metadata and the offet is added by batchSize of entry;
Support find position by a given offset.

aloyszhang · 2020-12-23T11:17:29Z

/pulsarbot run-failure-checks

sijie · 2020-12-23T19:00:07Z

+    /**
+     * Find position by sequenceId.
+     * */
+    CompletableFuture<PositionImpl> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);


Suggested change

CompletableFuture<PositionImpl> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);

CompletableFuture<Position> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);

sijie · 2020-12-23T19:00:42Z

+    /**
+     * Find position by sequenceId.
+     * */
+    CompletableFuture<PositionImpl> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);


Suggested change

CompletableFuture<PositionImpl> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);

CompletableFuture<PositionImpl> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);

Can we use the java.util.Predicate?

condition of OpFindNewest is com.google.common.base.Predicate type, so the parameter type here should also be com.google.common.base.Predicate

I'd recommend using java.util.Predicate in the interface. You can just write a wrapper to convert a java.util.Predicate to a guava Predicate.

sijie · 2020-12-23T20:30:25Z


+    @Override
+    public Position addEntry(byte[] data, int batchSize) throws InterruptedException, ManagedLedgerException {
+        return addEntry(data, batchSize,0, data.length);


Suggested change

return addEntry(data, batchSize,0, data.length);

return addEntry(data, batchSize, 0, data.length);

sijie · 2020-12-23T20:32:29Z

+            addOperation.failed(
+                    new ManagedLedgerInterceptException("Interceptor managed ledger before add to bookie failed."));
+            ReferenceCountUtil.release(addOperation.data);
+            log.error("[{}] Failed to interceptor entry before add to bookie.", name, e);


Suggested change

log.error("[{}] Failed to interceptor entry before add to bookie.", name, e);

log.error("[{}] Failed to intercept adding an entry to bookie.", name, e);

sijie · 2020-12-23T20:34:37Z

+/**
+ * Interceptor for ManagedLedger.
+ * */
+public interface ManagedLedgerInterceptor {


Can you add the interface annotations to the new interface? See https://github.com/apache/pulsar/wiki/PIP-72%3A-Introduce-Pulsar-Interface-Taxonomy%3A-Audience-and-Stability-Classification

sijie · 2020-12-23T20:34:49Z

+ * Interceptor for ManagedLedger.
+ * */
+public interface ManagedLedgerInterceptor {
+    OpAddEntry beforeAddEntry(OpAddEntry op, int batchSize);


Can you add javadoc to the methods?

sijie · 2020-12-23T20:37:28Z

+
+public class ManagedLedgerInterceptorImpl implements ManagedLedgerInterceptor {
+    private static final Logger log = LoggerFactory.getLogger(ManagedLedgerInterceptorImpl.class);
+    private static final String OFFSET = "offset";


kafka offset is not a very good term to be used for describing the index of a message within a log stream. Instead, I would suggest calling it index or logIndex similar to the term used in the Raft algorithm (https://cs.stackexchange.com/questions/97542/raft-algorithm-whats-the-meaning-of-concept-index).

modify offset to index

sijie · 2020-12-23T20:45:22Z

-                if (appendBrokerEntryMetadata(headersAndPayload, publishContext)) {
-                    ledger.asyncAddEntry(headersAndPayload, this, publishContext);
-                }
+                ledger.asyncAddEntry(headersAndPayload, this, publishContext);


How is the batchSize passed to asyncAddEntry? I failed to see how did you do that in this pull request.

At the same time, I think batchSize is not a good term. If I understand this correctly, it should be numberOfMessages, correct?

Anonymitaet · 2021-05-12T12:05:55Z

@aloyszhang thanks for your great work.

Shall the changes be documented to the Pulsar docs?

If so, could you please help add the docs accordingly? Then you can ping me to review, thanks

### Motivation When the Pulsar cluster enables broker entry metadata, sometimes there're some corrupted entries. See streamnative/kop#442 for example. It's because the broker entry metadata has been added twice. This bug might be introduced from #9039 https://github.com/apache/pulsar/blob/9b7c3275c904ac1e6a8ef67487a10a0506bb2c58/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1516-L1518 It happened during a managed ledger's rollover while there're some pending `OpAddEntry`s in `updateLedgersIdsComplete`, only the ledger id should be updated and the data of `OpAddEntry` should not be modified. ### Modifications Only call `beforeAddEntry` for once at the beginning of `internalAsyncAddEntry`.

### Motivation When the Pulsar cluster enables broker entry metadata, sometimes there're some corrupted entries. See streamnative/kop#442 for example. It's because the broker entry metadata has been added twice. This bug might be introduced from #9039 https://github.com/apache/pulsar/blob/9b7c3275c904ac1e6a8ef67487a10a0506bb2c58/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1516-L1518 It happened during a managed ledger's rollover while there're some pending `OpAddEntry`s in `updateLedgersIdsComplete`, only the ledger id should be updated and the data of `OpAddEntry` should not be modified. ### Modifications Only call `beforeAddEntry` for once at the beginning of `internalAsyncAddEntry`. (cherry picked from commit 9d44617)

### Motivation When the Pulsar cluster enables broker entry metadata, sometimes there're some corrupted entries. See streamnative/kop#442 for example. It's because the broker entry metadata has been added twice. This bug might be introduced from apache#9039 https://github.com/apache/pulsar/blob/9b7c3275c904ac1e6a8ef67487a10a0506bb2c58/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1516-L1518 It happened during a managed ledger's rollover while there're some pending `OpAddEntry`s in `updateLedgersIdsComplete`, only the ledger id should be updated and the data of `OpAddEntry` should not be modified. ### Modifications Only call `beforeAddEntry` for once at the beginning of `internalAsyncAddEntry`. (cherry picked from commit 9d44617)

### Motivation When the Pulsar cluster enables broker entry metadata, sometimes there're some corrupted entries. See streamnative/kop#442 for example. It's because the broker entry metadata has been added twice. This bug might be introduced from apache#9039 https://github.com/apache/pulsar/blob/9b7c3275c904ac1e6a8ef67487a10a0506bb2c58/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L1516-L1518 It happened during a managed ledger's rollover while there're some pending `OpAddEntry`s in `updateLedgersIdsComplete`, only the ledger id should be updated and the data of `OpAddEntry` should not be modified. ### Modifications Only call `beforeAddEntry` for once at the beginning of `internalAsyncAddEntry`.

aloyszhang added 10 commits December 21, 2020 17:27

add offset for broker entry metadata

1918e00

add managedledger interceptor

74a9a08

initialize intercptor for ML

c38bba1

interceptor entry before add to bookie to generator offset

483cee9

add asyncFindPosition method for ManagedLedger

01da511

support append offset in managedInterceptor

4d1dfb3

interceptor pending write request

d8cbd22

add offet to ManagedLedgerInfo

5137d6f

add test for offset

4205436

add test for ManagedLedgerInterceptor

a52038b

codelipenghui assigned aloyszhang Dec 23, 2020

codelipenghui added this to the 2.8.0 milestone Dec 23, 2020

codelipenghui requested review from codelipenghui, jiazhai, merlimat and sijie December 23, 2020 13:22

sijie requested changes Dec 23, 2020

View reviewed changes

sijie added area/broker type/feature The PR added a new feature or issue requested a new feature labels Dec 23, 2020

zymap mentioned this pull request Dec 24, 2020

[BUG] kafka-client stuck when reading data streamnative/kop#283

Closed

aloyszhang added 2 commits December 24, 2020 10:09

apply comment

3b98e9f

fix send error

d4fd5c8

aloyszhang requested a review from sijie December 24, 2020 06:04

sijie approved these changes Dec 24, 2020

View reviewed changes

codelipenghui approved these changes Dec 24, 2020

View reviewed changes

codelipenghui merged commit d85a5e2 into apache:master Dec 24, 2020

aloyszhang deleted the offset branch December 25, 2020 03:07

aloyszhang mentioned this pull request Dec 25, 2020

[BUG] upgrade dependency of pulsar to enable continuous index streamnative/kop#289

Closed

LeBW mentioned this pull request Aug 6, 2021

Exposing the broker entry metadata to client #11553

Merged

1 task

BewareMyPower mentioned this pull request Sep 18, 2021

Avoid adding duplicated BrokerEntryMetadata #12018

Merged

1 task

graysonzeng mentioned this pull request Jan 22, 2024

[improve][functions] EFFECTIVELY_ONCE mode support input topic batching #21812

Open

15 tasks

lhotari mentioned this pull request Feb 13, 2025

[improve][broker] Don't call ManagedLedger#asyncAddEntry in Netty I/O thread #23983

Closed

4 tasks

codelipenghui mentioned this pull request Jun 14, 2025

[improve][pip] PIP-426: Enhanced Consumer Throttling and Unacknowledged Message Tracking for Exclusive and Failover Subscriptions #24400

Open

15 tasks

	CompletableFuture<PositionImpl> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);
	CompletableFuture<Position> asyncFindPosition(com.google.common.base.Predicate<Entry> predicate);

	return addEntry(data, batchSize,0, data.length);
	return addEntry(data, batchSize, 0, data.length);

	log.error("[{}] Failed to interceptor entry before add to bookie.", name, e);
	log.error("[{}] Failed to intercept adding an entry to bookie.", name, e);

Uh oh!

Conversation

aloyszhang commented Dec 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Uh oh!

aloyszhang commented Dec 23, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aloyszhang Dec 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Anonymitaet commented May 12, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

aloyszhang commented Dec 23, 2020 •

edited

Loading

aloyszhang Dec 24, 2020 •

edited

Loading