Skip to content

[pulsar io] make KafkaSourceRecord ack() non-blocking to avoid deadlock - #11435

Merged
sijie merged 1 commit into
apache:masterfrom
streamnative:neng/kafka-source-async-ack
Jul 23, 2021
Merged

[pulsar io] make KafkaSourceRecord ack() non-blocking to avoid deadlock#11435
sijie merged 1 commit into
apache:masterfrom
streamnative:neng/kafka-source-async-ack

Conversation

@nlu90

@nlu90 nlu90 commented Jul 23, 2021

Copy link
Copy Markdown
Member

Motivation

The ack() method of the AbstractKafkaSourceRecord should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main public/default/debezium-mongodb-source-0 thread. And further blocks the whole debezium connector to work correctly.

Modifications

  1. remove the blocking future.get() call from ack()
  2. move the commit logic into callbacks

Verifying this change

  • Make sure that the change passes the CI checks.

Documentation

For contributor

For this PR, do we need to update docs?

No. Internal bug fix.

@sijie sijie added this to the 2.9.0 milestone Jul 23, 2021
eolivelli
eolivelli approved these changes Jul 23, 2021
View reviewed changes

@eolivelli eolivelli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eolivelli

eolivelli commented Jul 23, 2021

Copy link
Copy Markdown
Contributor

@dlg99 do you mind taking aa look as well ? is this fixing the deadlock you found ?

dlg99
dlg99 approved these changes Jul 23, 2021
View reviewed changes

@dlg99 dlg99 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.
I didn't run the integration tests (they are not enabled on the CI), I expect @nlu90 did that.

@nlu90

nlu90 commented Jul 23, 2021

Copy link
Copy Markdown
Member Author

And Here's the integration test result run on my local laptop:

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Pulsar :: Tests 2.9.0-SNAPSHOT:
[INFO]
[INFO] Apache Pulsar :: Tests ............................. SUCCESS [  1.147 s]
[INFO] Apache Pulsar :: Tests :: Integration .............. SUCCESS [06:12 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  06:13 min
[INFO] Finished at: 2021-07-23T11:05:54-07:00
[INFO] ------------------------------------------------------------------------
➜  sn-pulsar git:(neng/kafka-source-async-ack) ✗ mvn test -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-io-sources.xml -DintegrationTests -Dtest=org.apache.pulsar.tests.integration.io.sources.debezium.PulsarDebeziumSourcesTest

@nlu90 nlu90 changed the title make KafkaSourceRecord ack() non-blocking to avoid deadlock [pulsar io] make KafkaSourceRecord ack() non-blocking to avoid deadlock Jul 23, 2021
@sijie
sijie merged commit b3892ee into apache:master Jul 23, 2021
@sijie
sijie deleted the neng/kafka-source-async-ack branch July 23, 2021 20:13
@nlu90 nlu90 mentioned this pull request Jul 23, 2021
Merged
1 task
codelipenghui pushed a commit that referenced this pull request Jul 30, 2021
@nlu90 @codelipenghui
make KafkaSourceRecord ack() async to avoid deadlock (#11435)
9767a82 
The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks

(cherry picked from commit b3892ee)
@codelipenghui codelipenghui added the cherry-picked/branch-2.8 Archived: 2.8 is end of life label Jul 30, 2021
dlg99 pushed a commit to dlg99/pulsar that referenced this pull request Sep 24, 2021
@nlu90 @dlg99
make KafkaSourceRecord ack() async to avoid deadlock (apache#11435)
934a15e 
### Motivation
The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.


### Modifications

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks
eolivelli pushed a commit to datastax/pulsar that referenced this pull request Sep 24, 2021
@nlu90 @eolivelli
make KafkaSourceRecord ack() async to avoid deadlock (apache#11435)
b3ddb47 
### Motivation
The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.


### Modifications

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks
dlg99 pushed a commit to dlg99/pulsar that referenced this pull request Jan 11, 2022
@nlu90 @dlg99
make KafkaSourceRecord ack() async to avoid deadlock (apache#11435)
c3bf160 
The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks
@dlg99 dlg99 mentioned this pull request Jan 11, 2022
Merged
4 tasks
eolivelli pushed a commit to datastax/pulsar that referenced this pull request Jan 11, 2022
@dlg99 @nlu90
make KafkaSourceRecord ack() async to avoid deadlock (apache#11435) (#18
041ce94 
)

The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks

Co-authored-by: Neng Lu <nlu@streamnative.io>
bharanic-dev pushed a commit to bharanic-dev/pulsar that referenced this pull request Mar 18, 2022
@nlu90 @splunk-xingw
make KafkaSourceRecord ack() async to avoid deadlock (apache#11435)
00b1d45 
### Motivation
The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.


### Modifications

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sijie sijie sijie approved these changes
@dlg99 dlg99 dlg99 approved these changes
@eolivelli eolivelli eolivelli approved these changes
@codelipenghui codelipenghui codelipenghui approved these changes

Assignees

@nlu90 nlu90

Labels

area/connector cherry-picked/branch-2.8 Archived: 2.8 is end of life release/2.8.1

Projects

None yet

Milestone

2.9.0

Development

Successfully merging this pull request may close these issues.

5 participants

@nlu90 @eolivelli @sijie @dlg99 @codelipenghui