Skip to content

make KafkaSourceRecord ack() async to avoid deadlock (#11435) - #18

Merged
eolivelli merged 1 commit into
datastax:2.7.2_dsfrom
dlg99:ls272_pr11435
Jan 11, 2022
Merged

make KafkaSourceRecord ack() async to avoid deadlock (#11435)#18
eolivelli merged 1 commit into
datastax:2.7.2_dsfrom
dlg99:ls272_pr11435

Conversation

@dlg99

@dlg99 dlg99 commented Jan 11, 2022

Copy link
Copy Markdown
Collaborator

Cherry-pick of apache#11435 into LS272:

The ack() method of the AbstractKafkaSourceRecord should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main public/default/debezium-mongodb-source-0 thread. And further blocks the whole debezium connector to work correctly.

  1. remove the blocking future.get() call from ack()
  2. move the commit logic into callbacks

(If this PR fixes a github issue, please add Fixes #<xyz>.)

Fixes #

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

Explain here the context, and why you're making that change. What is the problem you're trying to solve.

Modifications

Describe the modifications you've done.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

The `ack()` method of the `AbstractKafkaSourceRecord` should be non-blocking. Otherwise there'll be deadlock for pulsar-client-io thread and the main `public/default/debezium-mongodb-source-0` thread. And further blocks the whole debezium connector to work correctly.

1. remove the blocking `future.get()` call from `ack()`
2. move the commit logic into callbacks

@eolivelli eolivelli left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@eolivelli
eolivelli merged commit 041ce94 into datastax:2.7.2_ds Jan 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants