Skip to content

[improve][connector] add reader config to pulsar-io-debezium and pulsar-io-kafka-connect-adaptor - #16675

Merged
codelipenghui merged 5 commits into
apache:masterfrom
freeznet:freeznet/add-sub-name-for-debezium-reader
Jul 25, 2022
Merged

[improve][connector] add reader config to pulsar-io-debezium and pulsar-io-kafka-connect-adaptor#16675
codelipenghui merged 5 commits into
apache:masterfrom
freeznet:freeznet/add-sub-name-for-debezium-reader

Conversation

@freeznet

@freeznet freeznet commented Jul 19, 2022

Copy link
Copy Markdown
Contributor

Fixes #16674

Motivation

Allow user customize the reader configs in pulsar-io-debezium and pulsar-io-kafka-connect-adaptor.

Modifications

  • add offset.storage.reader.config to PulsarKafkaWorkerConfig
  • add pulsar.reader.config to PulsarDatabaseHistory
  • add utils to pulsar-io-common
  • add pulsar-io-common dependency to pulsar-io-kafka-connect-adaptor
  • add tests

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@freeznet freeznet added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages area/connector labels Jul 19, 2022
@freeznet freeznet self-assigned this Jul 19, 2022
@freeznet freeznet added this to the 2.11.0 milestone Jul 19, 2022
@github-actions github-actions Bot added the doc-required Your PR changes impact docs and you will update later. label Jul 19, 2022
@dave2wave
dave2wave requested review from dlg99 and nicoloboschi July 19, 2022 15:15

@codelipenghui codelipenghui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just left a minor comment.

Comment thread pulsar-io/common/src/main/java/org/apache/pulsar/io/common/IOConfigUtils.java Outdated
@freeznet
freeznet force-pushed the freeznet/add-sub-name-for-debezium-reader branch from ed2e9d4 to 1470168 Compare July 21, 2022 01:30
@freeznet

Copy link
Copy Markdown
Contributor Author

/pulsarbot run-failure-checks

@freeznet
freeznet force-pushed the freeznet/add-sub-name-for-debezium-reader branch from 1470168 to 84cee00 Compare July 22, 2022 03:09
@freeznet

Copy link
Copy Markdown
Contributor Author

/pulsarbot run-failure-checks

@codelipenghui
codelipenghui merged commit 9c19976 into apache:master Jul 25, 2022
codelipenghui pushed a commit that referenced this pull request Jul 25, 2022
…ulsar-io-kafka-connect-adaptor` (#16675)

(cherry picked from commit 9c19976)
codelipenghui pushed a commit that referenced this pull request Jul 25, 2022
…ulsar-io-kafka-connect-adaptor` (#16675)

(cherry picked from commit 9c19976)
@momo-jun

Copy link
Copy Markdown
Contributor

Hi @freeznet, do you have any updates on the required doc?

@Anonymitaet Anonymitaet added doc-complete Your PR changes impact docs and the related docs have been already added. and removed doc-required Your PR changes impact docs and you will update later. labels Aug 2, 2022
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Aug 16, 2022
…ulsar-io-kafka-connect-adaptor` (apache#16675)

(cherry picked from commit 9c19976)
(cherry picked from commit d923c84)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connector cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-complete Your PR changes impact docs and the related docs have been already added. release/2.9.4 release/2.10.2 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ability to configure subscription name in debezium connector

5 participants