Skip to content

[feature][connector] JDBC sinks: support upsert and row deletion - #16448

Merged
nicoloboschi merged 1 commit into
apache:masterfrom
nicoloboschi:jdbc-sink-support-update-delete
Jul 14, 2022
Merged

[feature][connector] JDBC sinks: support upsert and row deletion#16448
nicoloboschi merged 1 commit into
apache:masterfrom
nicoloboschi:jdbc-sink-support-update-delete

Conversation

@nicoloboschi

@nicoloboschi nicoloboschi commented Jul 7, 2022

Copy link
Copy Markdown
Contributor

Motivation

Currently JDBC sink will perform an insert query for most of the cases. The only way to execute an UPDATE or DELETE is to pass a specific property in the message properties. This is not handy and requires to all the sources to add this specific option.

The goal is to provide a way to upsert and delete records based on the record content while using key-value records.

  1. Upsert: Insert or update the record based on the key fields.
  2. Delete: if the value is NULL, delete the record for the given row.

Modifications

  • new config option insertMode with values
    • INSERT: only perform blindly insert (default, in order to not break upgrades)
    • UPDATE: only perform blindly update
    • UPSERT: use upsert query. Main issue is that there's no SQL standard so every sink has his own implementation (by default it throws a not supported exception)
  • new config option nullValueAction with values
    • FAIL: update the row with null values (default, in order to not break upgrades)
    • DELETE: delete the row with the the given key/pk

Note that in order to not break compatibility the message property ACTION has the precedence on other logics.

For the upsert mode, only the following sinks support it:

  • SqlLite
  • Postgres
  • MariaDB

Also upgraded SqlLite to 3.36.0.3 to support upsert

others will throw an exception and the message will be rejected

  • doc

@github-actions

github-actions Bot commented Jul 7, 2022

Copy link
Copy Markdown

@nicoloboschi Please provide a correct documentation label for your PR.
Instructions see Pulsar Documentation Label Guide.

@nicoloboschi nicoloboschi added doc Your PR contains doc changes, no matter whether the changes are in markdown or code files. and removed doc-label-missing labels Jul 7, 2022

@eolivelli eolivelli left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work !

Comment thread site2/docs/io-jdbc-sink.md Outdated
Comment thread site2/docs/io-jdbc-sink.md Outdated
@nicoloboschi nicoloboschi reopened this Jul 12, 2022
@nicoloboschi
nicoloboschi force-pushed the jdbc-sink-support-update-delete branch from cc7b7cd to 392bcc6 Compare July 12, 2022 20:34
@nicoloboschi
nicoloboschi force-pushed the jdbc-sink-support-update-delete branch from 392bcc6 to 8faa374 Compare July 13, 2022 08:06
@nicoloboschi
nicoloboschi merged commit ac41cfe into apache:master Jul 14, 2022
@nicoloboschi nicoloboschi added this to the 2.11.0 milestone Jul 14, 2022
nicoloboschi added a commit to datastax/pulsar that referenced this pull request Jul 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connector doc Your PR contains doc changes, no matter whether the changes are in markdown or code files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants