Skip to content

[FLINK-34653] Support table merging with route#3129

Merged
leonardBang merged 1 commit into
apache:masterfrom
PatrickRen:table-merge
Apr 16, 2024
Merged

[FLINK-34653] Support table merging with route#3129
leonardBang merged 1 commit into
apache:masterfrom
PatrickRen:table-merge

Conversation

@PatrickRen

Copy link
Copy Markdown
Contributor

This pull request supports complex table merging with route, including:

  • Keeping all columns from source tables
  • Type coercion if type of columns have conflict
  • Schema evolution under table merging

@lvyanquan lvyanquan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @PatrickRen for this contribution, left two comments.

}
BinaryRecordDataGenerator recordDataGenerator =
new BinaryRecordDataGenerator(
routedTableSchema.getColumnDataTypes().toArray(new DataType[0]));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we cache a tunple of routedTableSchema and its recordDataGenerator to avoid creating it for every event?

} else {
RecordData.FieldGetter fieldGetter =
RecordData.createFieldGetter(
originalSchema.getColumn(columnName).get().getType(), columnIndex);

@lvyanquan lvyanquan Mar 13, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we cache a tunple of originalSchema and its fieldGetterList? As RecordData.createFieldGetter will need to judge DataType and create new Object.

@leonardBang

Copy link
Copy Markdown
Contributor

@PatrickRen Thanks for your contribution, Could you rebase this PR to latest master? And I found the PR contains a WIP commit, Could you resolve it?

@leonardBang leonardBang self-requested a review April 3, 2024 10:15

@leonardBang leonardBang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @PatrickRen for the nice work, LGTM

@leonardBang leonardBang merged commit 6017b16 into apache:master Apr 16, 2024
wuzhenhua01 pushed a commit to wuzhenhua01/flink-cdc-connectors that referenced this pull request Aug 4, 2024
ChaomingZhangCN pushed a commit to ChaomingZhangCN/flink-cdc that referenced this pull request Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants