Skip to content

[FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink#3296

Closed
yuxiqian wants to merge 9 commits into
apache:masterfrom
yuxiqian:FLINK-35243
Closed

[FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink#3296
yuxiqian wants to merge 9 commits into
apache:masterfrom
yuxiqian:FLINK-35243

Conversation

@yuxiqian

@yuxiqian yuxiqian commented May 6, 2024

Copy link
Copy Markdown
Member

This PR closes FLINK-35243:

  • Supports AlterColumnCommentEvent, AlterTableCommentEvent, DropTableEvent, RenameTableEvent, and TruncateTableEvent in pipeline framework
  • Adds SchemaChangeEventWithPreSchema interface, allowing SchemaRegistry backfilling schema before change payload
  • Fixed MySQL pipeline source doesn't capture ALTER TABLE ... MODIFY COLUMN DDL
  • Adds corresponding test cases

This requires #3339 being merged first.

@yuxiqian

yuxiqian commented May 8, 2024

Copy link
Copy Markdown
Member Author

@PatrickRen @ruanhang1993 PTAL

@yuxiqian yuxiqian force-pushed the FLINK-35243 branch 2 times, most recently from f22c3a4 to 1e8ea73 Compare May 9, 2024 06:11
@melin

melin commented May 10, 2024

Copy link
Copy Markdown

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

@yuxiqian

yuxiqian commented May 10, 2024

Copy link
Copy Markdown
Member Author

DropTableEvent TruncateTableEvent can be closed. Sometimes, tables cannot be deleted synchronously.

Hi @melin, thanks for your comments. It's a common use case to apply some "safe" schema change events (like AddColumnEvent) and ignore "dangerous" events (like Truncate / Drop tables). So we're designing a new feature allowing users to configure schema change behaviour for each type of schema change events in FLINK-35242. Also, we're trying to expose interfaces allowing Sink connectors to report their supported kind of schema change events.

This PR is still in early draft stage, so feel free to leave your comments if you have any other concerns.

@yuxiqian yuxiqian marked this pull request as draft May 31, 2024 02:34
@github-actions github-actions Bot added the cli label May 31, 2024
@yuxiqian yuxiqian changed the title [FLINK-35243][cdc] Support table schema change events & PreSchema backfill [FLINK-35243][cdc] Supports more schema change event types & backfill pre-schema payload for sink May 31, 2024
@melin

melin commented Aug 2, 2024

Copy link
Copy Markdown

support add/drop primary key event

@yuxiqian

yuxiqian commented Aug 2, 2024

Copy link
Copy Markdown
Member Author

Thanks for @melin's suggestion, will be working on this after #3339 got merged.

@yuxiqian

yuxiqian commented Aug 8, 2024

Copy link
Copy Markdown
Member Author

Seems it's not meaningful to rebase, since there are too many conflicts. Will open another PR to address this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants