Skip to content

feat(connector): add Iceberg sink connector#2191

Merged
spetz merged 39 commits intoapache:masterfrom
EdgarModesto23:iceberg-connector
Nov 19, 2025
Merged

feat(connector): add Iceberg sink connector#2191
spetz merged 39 commits intoapache:masterfrom
EdgarModesto23:iceberg-connector

Conversation

@EdgarModesto23
Copy link
Copy Markdown
Contributor

@EdgarModesto23 EdgarModesto23 commented Sep 23, 2025

This closes #1850.

This PR adds and Iceberg sink connector with the following features:

  • Support for S3-compatible storage
  • Support for REST catalogs
  • Single destination table
  • Multiple-table fan-out static routing
  • Multiple-table fan-out dynamic routing

I originally intended to include support for hms catalogs, but i had issues compiling the current implementation of the hms iceberg catalog in rust

It is also worth mentioning that although i included the code necessary to support glue, the iceberg glue catalog team is still working towards appending data files

@hubcio
Copy link
Copy Markdown
Contributor

hubcio commented Sep 23, 2025

Also, fix CI :)

@EdgarModesto23 EdgarModesto23 changed the title feat(connector): Add Iceberg sink connector (WIP) feat(connector): Add Iceberg sink connector Sep 27, 2025
@spetz
Copy link
Copy Markdown
Contributor

spetz commented Nov 13, 2025

Hey @EdgarModesto23 do you plan to finalize this PR?

@EdgarModesto23
Copy link
Copy Markdown
Contributor Author

Hey @EdgarModesto23 do you plan to finalize this PR?

Hey @spetz, this PR for the Iceberg Sink connector is ready.
As I mentioned in the original description, the Iceberg Rust team is still working on adding support for appending files on Glue. Once that’s available, I plan to add the necessary code in a separate PR. Other than that, this one should be good to go.

Regarding the Iceberg Source connector. My plan was to merge the Sink connector first, and then work on the Source one when I have time. However, if you prefer both connectors to be merged together, we can close this PR and reopen it once both are complete.

@spetz
Copy link
Copy Markdown
Contributor

spetz commented Nov 13, 2025

@EdgarModesto23 thank you for the response, would you mind resolving these few comments I added back then during the initial review? And then, there are just the basic conflicts due to updated deps, and we could merge this PR :)

@hubcio hubcio changed the title feat(connector): Add Iceberg sink connector feat(connector): add Iceberg sink connector Nov 18, 2025
@spetz spetz merged commit 0d3ca8e into apache:master Nov 19, 2025
49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Apache Iceberg connectors

4 participants