Support Kafka Connect within Iceberg

[kondamudikarthik]

Currently, data ingestion into Iceberg tables is only supported by Spark, Presto, and Flink. It would be nice to support ingestion via Kafka Connect as well.

cc @kbendick

[youngxinler]

Write to iceberg should be use compute enginer, computing and storage should be separated.
Iceberg itself has no computing power; Writing and reading should be done by a computing framework.
It may be better to use Flink SQL to make Kafka write to iceberg.

[kbendick]

Thanks for opening this @kondamudikarthik!

@youngxinler: first, thanks for your interest in this discussions. It’s not entirely true that writing to Iceberg tables needs to be achieved via a computing engine. It can absolutely be achieved with just the plain Java API (or the new Python refactor). While I generally recommend that a compute engine be used, as it’s an easier experience, my understanding is that Kafka Connect is a framework for map-side only / per-record transforms for writing data from Kafka to some other location (in this case Iceberg tables).

Near-real time data is very often written as-is or with only a handful of per-record transforms. For example, upstream producers will stage the data as they want it written, and then the folks who maintain the data lake might add some metadata to the record (ingestion timestamp etc) but otherwise write the data out as it’s been received, while ensuring that the data has consistent schema.

There are many scenarios in which Kafka Connect would be beneficial to end users and if there’s interest from the community then it’s something we should definitely consider supporting.

[kbendick]

That said, I’d love to know of / hear from people who have Kafka Connect experience and would be interested in helping contribute code or review code! I’m by no means an expert with Kafka Connect, but I can gladly help with the Iceberg side of things.

[youngxinler]

thanks for @kbendick discussion, i agree for @kbendick advice, although usually sometime we use compute engine to write, but kafka is very popular, if iceberg support kafka connector to iceberg, it will help more developer who use iceberg or want to use iceberg.

[kondamudikarthik]

Thanks, @kbendick. That's right having the ability to ingest data into Iceberg via Kafka Connect is a very powerful feature to have. @youngxinler I'm working on spec for the same, keep an eye on this space for more updates.

[felixYyu]

If read.streaming.enabled config via TimeTravel in real-time pipeline, I think more demand for real-time analysis scenarios.

[almogtavor]

A nice implementation project at the subject - https://github.com/getindata/kafka-connect-iceberg-sink

[code-magician323]

@kbendick Do you think there will be progress at this area soon?

[jaehyeon-kim]

+1 for Kafka connect.

[ddcprg]

Are there any progress on this area? If that's not the case, we are working on a connector at the moment and hopefully we'll be able to open source it

[kondamudikarthik]

We just started working this, timeline wise we think in a month or so we should be able to get it out.

[jvican]

Excited to hear you're working on it @kondamudikarthik Is progress happening in the OSS repo or internally somewhere else?

[ddcprg]

we've just open sourced this https://github.com/10xfuturetechnologies/kafka-connect-iceberg if anyone is interested. Note there still is work in progress, use it at your own risk and feel free to contribute if you find it useful

[mr-brobot]

Tabular recently created a Kafka Iceberg Sink connector. That seems like a good place to consolidate efforts.