build(deps): Kafka client 4.2.0 upgrade by Shekharrajak · Pull Request #19441 · apache/druid

Shekharrajak · 2026-05-09T17:57:21Z

Ref #19322

Description

Kafka client to 4.2.0
Docker image: apache/kafka:4.2.0
KafkaContainer using testcontainers-kafka jar
EmbeddedKafkaBroker test helper using Testcontainers
kafka-indexing-service tests to EmbeddedKafkaBroker; drop TestBroker

Release note

Upgraded the bundled Apache Kafka client and broker dependencies from 3.9.2 to 4.2.0; embedded test brokers now run on Testcontainers (apache/kafka:4.2.0) instead of the removed Scala/ZooKeeper-mode broker.

Key changed/added classes in this PR

pom.xml: apache.kafka.version → 4.2.0.
New EmbeddedKafkaBroker test helper backed by Testcontainers.
Migrated KafkaIndexTaskTest, KafkaRecordSupplierTest, KafkaSamplerSpecTest, KafkaSupervisorTest, KafkaStringHeaderFormatTest, and TestKafkaExtractionCluster off the legacy TestBroker / embedded ZooKeeper.
Deleted TestBroker; removed kafka_2.13, curator-test, and scala-library test dependencies.
KafkaResource Docker image bumped to apache/kafka:4.2.0.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

…tBroker

Shekharrajak · 2026-05-09T18:05:44Z

-      <scope>test</scope>
-    </dependency>
-    <dependency>
-      <groupId>org.apache.kafka</groupId>


Druid only ever depended on it from test scope, and only because TestBroker needed kafka.server.KafkaServer. Once TestBroker is gone, nothing imports anything from kafka.* (Scala namespace)

Shekharrajak · 2026-05-09T18:07:00Z

 import org.apache.druid.indexing.kafka.KafkaIndexTaskTuningConfig;
 import org.apache.druid.indexing.kafka.KafkaRecordSupplier;
-import org.apache.druid.indexing.kafka.test.TestBroker;
+import org.apache.druid.indexing.kafka.test.EmbeddedKafkaBroker;


TestBroker no longer compiles (Scala kafka.server.KafkaServer removed in Kafka 4.0); embedded ZooKeeper mode no longer supported by Kafka 4.x brokers.

Shekharrajak · 2026-05-09T18:08:22Z

      <scope>test</scope>
    </dependency>
-    <dependency>
-      <groupId>org.apache.curator</groupId>


this was only used to embed ZooKeeper for TestBroker/TestKafkaExtractionCluster; KRaft mode means no ZooKeeper.

Shekharrajak · 2026-05-09T18:12:23Z

+    </dependency>
+    <dependency>
+      <groupId>org.testcontainers</groupId>
+      <artifactId>testcontainers-kafka</artifactId>


kafka container specific - that will help in integration testing and running containers

Shekharrajak · 2026-05-09T18:15:14Z

  @BeforeClass
  public static void setupClass() throws Exception
  {
-    zkServer = new TestingCluster(1);


KRaft mode - hence no zk

Shekharrajak · 2026-05-09T18:18:11Z

This will be the first step to improve & simplify the kafka integration testing , version upgrade to open the path for Queue Semantics Kafka Ingestion #19311

cc @gianm @a2l007 @clintropolis @FrankChen021

+  {
+    final String url = getBootstrapServerUrl();
+    final int colon = url.lastIndexOf(':');
+    return Integer.parseInt(url.substring(colon + 1));


Shekharrajak · 2026-05-09T18:36:25Z

It looks like a flaky test failure https://github.com/apache/druid/actions/runs/25607936768/job/75172943327?pr=19441

Shekharrajak · 2026-05-09T19:30:05Z

Please help in triggering the failing CI checks again.

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 14 of 14 changed files.

This is an automated review by Codex GPT-5.5

FrankChen021 · 2026-05-11T02:05:23Z

please check the error of this check: Static Checks CI / strict-compilation / strict-compilation (pull_request)Failing after 39m

…ent-4.2.0-upgrade

Shekharrajak · 2026-05-12T03:58:40Z

@FrankChen021 thanks for checking, CI checks are looking good now.

Once we have these changes ready the Queue implementation of phase 1 will be easier to review.

Shekharrajak · 2026-05-12T13:04:23Z

Please re-run the CI check https://github.com/apache/druid/actions/runs/25716029780/job/75506218870?pr=19441 - it is not looking related to our changes.

FrankChen021

I have reviewed the code for correctness, edge cases, concurrency, and integration risks; no issues found.

Reviewed 14 of 14 changed files.

This is an automated review by Codex GPT-5.5

Shekharrajak added 5 commits May 9, 2026 23:20

Bump Kafka client to 4.2.0 (apache#19322)

35339d5

Migrate Kafka Consumer.poll(long) to poll(Duration) for Kafka 4.x

1bf5da5

Add EmbeddedKafkaBroker test helper using Testcontainers

2fa589d

Migrate kafka-indexing-service tests to EmbeddedKafkaBroker; drop Tes…

7fec5d6

…tBroker

Migrate TestKafkaExtractionCluster to Testcontainers; bump licenses

ea3d1f9

github-actions Bot added Area - Streaming Ingestion Area - Lookups Area - Dependencies labels May 9, 2026

Shekharrajak changed the title ~~Feature/kafka client 4.2.0 upgrade~~ build(dep) Kafka client 4.2.0 upgrade May 9, 2026

Shekharrajak changed the title ~~build(dep) Kafka client 4.2.0 upgrade~~ build(deps): Kafka client 4.2.0 upgrade May 9, 2026

Shekharrajak commented May 9, 2026

View reviewed changes

github-advanced-security AI found potential problems May 9, 2026

View reviewed changes

Comment thread ...indexing-service/src/test/java/org/apache/druid/indexing/kafka/test/EmbeddedKafkaBroker.java

{

final String url = getBootstrapServerUrl();

final int colon = url.lastIndexOf(':');

return Integer.parseInt(url.substring(colon + 1));

Shekharrajak mentioned this pull request May 9, 2026

Flaky test: MultiPhaseParallelIndexingRowStatsTest.testHashPartitionRowStats fails on JDK 25 #19443

Closed

FrankChen021 reviewed May 10, 2026

View reviewed changes

Shekharrajak added 3 commits May 11, 2026 18:23

Merge remote-tracking branch 'upstream/master' into feature/kafka-cli…

0744131

…ent-4.2.0-upgrade

Drop unused throws Exception from setupClass in Kafka tests

0ea8e7d

Add curator-test test dep to kafka-indexing-service for embedded ZK

0f669b7

Shekharrajak force-pushed the feature/kafka-client-4.2.0-upgrade branch 2 times, most recently from 53afc19 to 0f669b7 Compare May 12, 2026 05:47

FrankChen021 reviewed May 12, 2026

View reviewed changes

FrankChen021 approved these changes May 12, 2026

View reviewed changes

FrankChen021 merged commit eb34e2a into apache:master May 13, 2026
100 of 101 checks passed

github-actions Bot added this to the 38.0.0 milestone May 13, 2026

Shekharrajak mentioned this pull request May 15, 2026

Flaky test: KafkaIndexFaultToleranceTest times out at Testcontainers Kafka startup #19464

Open

Conversation

Shekharrajak commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Release note

Key changed/added classes in this PR

Uh oh!

Shekharrajak May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Shekharrajak May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Shekharrajak May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Shekharrajak May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Shekharrajak May 9, 2026

Choose a reason for hiding this comment

Uh oh!

Shekharrajak commented May 9, 2026

Uh oh!

Shekharrajak commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shekharrajak commented May 9, 2026

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

FrankChen021 commented May 11, 2026

Uh oh!

Shekharrajak commented May 12, 2026

Uh oh!

Shekharrajak commented May 12, 2026

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shekharrajak commented May 9, 2026 •

edited

Loading

Shekharrajak commented May 9, 2026 •

edited

Loading