[SPARK-49064][BUILD] Upgrade Kafka to 3.8.0#47540
[SPARK-49064][BUILD] Upgrade Kafka to 3.8.0#47540panbingkun wants to merge 4 commits intoapache:masterfrom
Conversation
panbingkun
changed the title
|
Since the title is changed, could you make it |
| private val cleanupLogsPrivateMethod = PrivateMethod[LogManager](Symbol("cleanupLogs")) | ||
| def cleanupLogs(): Unit = { | ||
| server.logManager.cleanupLogs() | ||
| server.logManager.invokePrivate(cleanupLogsPrivateMethod()) |
There was a problem hiding this comment.
It seems that Kafka community changed this without the official KAFKA issue.
There was a problem hiding this comment.
Yea, that's right, it was this PR that caused this change.
| messages.foreach { case (k, v) => | ||
| val record = new SimpleRecord(k.getBytes, v.getBytes) | ||
| log.appendAsLeader(MemoryRecords.withRecords(CompressionType.NONE, record), 0); | ||
| log.appendAsLeader(MemoryRecords.withRecords(Compression.NONE, Seq(record): _*), 0); |
There was a problem hiding this comment.
This is a new feature of Kafka 3.8.0 (KAFKA-7632, KIP-390).
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM for Apache Spark 4.0.0-preview2. Thank you for keeping tracking this area, @panbingkun .
| } | ||
|
|
||
| private val cleanupLogsPrivateMethod = PrivateMethod[LogManager](Symbol("cleanupLogs")) | ||
| def cleanupLogs(): Unit = { |
There was a problem hiding this comment.
In v3.8.0, the access modifier of method cleanupLogs has been changed to private
- v3.7.1
https://github.com/apache/kafka/blob/3.7.1/core/src/main/scala/kafka/log/LogManager.scala#L1390
- v3.8.0
https://github.com/apache/kafka/blob/3.8.0/core/src/main/scala/kafka/log/LogManager.scala#L1395
| val record = new SimpleRecord(k.getBytes, v.getBytes) | ||
| log.appendAsLeader(MemoryRecords.withRecords(CompressionType.NONE, record), 0); | ||
| log.appendAsLeader(MemoryRecords.withRecords(Compression.NONE, Seq(record): _*), 0); | ||
| } |
There was a problem hiding this comment.
In v3.8.0, the signature of method withRecords has been changed to MemoryRecords withRecords(Compression compression, ...)
Done. |
Thank you for your review! ❤️ |
|
The failed Merged to master for Apache Spark 4.0.0-preview2. |
2 participants
What changes were proposed in this pull request?
The pr aims to upgrade
kafkafrom3.7.1to3.8.0.Why are the changes needed?
https://downloads.apache.org/kafka/3.8.0/RELEASE_NOTES.html
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA.
Was this patch authored or co-authored using generative AI tooling?
No.