Skip to content

KAFKA-16954: fix consumer close to release assignment in background#16376

Merged
lucasbru merged 3 commits intoapache:3.8from
lianetm:3.8-kip-848-client-close
Jun 18, 2024
Merged

KAFKA-16954: fix consumer close to release assignment in background#16376
lucasbru merged 3 commits intoapache:3.8from
lianetm:3.8-kip-848-client-close

Conversation

@lianetm
Copy link
Copy Markdown
Member

@lianetm lianetm commented Jun 17, 2024

Cherry-picking 2 commits required for fixing KAFKA-16954 for 3.8 (fix consumer close to release assignment in background)

lianetm and others added 3 commits June 17, 2024 16:53
@lianetm
MINOR: Improving log for outstanding requests on close and cleanup (a…
e56a0de 
…pache#16304)

Reviewers: Andrew Schofield <aschofield@confluent.io>, Chia-Ping Tsai <chia7712@gmail.com>
@lianetm
KAFKA-16954: fix consumer close to release assignment in background (a…
b01c578 
…pache#16343)

This PR fixes consumer close to avoid updating the subscription state object in the app thread. Now the close simply triggers an UnsubscribeEvent that is handled in the background to trigger callbacks, clear assignment, and send leave heartbeat. Note that after triggering the event, the unsubscribe will continuously process background events until the event completes, to ensure that it allows for callbacks to run in the app thread.
The logic around what happens if the unsubscribe fails remain unchanged: close will log, keep the first exception and carry on.

It also removes the redundant LeaveOnClose event (it used to do the the exact same thing as the UnsubscribeEvent, both calling membershipMgr.leaveGroup).

Reviewers: Lucas Brutschy <lbrutschy@confluent.io>
@lianetm
Remove unrelated cleanup
ceb4ace
@lianetm
Copy link
Copy Markdown
Member Author

lianetm commented Jun 17, 2024

Hey @lucasbru , here is the PR cherry-picking only the required changes for 3.8 for the consumer close fix. The conflicts were due to this client cleanup commit 79ea7d6 that is not in 3.8 (and it's not needed), so I left it out. Thanks!

@lianetm lianetm changed the title 3.8 cherry-pick for KAFKA-16954 KAFKA-16954: fix consumer close to release assignment in background Jun 17, 2024
@lianetm
Copy link
Copy Markdown
Member Author

lianetm commented Jun 18, 2024

Build completed with 4 unrelated failures:

Build / JDK 8 and Scala 2.12 / testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest
Build / JDK 8 and Scala 2.12 / testTaskRequestWithOldStartMsGetsUpdated() – org.apache.kafka.trogdor.coordinator.CoordinatorTest
Build / JDK 21 and Scala 2.13 / "testNoConsumeWithDescribeAclViaSubscribe(String).quorum=kraft" – kafka.api.DelegationTokenEndToEndAuthorizationWithOwnerTest
Build / JDK 21 and Scala 2.13 / testDescribeQuorumReplicationSuccessful [1] Type=Raft-Isolated, MetadataVersion=4.0-IV0,Security=PLAINTEXT – org.apache.kafka.tools.MetadataQuorumCommandTest

lucasbru
lucasbru approved these changes Jun 18, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@lucasbru lucasbru left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@lucasbru lucasbru merged commit 86abaf2 into apache:3.8 Jun 18, 2024
@lucasbru
Copy link
Copy Markdown
Member

lucasbru commented Jun 18, 2024

@jlprat Fix for blocker KAFKA-16954 merged to 3.8 branch

@jlprat
Copy link
Copy Markdown
Contributor

jlprat commented Jun 18, 2024

Thanks @lucasbru , closed the Jira issue as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucasbru lucasbru lucasbru approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lianetm @lucasbru @jlprat