[feat] PIP-468: Scalable Topic Controller

[merlimat]

Summary

Adds PIP-468, a sub-PIP of PIP-460: Scalable Topics, describing the broker-side Scalable Topic Controller — the component responsible for segment lifecycle management, split/merge coordination, consumer assignment, leader election, and DAG watch sessions for scalable topics.

The document covers:

• Architecture — the four-layer design (Admin API → ScalableTopicService → ScalableTopicController → ScalableTopicResources) and the three broker roles (any broker for DAG watch sessions, controller leader for layout mutations and consumer assignment, segment-owning brokers for produce/consume).
• Split and merge protocols — the step-by-step create-first / update-metadata-last ordering that guarantees no messages are lost and all subscription cursors exist before producers are redirected, plus cross-broker coordination via the Segments admin API.
• Consumer assignment — StreamConsumer / CheckpointConsumer receive coordinated assignments from the controller leader; QueueConsumer attaches to all segments with no controller-side assignment. Includes a detailed Consumer Session Lifecycle section clarifying what is persisted (the ConsumerRegistration) vs. what is in-memory on the leader (the keep-alive state and grace-period timer), and how controller leader failover restarts grace-period timers from zero.
• Leader election, failover, and metadata store schema — including the new SubscriptionMetadata and ConsumerRegistration paths.
• Public-facing changes — new binary protocol commands (CommandScalableTopicLookup / Update / Close), new Admin REST endpoints for scalable topics and segments, new ScalableTopics admin client interface, and the scalableTopicEnabled broker configuration.

This PIP only changes documentation under pip/. Implementation will land in subsequent PRs against the parent PIP-460 tracking issue.

Test plan

• Review the document on the PIP mailing list thread (TBD link to be added once thread is open).

[lhotari]

some initial comments

[lhotari]

LGTM

[liangyepianzhou]

I have a question: if my ScalableTopic has a large number of sub-partitions — say, 1 million — would there be a situation where every broker needs to perform calculations across all 1 million sub-partitions of that ScalableTopic just to determine which sub-partitions belong to it? I'm a bit worried about scalability issues caused by this kind of metadata computation.

[merlimat]

I have a question: if my ScalableTopic has a large number of sub-partitions — say, 1 million — would there be a situation where every broker needs to perform calculations across all 1 million sub-partitions of that ScalableTopic just to determine which sub-partitions belong to it? I'm a bit worried about scalability issues caused by this kind of metadata computation.

The number of "segments" is dynamically adjusted based on several factors (eg: current/historic traffic, consumer parallelisms), though there would never be a reason to have such a large number of segments. Most likely scenario is for segments to be at most few 100s, and in some exceptional cases to touch 1000s.

The reverse can be true though: eg. having millions of scalable topics (with most of the having few or just one segment)

[liangyepianzhou]

I have a question: if my ScalableTopic has a large number of sub-partitions — say, 1 million — would there be a situation where every broker needs to perform calculations across all 1 million sub-partitions of that ScalableTopic just to determine which sub-partitions belong to it? I'm a bit worried about scalability issues caused by this kind of metadata computation.

The number of "segments" is dynamically adjusted based on several factors (eg: current/historic traffic, consumer parallelisms), though there would never be a reason to have such a large number of segments. Most likely scenario is for segments to be at most few 100s, and in some exceptional cases to touch 1000s.

The reverse can be true though: eg. having millions of scalable topics (with most of the having few or just one segment)

In scenarios where users require strict ordering and very slow consumption speeds, and considering that k-shared mode may cause out-of-order delivery upon consumer disconnection — or in cases where a proxy is inserted between the broker and client, making k-shared unusable — the only option is to increase the number of partitions to boost consumption concurrency. This could result in a single ScalableTopic having millions of sub-partitions.

In such scenarios, can the new design avoid the scalability issues caused by metadata computation?

[merlimat]

I have a question: if my ScalableTopic has a large number of sub-partitions — say, 1 million — would there be a situation where every broker needs to perform calculations across all 1 million sub-partitions of that ScalableTopic just to determine which sub-partitions belong to it? I'm a bit worried about scalability issues caused by this kind of metadata computation.

The number of "segments" is dynamically adjusted based on several factors (eg: current/historic traffic, consumer parallelisms), though there would never be a reason to have such a large number of segments. Most likely scenario is for segments to be at most few 100s, and in some exceptional cases to touch 1000s.
The reverse can be true though: eg. having millions of scalable topics (with most of the having few or just one segment)

In scenarios where users require strict ordering and very slow consumption speeds, and considering that k-shared mode may cause out-of-order delivery upon consumer disconnection — or in cases where a proxy is inserted between the broker and client, making k-shared unusable — the only option is to increase the number of partitions to boost consumption concurrency. This could result in a single ScalableTopic having millions of sub-partitions.

In such scenarios, can the new design avoid the scalability issues caused by metadata computation?

Scaling up the number of segments (or partitions for what it matters) for this scenario is not an efficient way of solving the problem.

Scalable topics are going to address these scenarios with the same mechanism used today for KeyShared subscriptions.