throttling for concurrent topic loading on broker cold-restart

[rdhabalia]

In regards of #196, we performed one the quick load test which loads 30K topics under 5K bundles and it continuously try to unload all the bundles from each broker concurrently which makes all topics to be loaded to different brokers again and again. As each topic loading executes around ~8 zk operations, it increases zk-latency and zk-outstanding request numbers also. So, it seems on cold-restart: broker should have a way to throttle concurrent topic-loading to avoid creating back pressure on zk which can help to control zk latency and help broker to not lose zk-session and avoid recursive restart.
@merlimat any thought?

[merlimat]

Is the 5K bundles realistic?
Wasn't the model to be more like: 1 bundle == 1K topics ?

The unload is already working bundle by bundle, we would just need to throttle on the load path.

I think it can be either done at the bundle or (probably better) at the topic level, defining a max number of concurrent load and deferring the rest.

The "defer" could be done in few different ways:

• Fail the load and let the clients retry later
• Or, just queue the topic load request and whenever we're done loading a topic check the queue for more topics to resume loading.

[rdhabalia]

Is the 5K bundles realistic?
Wasn't the model to be more like: 1 bundle == 1K topics ?

actually, while running load test topics were not distributed equally to each bundle so, topics couldn't distributed uniformly.

and yes, we have made similar change as you mentioned and will perform similar load test with it.