[None][perf] executor: batch RPC submit to relieve rank-0 GIL contention

[lancelly]

Problem

On the RPC executor path (GenerationExecutorRpcProxy / RpcWorker), rank 0 is the sole RPC ingress for the whole instance (RpcWorkerMixin.start_rpc_server binds the server only on rank == 0) and also runs the co-located rank-0 model worker. Every request therefore pays, on rank 0:

• pickle + HMAC of the GenerationRequest (incl. prompt_token_ids) on send and the matching pickle.loads on receive,
• a ZMQ round-trip,
• a run_in_executor dispatch on the RPC server thread pool,
• and a GIL acquisition,

all competing with the executor loop on the same GIL. This per-request overhead scales with concurrency, so at high load the executor loop (worker0) gets starved. In nsys this shows up as RpcWorker.submit holding the GIL while the worker loop waits — mild at low concurrency (e.g. c400) but severe at high concurrency (e.g. c2048).

#14889 (avoid deepcopy of prompt_token_ids on enqueue) cut a per-request constant; it does not change how the cost scales with concurrency.

Change

Add an opt-in batched-submit path that coalesces requests on the proxy into a single submit_batch RPC, collapsing the fixed per-request costs (RPC framing, pickle/HMAC call, ZMQ op, dispatch, GIL acquisition count) by ~the batch size:

• RpcWorkerMixin.submit_batch(requests) — enqueue a list of requests in one RPC. The C++ enqueue inside super().submit() releases the GIL, so the loop yields naturally between requests.
• RpcExecutorMixin — buffer requests and flush on size (TLLM_RPC_SUBMIT_BATCH_MAX) or after a short delay (TLLM_RPC_SUBMIT_BATCH_DELAY_MS). The time-triggered flush runs on the proxy main loop and uses remote_future (non-blocking) so it never blocks the loop or the response path.
• The GenerationResult is now registered before sending, so a response can never arrive for an untracked client_id.

This is orthogonal to (and stacks with) shrinking the per-request payload (e.g. shipping prompt_token_ids as a raw ZMQ frame instead of pickling) — batching cuts the count-based overheads, payload work cuts the bytes-based ones.

Knobs (env)

Var	Default	Meaning
TLLM_RPC_SUBMIT_BATCH_MAX	1 (disabled)	Max requests per coalesced RPC; >1 enables batching
TLLM_RPC_SUBMIT_BATCH_DELAY_MS	0.5	Straggler-flush window for partially-filled batches

Disabled by default — no behavior change unless opted in.

Test plan

• A/B at high concurrency (c2048, DSeek-V4 disagg GEN): nsys python-gil — fraction of time worker0 spends in GIL-wait per decode step, with batching off vs on; sweep TLLM_RPC_SUBMIT_BATCH_MAX ∈ {8,16,32,64}.
• TTFT / throughput / output-speed at c400 and c2048 (off vs on).
• Correctness: streaming + non-streaming, abort, disaggregated gen-only.

Draft — opening for early review / to run the A/B above.