[None][perf] executor: in-process fast-path for rank-0 RPC submit

[lancelly]

Problem

On the RPC executor path (GenerationExecutorRpcProxy / RpcWorker), rank 0 is the sole RPC ingress for the whole instance (RpcWorkerMixin.start_rpc_server binds the server only on rank == 0). Under MpiCommSession (the common mpirun/srun launch), the rank-0 RpcWorker runs in a thread of the proxy process (MpiCommSession.submit dispatches rank 0 to a local ThreadPoolExecutor; ranks 1..N-1 go through MPI).

Yet the proxy still reaches that in-process worker over a loopback ZMQ socket: every submit does pickle + HMAC on send, ZMQ round-trip, HMAC + pickle.loads on receive, and a run_in_executor dispatch — all on rank 0's GIL, competing with the co-located executor loop (_executor_loop_overlap). For a local worker this serialization is pure self-inflicted overhead, and since submit rate scales with concurrency it starves the executor loop at high load (visible in nsys as RpcWorker.submit holding the GIL while the worker loop waits, e.g. at c2048).

Note ranks 1..N-1 do not receive requests via this Python RPC — they are fed by the C++ executor's internal request broadcast — so the per-request Python pickle/HMAC/ZMQ is a single proxy→rank-0 loopback hop.

Change

Add an opt-in in-process fast-path that lets the proxy call the rank-0 worker directly, skipping pickle / HMAC / ZMQ / dispatch entirely:

• utils.py — a per-process registry (register/unregister/get_local_rpc_worker) keyed by rpc_addr.
• rpc_worker.py — the rank-0 worker registers itself on start and unregisters on shutdown.
• rpc_proxy_mixin.py — submit() looks up the local worker by self.rpc_addr and, if present, calls worker.submit(request) directly; otherwise it uses RPC. The GenerationResult is now registered before dispatch so a response can never race ahead of result tracking.

The worker-side logic is unchanged (RpcWorker.submit → BaseWorker.submit), so result tracking, responses, abort, and stats all behave identically.

The local/remote decision is automatic per-process: the registry lives in the proxy's process, so in the spawn-proxy topology (worker in a different process) the lookup misses and it transparently falls back to RPC — no explicit topology detection.

Scope / knobs

Var	Default	Meaning
TLLM_RPC_LOCAL_FASTPATH	0 (off)	1 enables the in-process submit fast-path

Disabled by default — no behavior change unless opted in. Orthogonal to and stackable with batched submit (#15109): batching cuts the count of RPC ops; this removes the per-op serialization for the local worker.

This PR covers the submit hot path (Phase 1). The response stream (fetch_responses, O(C)/iter, also loopback-pickled today) can be moved to the same in-process path as a follow-up.

What it does and does not remove

• Removes: pickle/HMAC/ZMQ/run_in_executor on the proxy→rank-0 submit hop (the GIL-heavy pickle is the main win; HMAC mostly releases the GIL).
• Does not remove: the enqueue's intrinsic Python cost (list(prompt_token_ids), GenerationResult, dict bookkeeping) — those still run, now on the proxy thread (same process/GIL), and are addressed by separate changes.

Test plan

• c2048 DeepSeek-V4 disagg GEN: nsys python-gil — worker0 GIL-wait per decode step, RPC vs fast-path; verify RPCExecutor.submit.local replaces the pickle/ZMQ ranges.
• TTFT / throughput / output-speed at c400 and c2048 (off vs on); stack with [None][perf] executor: batch RPC submit to relieve rank-0 GIL contention #15109.
• Correctness: streaming + non-streaming, abort, disaggregated gen-only; confirm RPC fallback in spawn-proxy topology.
• Audit reference-sharing: ensure neither proxy nor worker mutates the GenerationRequest after hand-off (no pickle copy on the local path).

Draft — opening for early review / to run the A/B above.