[None][feat] serve: multi-process HTTP frontends on the classic IPC executor path - #16413
Closed
lancelly wants to merge 1 commit into
Closed
[None][feat] serve: multi-process HTTP frontends on the classic IPC executor path#16413lancelly wants to merge 1 commit into
lancelly wants to merge 1 commit into
Conversation
…xecutor path At high concurrency a trtllm-serve worker is host-bound on its single serving process: one asyncio event loop (one GIL) performs every request json.loads+pydantic validate and every SSE chunk write, and queuing on that loop dominates first-token latency while the GPUs idle (measured on DSv4 disagg GEN: gen_preprocessing p50 54ms@c512 -> 1520ms@c2048). vLLM/SGLang address the same limit with multiple API-server processes. TLLM_SERVE_NUM_FRONTENDS=K (env-gated, default off) runs K HTTP frontend processes against ONE executor on the default (classic IPC) orchestrator: - Launcher/attach split: frontend 0 builds the LLM and launches workers as usual, then spawns K-1 children that re-exec the command line with TLLM_EXECUTOR_ATTACH_INFO set; their executor attaches via the new GenerationExecutorFrontendProxy (no MPI session, no worker launch, and shutdown never emits the worker's None engine-shutdown sentinel -- the launcher alone owns the engine lifecycle). - Deterministic ipc endpoints, pre-generated by the launcher (TLLM_MULTI_FRONTEND_IPC_DIR/_HMAC): the rank0 worker BINDS the request ingress (PULL) so every frontend PUSH-connects; each frontend binds its own result lane (PULL) that the worker (non-postproc) or every postprocess worker (one PUSH pipe per frontend) sends to. - client-id namespacing: the top 16 bits of the uint64 client id carry the frontend id; responses are routed by client_id>>48. Responses without a usable client id (e.g. attention-DP dummy requests carry client_id=None) route to the launcher, preserving today's silent discard semantics. Frontend 0 keeps ids bit-identical to today. - All frontends bind the public port with SO_REUSEPORT, so the kernel load-balances accepted connections across their independent processes (and GILs); clients and the disagg orchestrator still see one URL. Known limitations (documented): /metrics and /perf_metrics are served per-frontend (SO_REUSEPORT samples one frontend per request); enable_resource_governor is rejected in multi-frontend mode. Validation: 12 CPU-only unit tests (id namespacing, response-lane bucketing incl. the ADP-dummy None guard, attached-proxy submit/cancel/ never-sends-sentinel over real ipc sockets). E2E A/B on DSv4-Pro disagg (11xCTX-DEP4 + 1xGEN-DEP16, c3120, back-to-back on identical nodes): gen_preprocessing p50 1075ms -> 37.6ms (-96.5%), p95 3145ms -> 75.7ms; per-request SSE speed p50 +30%; 16 frontends, 75 min, zero tracebacks; engine-side phases (gen_queue, kv_transfer) byte-identical to baseline. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>
lancelly
force-pushed
the
feat/serve-multi-frontend
branch
from
July 16, 2026 05:27
5632b93 to
e4cbe62
Compare
lancelly
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
At high concurrency a
trtllm-serveworker is host-bound on its single serving process: one asyncio event loop (one GIL) performs every requestjson.loads+pydantic validate and every SSE chunk write. Queuing on that loop — not GPU time — then dominates first-token latency (measured on DSv4 disagg GEN:gen_preprocessingp50 54ms@c512 → 109ms@c1024 → 1520ms@c2048, an M/M/1-style blow-up while decode step time stayed flat).This PR adds a prototype multi-frontend mode on the default (classic IPC) executor path:
TLLM_SERVE_NUM_FRONTENDS=K(env-gated, default off) runs K HTTP frontend processes against one executor.Design
TLLM_EXECUTOR_ATTACH_INFOset; their executor attaches via the newGenerationExecutorFrontendProxy(no MPI session, no worker launch, and shutdown never emits the worker'sNoneengine-shutdown sentinel — the launcher alone owns the engine lifecycle).TLLM_MULTI_FRONTEND_IPC_DIR/_HMAC): the rank0 worker binds the request ingress (PULL) so every frontend PUSH-connects; each frontend binds its own result lane (PULL) that the worker (non-postproc) or every postprocess worker (one PUSH pipe per frontend) sends to.client_id>>48. Responses without a usable client id (e.g. attention-DP dummy requests carryclient_id=None) route to the launcher, preserving today's silent-discard semantics. Frontend 0 / feature-off keeps ids bit-identical to today.Scope / limitations (prototype)
orchestrator_type: rpc/rayare rejected with a clear error (an earlier RPC-path variant was dropped after e2e showed a response-stream race — can be revisited separately).num_serve_frontends) can follow review./metrics,/perf_metrics,/healthare served per-frontend (SO_REUSEPORT samples one frontend per request); cross-frontend aggregation is follow-up work.enable_resource_governoris rejected in multi-frontend mode (the governor signal only reaches the launcher).