[TRTLLM-14510][feat] serve: multi-process HTTP frontends on the classic IPC executor path by lancelly · Pull Request #16523 · NVIDIA/TensorRT-LLM

lancelly · 2026-07-17T03:48:07Z

Description

At high concurrency a trtllm-serve worker is host-bound on its single serving process: one asyncio event loop (one GIL) performs every request json.loads+pydantic validate and every SSE chunk write. Queuing on that loop — not GPU time — then dominates first-token latency (measured on DSv4 disagg GEN: gen_preprocessing p50 54ms@c512 → 109ms@c1024 → 1520ms@c2048, an M/M/1-style blow-up while decode step time stayed flat).
This PR adds a prototype multi-frontend mode on the default (classic IPC) executor path: num_serve_frontends: K (llm-args knob / --num_serve_frontends, default 1 = off, max 64) runs K HTTP frontend processes against one executor.

Design

Launcher/attach split — the launcher (frontend 0) builds the LLM and launches workers as usual, then spawns K-1 children that re-exec the command line with TLLM_EXECUTOR_ATTACH_INFO set; their executor attaches via the new GenerationExecutorFrontendProxy (no MPI session, no worker launch, and shutdown never emits the worker's None engine-shutdown sentinel — the launcher alone owns the engine lifecycle).
Deterministic ipc endpoints, pre-generated by the launcher (TLLM_MULTI_FRONTEND_IPC_DIR/_HMAC): the rank0 worker binds the request ingress (PULL) so every frontend PUSH-connects; each frontend binds its own result lane (PULL) that the worker (non-postproc) or every postprocess worker (one PUSH pipe per frontend) sends to.
client-id namespacing — the top 16 bits of the uint64 client id carry the frontend id; responses route by client_id>>48. Responses without a usable client id (e.g. attention-DP dummy requests carry client_id=None) route to the launcher, preserving today's silent-discard semantics. Frontend 0 / feature-off keeps ids bit-identical to today.
SO_REUSEPORT — all frontends bind the public port; the kernel load-balances accepted connections across independent processes (and GILs). Clients and the disagg orchestrator still see one URL.

Scope / limitations (prototype)

Classic IPC executor path only; orchestrator_type: rpc/ray are rejected with a clear error.
Configured via the num_serve_frontends llm-args knob (prototype status), capped at 64.
/metrics, /perf_metrics, /health are served per-frontend (SO_REUSEPORT samples one frontend per request); cross-frontend aggregation is follow-up work.
enable_resource_governor is rejected in multi-frontend mode (the governor signal only reaches the launcher).

Summary by CodeRabbit

New Features
- Added support for running multiple HTTP serving frontends per executor.
- Added the num_serve_frontends configuration option, supporting 1–64 frontends.
- Improved request and response routing so each frontend receives its corresponding results.
- Added frontend attachment support for connecting to an existing serving process.
Bug Fixes
- Prevented attached frontends from unnecessarily starting duplicate worker sessions.
Tests
- Added coverage for multi-frontend routing, client isolation, response delivery, and shutdown behavior.

…xecutor path At high concurrency a trtllm-serve worker is host-bound on its single serving process: one asyncio event loop (one GIL) performs every request json.loads+pydantic validate and every SSE chunk write, and queuing on that loop dominates first-token latency while the GPUs idle (measured on DSv4 disagg GEN: gen_preprocessing p50 54ms@c512 -> 1520ms@c2048). vLLM/SGLang address the same limit with multiple API-server processes. TLLM_SERVE_NUM_FRONTENDS=K (env-gated, default off) runs K HTTP frontend processes against ONE executor on the default (classic IPC) orchestrator: - Launcher/attach split: frontend 0 builds the LLM and launches workers as usual, then spawns K-1 children that re-exec the command line with TLLM_EXECUTOR_ATTACH_INFO set; their executor attaches via the new GenerationExecutorFrontendProxy (no MPI session, no worker launch, and shutdown never emits the worker's None engine-shutdown sentinel -- the launcher alone owns the engine lifecycle). - Deterministic ipc endpoints, pre-generated by the launcher (TLLM_MULTI_FRONTEND_IPC_DIR/_HMAC): the rank0 worker BINDS the request ingress (PULL) so every frontend PUSH-connects; each frontend binds its own result lane (PULL) that the worker (non-postproc) or every postprocess worker (one PUSH pipe per frontend) sends to. - client-id namespacing: the top 16 bits of the uint64 client id carry the frontend id; responses are routed by client_id>>48. Responses without a usable client id (e.g. attention-DP dummy requests carry client_id=None) route to the launcher, preserving today's silent discard semantics. Frontend 0 keeps ids bit-identical to today. - All frontends bind the public port with SO_REUSEPORT, so the kernel load-balances accepted connections across their independent processes (and GILs); clients and the disagg orchestrator still see one URL. Known limitations (documented): /metrics and /perf_metrics are served per-frontend (SO_REUSEPORT samples one frontend per request); enable_resource_governor is rejected in multi-frontend mode. Validation: 12 CPU-only unit tests (id namespacing, response-lane bucketing incl. the ADP-dummy None guard, attached-proxy submit/cancel/ never-sends-sentinel over real ipc sockets). E2E A/B on DSv4-Pro disagg (11xCTX-DEP4 + 1xGEN-DEP16, c3120, back-to-back on identical nodes): gen_preprocessing p50 1075ms -> 37.6ms (-96.5%), p95 3145ms -> 75.7ms; per-request SSE speed p50 +30%; 16 frontends, 75 min, zero tracebacks; engine-side phases (gen_queue, kv_transfer) byte-identical to baseline. Ported to main from the feat/deepseek_v4-based branch (PR NVIDIA#16413), with review cleanups folded in: attached-frontend child env now uses split_mpi_env() (strips SLURM_/UCX_/PMI_ etc., not just 3 prefixes); TLLM_SERVE_NUM_FRONTENDS parsing/validation unified in get_num_serve_frontends(); result-lane selection unified in frontend_lane_index(); stale rpc_common docstring reference fixed. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…e env gate Review follow-ups on the multi-frontend prototype: - The attach-info JSON (mkstemp, carries the executor HMAC keys in hex) and the shared ipc directory (mkdtemp, holds the request/result .sock files) were never removed: every multi-frontend run leaked a secret-bearing file plus a directory in /tmp. launch_server's finally now unlinks the attach file and rmtree's the ipc dir after the attached frontends have been terminated (established zmq connections are not disturbed by unlinking ipc paths; the engine can still drain its lanes at teardown). - launch_server read TLLM_SERVE_NUM_FRONTENDS from the global env unconditionally, but disaggregated ctx/gen MPI workers reach launch_server too (_launch_disaggregated_server) and inherit the submitter's env under Slurm export-all: an exported knob would mis-switch them into multi-frontend mode. launch_server gains multi_frontend_enabled (default True); the disagg worker path passes False and an ignored knob logs a warning. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Extract the env-gate block at the top of launch_server into _init_multi_frontend_mode(), returning a MultiFrontendMode NamedTuple whose .active / .is_launcher properties replace the repeated 'multi_frontend and not is_attached_frontend' expressions at the SO_REUSEPORT, spawn, and cleanup sites. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…sponse routing hot path Every response type that reaches _send_rsp and bucket_responses_by_frontend (tllm.Response, ErrorResponse, ResponseWrapper via __getattr__ delegation, PostprocWorker.Output) defines client_id, and the None-VALUE case (ADP dummy requests) is already handled by frontend_lane_index. The getattr(..., None) default was never exercised for a missing attribute; worse, it would silently route a hypothetical malformed response to lane 0 (where the launcher discards it, hanging the client) instead of failing loudly. Use direct attribute access: cheaper on the per-response loop and loud on real bugs. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…serve_frontends knob Promote the multi-frontend switch from TLLM_SERVE_NUM_FRONTENDS to a typed config knob, following the vLLM --api-server-count shape: - llm_args gains num_serve_frontends (int, default 1, ge=1 le=65536, status=prototype) next to the other serving knobs; exposed on trtllm-serve as --num_serve_frontends and via the config yaml. api_stability reference updated. - GenerationExecutorProxy now reads the count from llm_args and OWNS the shared ipc directory + HMAC key (generated in __init__, removed in shutdown): the TLLM_MULTI_FRONTEND_IPC_DIR / TLLM_MULTI_FRONTEND_HMAC env side-channel between serve.py and the proxy is deleted, and the ipc-dir cleanup moves from serve.py to the proxy that created it. - serve.py _init_multi_frontend_mode normalizes the knob into llm_args; TLLM_SERVE_NUM_FRONTENDS is kept as a fallback for when the knob is unset (existing deployments), translated to the knob at the CLI entry. The disabled entry points (disagg MPI workers) now also scrub the knob from llm_args so the executor cannot enter multi-frontend mode there. - executor.py validates the TLLM_EXECUTOR_FRONTEND_ID / TLLM_EXECUTOR_ATTACH_INFO pairing with a clear error instead of a raw KeyError. The attach-info env+file bootstrap remains: it crosses the child re-exec boundary, which a config knob cannot. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…op the env fallback 64 frontends already exceeds what one node's cores can usefully serve; the previous 1<<16 bound was the client-id encoding capacity, not a sane policy limit (the 16-bit wire format is unchanged). TLLM_SERVE_NUM_FRONTENDS is no longer honored: the knob is the only switch. A set env now logs a pointer to --num_serve_frontends instead of being silently ignored. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…ghten comments TLLM_SERVE_NUM_FRONTENDS never shipped in a release, so there is nothing to deprecate -- remove the warning shim entirely; the knob is the only switch. Also condense the multi-frontend comments/docstrings: the architecture is described once (MultiFrontendMode / GenerationExecutorFrontendProxy); other sites keep only the local constraint they enforce. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…tattr for the frontend count llm_args can legitimately be None (legacy TensorRT path), but any non-None llm_args has the num_serve_frontends field (BaseLlmArgs), so handle the None case explicitly like the adjacent lines and let a renamed field fail loud. The 'or 1' was dead: pydantic enforces ge=1. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

The 1<<16 upper bound (wire-format capacity) was dead code after the num_serve_frontends<=64 cap: the lane-count check already rejects anything >= len(result_addrs). Fold both into one range check with a message that names the valid ids. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…ot a temp file Review feedback (reasonsolo): serialize the attach payload directly into TLLM_EXECUTOR_ATTACH_INFO instead of writing a 0600 temp file and pointing the env at it. This deletes the file lifecycle entirely (no mkstemp, no unlink, no key left on disk after a hard kill), and the child pops the env once consumed so the HMAC keys cannot leak into descendant processes. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Review feedback (reasonsolo): drop the single-use 'active' property (its one call site reads clearer as launcher-or-attached), fold its definition into is_launcher, and shorten the docstrings. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

…ally Review feedback (reasonsolo): no backward compatibility needed for the in-tree call chain -- worker_main always passes a list (a single lane in single-frontend mode), so PostprocWorker/postproc_worker_main drop the Union[tuple, List[tuple]] signatures and the isinstance normalization. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Review feedback (reasonsolo): fold the disabled-path get/warn/pop into one pop-with-default, drop the duplicated knob read, and shorten the docstring and error text. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Review feedback (reasonsolo) on the id encoding: - The launcher (lane 0) now applies the same namespace rule as attached frontends whenever multi-frontend is active, so a long-lived request counter can never bleed into the frontend-id bits and misroute responses (it was an unguarded invariant before; the re-encode is a numeric no-op until the counter wraps). Single-frontend mode keeps raw ids. - The frontend-id field width is now the single source of truth: FRONTEND_ID_BITS = 6 derives MAX_NUM_FRONTENDS (64), the shift and the counter mask. The CLI cap imports the constant; the llm_args bound cannot (import cycle) and is pinned by a unit test instead. - The field sits just below the sign bit (shift 57): bit 63 stays clear, so ids remain positive in signed-int64 contexts. Field order stays frontend-id-high: the launcher's ids keep their numeric values, and a stray un-namespaced id degrades to lane 0 (the launcher) instead of spraying across lanes. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

coderabbitai · 2026-07-17T09:41:30Z

📝 Walkthrough

Walkthrough

Adds --num_serve_frontends for classic executor serving, launches attached HTTP frontends sharing a port, creates frontend-specific IPC result lanes, namespaces client IDs, routes responses to lanes, and introduces an attached frontend executor proxy with lifecycle tests.

Changes

Multi-Frontend Serving

Layer / File(s)	Summary
Serve configuration and process orchestration `tensorrt_llm/commands/serve.py`, `tensorrt_llm/llmapi/llm.py`, `tensorrt_llm/llmapi/llm_args.py`, `tests/unittest/api_stability/references/llm.yaml`	Adds the frontend-count API and CLI option, propagates it into LLM arguments, distinguishes launcher and attached processes, enables shared-port binding, manages child frontends, and excludes disaggregated workers.
Frontend-aware queue and response routing `tensorrt_llm/executor/utils.py`, `tensorrt_llm/executor/base_worker.py`, `tensorrt_llm/executor/worker.py`, `tensorrt_llm/executor/postproc_worker.py`	Adds client-ID namespacing, IPC address helpers, per-frontend result queues, lane-aware worker dispatch, and multi-output postprocessing.
Attached frontend executor lifecycle `tensorrt_llm/executor/proxy.py`, `tensorrt_llm/executor/executor.py`	Creates launcher IPC attachment metadata, adds `GenerationExecutorFrontendProxy`, bypasses executor startup for attached frontends, and handles frontend-specific shutdown and cleanup.
Routing and compatibility validation `tests/unittest/executor/test_multi_frontend_routing.py`, `tests/unittest/llmapi/test_executor.py`	Validates client-ID encoding, response bucketing, lane routing, attached proxy request and shutdown behavior, and the updated postprocessing-worker argument shape.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ServeCLI
  participant GenerationExecutorProxy
  participant worker_main
  participant PostprocWorker
  participant FrontendProxy
  ServeCLI->>GenerationExecutorProxy: configure frontend lanes
  GenerationExecutorProxy->>worker_main: provide shared request and result IPC addresses
  ServeCLI->>FrontendProxy: spawn with attachment metadata
  FrontendProxy->>worker_main: submit namespaced request
  worker_main->>PostprocWorker: route response to frontend lane
  PostprocWorker-->>FrontendProxy: deliver response
  FrontendProxy->>FrontendProxy: cancel own requests on shutdown

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise, specific, and accurately summarizes the main change: multi-process HTTP frontends on the classic IPC executor path.
Description check	✅ Passed	The description covers the problem, design, scope, and validation, though it lacks the template’s explicit Test Coverage and PR Checklist sections.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

lancelly · 2026-07-17T09:48:27Z

/bot run --disable-fail-fast

lancelly · 2026-07-17T09:49:11Z

/bot run --disable-fail-fast

… tests Frontend shutdown deliberately leaves the dispatch thread to process teardown (closing its ZMQ socket from another thread is not safe), so the two proxy end-to-end tests left proxy_dispatch_result_thread running and pytest-threadleak failed them in the CI CPU stage. End each test the way the worker ends the thread at engine teardown: push the per-lane None sentinel from the fake worker and join the thread, which also covers the sentinel exit path. Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

lancelly · 2026-07-21T12:27:37Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-21T12:33:55Z

PR_Github #60690 [ run ] triggered by Bot. Commit: f012263 Link to invocation

tensorrt-cicd · 2026-07-21T16:03:17Z

PR_Github #60690 [ run ] completed with state FAILURE. Commit: f012263
/LLM/main/L0_MergeRequest_PR pipeline #48982 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-21T23:06:54Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-21T23:12:43Z

PR_Github #60809 [ run ] triggered by Bot. Commit: f012263 Link to invocation

tensorrt-cicd · 2026-07-22T03:27:54Z

PR_Github #60809 [ run ] completed with state SUCCESS. Commit: f012263
/LLM/main/L0_MergeRequest_PR pipeline #49085 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lancelly · 2026-07-22T03:34:37Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-07-22T03:41:08Z

PR_Github #60871 [ run ] triggered by Bot. Commit: f012263 Link to invocation

tensorrt-cicd · 2026-07-22T04:44:29Z

PR_Github #60871 [ run ] completed with state SUCCESS. Commit: f012263
/LLM/main/L0_MergeRequest_PR pipeline #49145 completed with status: 'SUCCESS'

CI Report

Link to invocation

…ool-session test test_proxy_fast_death builds GenerationExecutorProxy / MpiPoolSession via __new__ (bypassing __init__), then drives shutdown(). A recent main change (NVIDIA#16523, multi-process HTTP frontends) made GenerationExecutorProxy.shutdown() -> _cleanup_multi_frontend_ipc_dir() read self._multi_frontend_ipc_dir unguarded, so the __new__-built proxies raise AttributeError at GC/teardown (test_shutdown_does_not_block_on_dead_engine and the two sibling shutdown tests). This is a cross-PR merge-skew: NVIDIA#16312 added the __new__-based tests, NVIDIA#16523 later added the unguarded attribute access; neither PR tested the combination. Seed _multi_frontend_ipc_dir / _multi_frontend_hmac in the shared _bare_proxy() helper so the teardown path is a clean no-op, mirroring how NVIDIA#16630 (nvbugs/6480574) handled the earlier n_workers/_wait_shutdown variant. Also re-align with main: restore the n_workers/_wait_shutdown seeding in test_pool_session_shutdown_never_blocks_after_release and drop its now-removed waive (main dropped it in NVIDIA#16630). Whole file: 23 passed (was 3 failed + 1 waived/broken). Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

…roxy_fast_death __new__ seeding Two base-branch (main) regressions surfaced by rebasing onto latest main; neither is in the KVCacheManagerV2 change set. 1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__ forbids subclasses from overriding forward() -- they must implement _forward_impl so the base forward() wrapper can guarantee spec-dec attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(), so 'import tensorrt_llm' raised TypeError at class-definition time via the normal _torch.models -> speculative -> mtp_dynamic_tree import chain, breaking all test collection / package sanity. Rename its forward() to _forward_impl(), matching every sibling worker (MTPEagleWorker, DSparkWorker, eagle3, eagle3_dynamic_tree); callers use the base forward() wrapper so no caller changes. Verified: 'import tensorrt_llm' succeeds again. 2) test_proxy_fast_death cross-PR merge-skew: NVIDIA#16312 added __new__-built proxy tests; NVIDIA#16523 later made GenerationExecutorProxy.shutdown() -> _cleanup_multi_frontend_ipc_dir() read self._multi_frontend_ipc_dir unguarded, so the __new__ proxies raised AttributeError at teardown. Seed _multi_frontend_ipc_dir/_multi_frontend_hmac in the shared _bare_proxy() helper so GC-time teardown is a clean no-op, mirroring NVIDIA#16630 (nvbugs/6480574) for the earlier n_workers variant. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

…ew__ seeding Base-branch (main) regressions surfaced by rebasing onto latest main; none are in the KVCacheManagerV2 change set. Two distinct patterns, both fixed by matching the existing sibling conventions. 1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__ forbids subclasses from overriding forward() -- they must implement _forward_impl so the base forward() wrapper can guarantee spec-dec attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(), so 'import tensorrt_llm' raised TypeError at class-definition time, breaking all test collection. Rename its forward() to _forward_impl(), matching every sibling worker; callers use the base forward() wrapper so no caller changes. 2) executor tests build objects via __new__ (bypassing __init__) then exercise teardown/lifecycle paths. main NVIDIA#16523 (multi-process HTTP frontends) added attributes to BaseWorker/GenerationExecutorProxy __init__ that teardown reads unguarded, so the __new__-built test objects raised AttributeError: - test_proxy_fast_death.py _bare_proxy: seed _multi_frontend_ipc_dir/_hmac. - test_event_loop_error_broadcast.py _WorkerStub: seed frontend_result_queues (responses_handler read it unguarded -> 4/6 tests failed). - test_proxy_postproc_terminate.py _make_proxy: seed workers_started + _multi_frontend_ipc_dir/_hmac (GC __del__ -> shutdown teardown). - test_base_worker.py __new__ shell: seed doing_shutdown (GC __del__). Each seeds only what that object's teardown reads, matching the _bare_proxy convention. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

…new__ proxy mock (drop after upstream fix) NVIDIA#16523 added _multi_frontend_ipc_dir (set in __init__) and shutdown() now always calls _cleanup_multi_frontend_ipc_dir(), including the workers_started=False early path. test_proxy_fast_death.py builds proxies via GenerationExecutorProxy.__new__ without __init__, so every shutdown() (and the __del__ GC path) raises AttributeError on main. Seed the attr in _bare_proxy(), same pattern as NVIDIA#16630. Temporary carry so this PR's CI can go green; becomes a no-op once the equivalent fix lands on main. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: longcheng-nv <243710427+longcheng-nv@users.noreply.github.com>

longcheng-nv · 2026-07-22T22:25:31Z

Heads-up: this PR appears to have introduced a semantic conflict with tests/unittest/executor/test_proxy_fast_death.py that is failing on current main (unwaived).

This PR added self._multi_frontend_ipc_dir (assigned in __init__, proxy.py:174) and shutdown() now always calls _cleanup_multi_frontend_ipc_dir() — including the workers_started=False early-return path.
test_proxy_fast_death.py constructs proxies via GenerationExecutorProxy.__new__ without running __init__ (_bare_proxy()), so every proxy.shutdown() — and the __del__ GC path of every such mock — raises:

AttributeError: 'GenerationExecutorProxy' object has no attribute '_multi_frontend_ipc_dir'
proxy.py:797 shutdown -> proxy.py:739 _cleanup_multi_frontend_ipc_dir

Observed failing in PR CI runs L0 #49263 and #49333 (A10 stage, 3 tests, persists through rerun; reproduced locally on main tip).
Suggested fix (same pattern as [https://nvbugs/6480574][fix] seed attrs bypassed by __new__ in pool session shutdown test #16630, which seeded other bypassed attrs in this file): add proxy._multi_frontend_ipc_dir = None to _bare_proxy(); alternatively make _cleanup_multi_frontend_ipc_dir use getattr(self, '_multi_frontend_ipc_dir', None) for robustness.

I am temporarily carrying the one-line test fix on PR #16457 to unblock its CI; happy to drop it once a proper fix lands.

Base-branch (main) regressions surfaced by rebasing onto latest main; none are in the KVCacheManagerV2 change set. Two distinct patterns, both fixed by matching the existing sibling conventions. 1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__ forbids subclasses from overriding forward() -- they must implement _forward_impl so the base forward() wrapper can guarantee spec-dec attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(), so 'import tensorrt_llm' raised TypeError at class-definition time, breaking all test collection. Rename its forward() to _forward_impl(), matching every sibling worker; callers use the base forward() wrapper so no caller changes. 2) executor tests build objects via __new__ (bypassing __init__) then exercise teardown/lifecycle paths. main NVIDIA#16523 (multi-process HTTP frontends) added attributes to BaseWorker/GenerationExecutorProxy __init__ that teardown reads unguarded, so the __new__-built test objects raised AttributeError: - test_proxy_fast_death.py _bare_proxy: seed _multi_frontend_ipc_dir/_hmac. - test_event_loop_error_broadcast.py _WorkerStub: seed frontend_result_queues (responses_handler read it unguarded -> 4/6 tests failed). - test_proxy_postproc_terminate.py _make_proxy: seed workers_started + _multi_frontend_ipc_dir/_hmac (GC __del__ -> shutdown teardown). - test_base_worker.py __new__ shell: seed doing_shutdown (GC __del__). Each seeds only what that object's teardown reads, matching the _bare_proxy convention. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

…ic IPC executor path (NVIDIA#16523) Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Base-branch (main) regressions surfaced by rebasing onto latest main; none are in the KVCacheManagerV2 change set. Two distinct patterns, both fixed by matching the existing sibling conventions. 1) tensorrt_llm import break (nvbugs/6442074): SpecWorkerBase.__init_subclass__ forbids subclasses from overriding forward() -- they must implement _forward_impl so the base forward() wrapper can guarantee spec-dec attn-metadata cleanup. MTPEagleDynamicTreeWorker still overrode forward(), so 'import tensorrt_llm' raised TypeError at class-definition time, breaking all test collection. Rename its forward() to _forward_impl(), matching every sibling worker; callers use the base forward() wrapper so no caller changes. 2) executor tests build objects via __new__ (bypassing __init__) then exercise teardown/lifecycle paths. main NVIDIA#16523 (multi-process HTTP frontends) added attributes to BaseWorker/GenerationExecutorProxy __init__ that teardown reads unguarded, so the __new__-built test objects raised AttributeError: - test_proxy_fast_death.py _bare_proxy: seed _multi_frontend_ipc_dir/_hmac. - test_event_loop_error_broadcast.py _WorkerStub: seed frontend_result_queues (responses_handler read it unguarded -> 4/6 tests failed). - test_proxy_postproc_terminate.py _make_proxy: seed workers_started + _multi_frontend_ipc_dir/_hmac (GC __del__ -> shutdown teardown). - test_base_worker.py __new__ shell: seed doing_shutdown (GC __del__). Each seeds only what that object's teardown reads, matching the _bare_proxy convention. Signed-off-by: Yao Yao <lowsfer@users.noreply.github.com>

github-actions Bot assigned lancelly Jul 17, 2026

lancelly added 8 commits July 16, 2026 20:56

lancelly added the api-compatible Accepted LLM API contract change that is backwards-compatible label Jul 17, 2026

[None][chore] apply pre-commit yapf formatting

757660a

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

reasonsolo reviewed Jul 17, 2026

View reviewed changes

Comment thread tensorrt_llm/commands/serve.py Outdated

Comment thread tensorrt_llm/commands/serve.py

Comment thread tensorrt_llm/executor/postproc_worker.py Outdated

Comment thread tensorrt_llm/executor/utils.py Outdated

lancelly added 5 commits July 17, 2026 02:03

reasonsolo approved these changes Jul 17, 2026

View reviewed changes

lancelly marked this pull request as ready for review July 17, 2026 09:39

lancelly requested review from a team as code owners July 17, 2026 09:39

lancelly requested review from QiJune and YihuiLu512 July 17, 2026 09:39

[None][chore] apply pre-commit isort/yapf formatting

d6a2eb6

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

[None][chore] fix D205 docstring layout in _init_multi_frontend_mode

eb3e425

Signed-off-by: Lance Liao <108499334+lancelly@users.noreply.github.com>

Merge branch 'main' into feat/serve-multi-frontend-main

cdbc7d6

lancelly force-pushed the feat/serve-multi-frontend-main branch from 7d5e91f to f012263 Compare July 21, 2026 12:20

longlee0622 enabled auto-merge (squash) July 22, 2026 01:27

longlee0622 merged commit 09e9738 into NVIDIA:main Jul 22, 2026
8 checks passed

longcheng-nv mentioned this pull request Jul 22, 2026

[None][perf] GVR top-K decode: enable R0 histogram-ladder admission by default #16457

Open

Uh oh!

Conversation

lancelly commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Design

Scope / limitations (prototype)

Summary by CodeRabbit

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coderabbitai Bot commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

lancelly commented Jul 17, 2026

Uh oh!

lancelly commented Jul 17, 2026

Uh oh!

lancelly commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

lancelly commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 21, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

lancelly commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

tensorrt-cicd commented Jul 22, 2026

Uh oh!

Uh oh!

longcheng-nv commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

lancelly commented Jul 17, 2026 •

edited

Loading

coderabbitai Bot commented Jul 17, 2026 •

edited

Loading