[None][perf] PyExecutor: skip empty MPI collectives in _fetch_new_requests (+ probe hang-detector fix) by hyukn · Pull Request #14912 · NVIDIA/TensorRT-LLM

hyukn · 2026-06-03T15:41:47Z

Summary

Three changes to PyExecutor._fetch_new_requests that cut empty-iter MPI collective overhead on the host critical path, plus a hang-detector fix for the new probe.

Commits:

[perf] skip empty gather_prefix_matches (F1) — guard the KV-cache-aware prefix-match allgather on new_requests being non-empty.
[perf] probe before heavy request broadcast (F2) — broadcast a 1-int probe of rank-0's count before the pickle-heavy request_broadcaster.broadcast() so all ranks symmetrically skip when there is nothing to send.
[fix] pause hang detector around request-count probe — wrap F2's probe broadcast in hang_detector.pause() (see "Hang-detector fix" below).

Both skips are rank-symmetric (they branch on data already identical across ranks after the prior _pop_from_waiting_queue / probe), so they cannot deadlock.

Measurements

DSv4-Pro, CTX-only, agentperf-v2, DEP=4 + attention-DP + KV-cache-aware routing, c=128, 1 node x 4 GB200.

Variant	tot tok/s	delta vs baseline
baseline	811,856	-
F1+F2	854,886	+5.3%
F1+F2+probe-pause (this PR)	847,621 (mean of 3)	+4.4%
F1+F2 + PR #14889	920,670	+13.4%

The probe-pause commit only touches the idle watchdog path, so steady-state throughput is unchanged from F1+F2 (the +4.4% vs +5.3% is run-to-run variance, ±2-3%).

Per-NVTX-range (100 captured iters, rank 0):

range	baseline	F1+F2
`_fetch_new_requests` (parent)	16,384 ms	13,956 ms (-14.8%)
`broadcast_payload` (heavy pickle)	5,821 ms	4,478 ms
`adp_gather_prefix_matches`	6,887 ms	6,267 ms

Hang-detector fix (commit 3) — important for disagg

F2's probe self.dist.broadcast(_new_request_count, root=0) was NOT wrapped in hang_detector.pause(). When the server is idle, rank 0 blocks in get_from_request_queue(timeout=None) (which IS pause-wrapped) waiting for the next request, while the non-root ranks skip the fetch and block on the probe broadcast waiting for rank 0. With the probe un-paused, the non-root ranks' 300s hang-detector watchdog falsely trips at the warmup->serving boundary and shuts the server down.

Pre-PR, that same non-root idle wait happened inside RequestBroadcaster.broadcast -> _broadcast_requests, whose non-root path is wrapped in hang_detector.pause(). F2 moved the wait out of a paused region; commit 3 restores it.

Reproduced on a DSv4-Pro disagg GEN server (DEP8). A/B on the same cluster, same workload, identical except the one-line wrap:

	F2 only	F2 + pause-fix
GEN watchdog hang	ranks 1-7 hang on `comm.Bcast` at the probe; server self-terminates	no hang

Why F1 / F2 are safe

After _pop_from_waiting_queue, every rank holds an identical new_requests list (deterministic pop from a globally-broadcast waiting_queue), so len(new_requests) == 0 (F1) and the probe count (F2) are rank-symmetric: all ranks either skip together or execute together. On F2's skip path py_request_objects is None; downstream attach_py_objects_to_requests is already guarded by if py_request_objects. F2's probe uses the same self.dist.broadcast(int, root=0) idiom as the pre-existing _sync_and_process_resource_governor_queue.

Test plan

CTX-only DSv4-Pro c128: +4.4% mean over 3 runs, no regression vs F1+F2; cache hit rate unchanged.
nsys event count confirms F1+F2 fire on the empty-iter windows.
disagg GEN A/B: pause-fix eliminates the warmup->serving watchdog hang.
CI: PyExecutor + ADP router unit tests.

When attention-DP is enabled, the KV-cache-aware router runs an allgather of per-request prefix-match lengths every iteration. The allgather still fires on an empty new_requests list, costing ~67 ms per empty iter from pure MPI collective rendezvous on DSv4 c=128. After _pop_from_waiting_queue, every rank holds an identical new_requests list (deterministic pop from a globally-broadcast waiting_queue), so the skip decision is rank-symmetric and cannot deadlock. Measured: 7% of steady-state iters on DSv4-Pro c128 have empty new_requests; skipping the allgather there saves ~4 ms/iter average. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

request_broadcaster.broadcast() pickles new_requests + py_request_objects and runs two MPI_Bcasts even when rank 0's request queue is empty, costing ~58 ms per empty iter on DSv4 c=128 (~7% of iters in steady state). Add a tiny scalar broadcast first so all ranks symmetrically learn rank 0's len(new_requests) and can skip the heavy pickle path when it's zero. The probe is ~75 us, dwarfed by the saving on the skip path. Symmetric skip across all ranks cannot deadlock. py_request_objects is set to None on the skip path; downstream attach_py_objects_to_requests is already guarded by 'if py_request_objects'. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The probe broadcast added by the previous commit (self.dist.broadcast(_new_request_count, root=0)) is not wrapped in hang_detector.pause(). When the server is idle, rank 0 blocks in get_from_request_queue(timeout=None) waiting for the next request -- that wait IS pause-wrapped -- while the non-root ranks skip the fetch and block on the probe broadcast waiting for rank 0. Because the probe is unpaused, the non-root ranks' 300s hang-detector watchdog falsely trips at the warmup->serving boundary (the first idle gap before requests flow), shutting the server down. Observed on DSv4-Pro disagg GEN (ranks 1-7 all stuck on comm.Bcast at the probe, 0 successful requests). Pre-probe, the equivalent non-root idle wait happened inside RequestBroadcaster.broadcast -> _broadcast_requests, whose non-root path is wrapped in hang_detector.pause(). The probe must do the same. Fix: wrap the probe broadcast in hang_detector.pause(), restoring the pre-change property that idle broadcast-waits do not trip the watchdog. Signed-off-by: Yukun He <23156053+hyukn@users.noreply.github.com> Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

hyukn · 2026-06-04T07:02:31Z

/bot run --disable-fail-fast --add-multi-gpu-test

tensorrt-cicd · 2026-06-04T07:09:23Z

PR_Github #52001 [ run ] triggered by Bot. Commit: 95b147c Link to invocation

tensorrt-cicd · 2026-06-04T10:30:59Z

PR_Github #52001 [ run ] completed with state SUCCESS. Commit: 95b147c
/LLM/main/L0_MergeRequest_PR pipeline #41345 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

hyukn · 2026-06-04T10:56:26Z

/bot run --disable-fail-fast --add-multi-gpu-test

tensorrt-cicd · 2026-06-04T11:02:02Z

PR_Github #52053 [ run ] triggered by Bot. Commit: 95b147c Link to invocation

tensorrt-cicd · 2026-06-04T12:22:34Z

PR_Github #52053 [ run ] completed with state SUCCESS. Commit: 95b147c
/LLM/main/L0_MergeRequest_PR pipeline #41386 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

hyukn · 2026-06-04T13:58:18Z

/bot run --disable-fail-fast --add-multi-gpu-test

tensorrt-cicd · 2026-06-04T14:04:17Z

PR_Github #52078 [ run ] triggered by Bot. Commit: 95b147c Link to invocation

tensorrt-cicd · 2026-06-04T17:53:57Z

PR_Github #52078 [ run ] completed with state ABORTED. Commit: 95b147c

Link to invocation

hyukn · 2026-06-04T18:06:42Z

/bot run --disable-fail-fast --add-multi-gpu-test

tensorrt-cicd · 2026-06-04T18:13:27Z

PR_Github #52133 [ run ] triggered by Bot. Commit: 95b147c Link to invocation

tensorrt-cicd · 2026-06-04T18:48:41Z

PR_Github #52133 [ run ] completed with state FAILURE. Commit: 95b147c
/LLM/main/L0_MergeRequest_PR pipeline #41458 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

peihu-nv · 2026-06-04T19:26:04Z

/bot run --disable-fail-fast --add-multi-gpu-test

tensorrt-cicd · 2026-06-04T19:31:53Z

PR_Github #52149 [ run ] triggered by Bot. Commit: a49a44f Link to invocation

tensorrt-cicd · 2026-06-04T22:32:35Z

PR_Github #52149 [ run ] completed with state SUCCESS. Commit: a49a44f
/LLM/main/L0_MergeRequest_PR pipeline #41471 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

lfr-0531 · 2026-06-04T23:49:35Z

/bot run

tensorrt-cicd · 2026-06-04T23:55:49Z

PR_Github #52185 [ run ] triggered by Bot. Commit: a49a44f Link to invocation

peihu-nv · 2026-06-05T00:09:35Z

CI failures are unrelated node issues. x86 single hit the known DS B200 KV-transfer failure that hitting multiple PRs. Skipping CI and merging.

tensorrt-cicd · 2026-06-05T01:18:59Z

PR_Github #52185 [ run ] completed with state SUCCESS. Commit: a49a44f
/LLM/main/L0_MergeRequest_PR pipeline #41505 completed with status: 'SUCCESS'

CI Report

Link to invocation

…uests (+ probe hang-detector fix) (NVIDIA#14912) Source-Commit: 49f1ce6ce5d6847af7217e3cf5ba3ff9f5e2fdb2 Signed-off-by: Jiagan Cheng <jiaganc@nvidia.com>

hyukn requested a review from a team as a code owner June 3, 2026 15:41

hyukn requested review from achartier and removed request for a team June 3, 2026 15:41

github-actions Bot assigned hyukn Jun 3, 2026

hyukn requested a review from lancelly June 3, 2026 15:43

hyukn changed the title ~~[None][perf] PyExecutor: skip empty MPI collectives in _fetch_new_requests~~ [None][perf] PyExecutor: skip empty MPI collectives in _fetch_new_requests (+ probe hang-detector fix) Jun 4, 2026

hyukn and others added 3 commits June 4, 2026 15:02

hyukn force-pushed the perf/fetch_request_host_opt branch from 5e5e54e to 95b147c Compare June 4, 2026 07:02

hyukn added the deepseek-v4 label Jun 4, 2026

pcastonguay reviewed Jun 4, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/pyexecutor/py_executor.py Outdated

[None][fix] Make request-count probe PP-aware

a49a44f

Signed-off-by: peihengh <259410613+peihu-nv@users.noreply.github.com>

pcastonguay approved these changes Jun 4, 2026

View reviewed changes

peihu-nv merged commit 49f1ce6 into NVIDIA:feat/deepseek_v4 Jun 5, 2026
5 of 6 checks passed

Uh oh!

Conversation

hyukn commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Measurements

Hang-detector fix (commit 3) — important for disagg

Why F1 / F2 are safe

Test plan

Related

Uh oh!

hyukn commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

tensorrt-cicd commented Jun 3, 2026

Uh oh!

hyukn commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

hyukn commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

hyukn commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

hyukn commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

hyukn commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

peihu-nv commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

lfr-0531 commented Jun 4, 2026

Uh oh!

tensorrt-cicd commented Jun 4, 2026

Uh oh!

peihu-nv commented Jun 5, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hyukn commented Jun 3, 2026 •

edited

Loading