[None][perf] Skip transceiver tp_allgather when no sessions ever opened by Shixiaowei02 · Pull Request #14042 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-05-12T08:20:38Z

Description

V4-Pro 8k/1k MTP3 disagg (ctx1 TP=4 + gen4 TP=8), concurrency=4/server (total 16), 128 prompts × 8 rounds, gb300:

Metric	Before	After	Δ
output_throughput (tok/s)	600.1	626.5	+4.41%
user_throughput (tok/s)	45.87	47.38	+3.28%
mean_tpot_ms	6.373	6.126	−3.87%
mean_ttft_ms	16491	16300	−1.16%
mean_itl_ms	1434.9	1416.4	−1.29%
mean_e2el_ms	22354	21932	−1.89%
avg_decoded_tokens_per_iter	2.654	2.676	+0.83%
duration (s)	196.2	188.0	−4.22%

GPU compute is unchanged . The full 4.4% e2e throughput win comes from eliminating one tp_allgather per decode iter on the GEN worker, which had previously synced all 8 TP ranks every iter against an empty consensus list.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-05-12T13:31:06Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-12T13:39:31Z

PR_Github #47973 [ run ] triggered by Bot. Commit: 70162aa Link to invocation

tensorrt-cicd · 2026-05-12T15:57:48Z

PR_Github #47973 [ run ] completed with state SUCCESS. Commit: 70162aa
/LLM/main/L0_MergeRequest_PR pipeline #37813 completed with status: 'SUCCESS'

CI Report

Link to invocation

…ed (#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

…nsensus deadlock The sticky-role short-circuits in KvCacheTransceiverV2.check_context_transfer_status and check_gen_transfer_status (sticky markers from NVIDIA#14042) returned early when a transceiver had never opened a send/recv session, to skip the per-iter consensus allgather on pure-role transceivers. Under pipeline parallelism (and ADP request sharding) the per-rank _ever_had_send_session / _ever_had_recv_session markers flip asymmetrically across PP stages, so some ranks short-circuited and skipped the _ctx_consensus pp_allgather barrier while peers entered it, deadlocking the collective. This was observed as a hang in tests/unittest/disaggregated/test_cache_transceiver_single_process.py ::test_cache_transceiver[*-tp4_pp2_dp_both] (8-rank ADP+PP2). Guard the short-circuit with `not self._ctx_need_pp_sync` (pp_size == 1) so it only fires when there is no cross-stage consensus barrier: preserves the pure-role tp_allgather-skip optimization for PP=1 while doing the full symmetric consensus whenever pipeline parallelism is enabled. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

…nsensus deadlock The sticky-role short-circuits in KvCacheTransceiverV2.check_context_transfer_status and check_gen_transfer_status (sticky markers from NVIDIA#14042) returned early when a transceiver had never opened a send/recv session, to skip the per-iter consensus allgather on pure-role transceivers. Under pipeline parallelism (and ADP request sharding) the per-rank _ever_had_send_session / _ever_had_recv_session markers flip asymmetrically across PP stages, so some ranks short-circuited and skipped the _ctx_consensus pp_allgather barrier while peers entered it, deadlocking the collective. This was observed as a hang in tests/unittest/disaggregated/test_cache_transceiver_single_process.py ::test_cache_transceiver[*-tp4_pp2_dp_both] (8-rank ADP+PP2). Guard the short-circuit with `not self._ctx_need_pp_sync` (pp_size == 1) so it only fires when there is no cross-stage consensus barrier: preserves the pure-role tp_allgather-skip optimization for PP=1 while doing the full symmetric consensus whenever pipeline parallelism is enabled. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…and_schedule_batch The cherry-pick of NVIDIA#14042-era "Fix deepseekv4 stall" replaced main's immediate "Insufficient KV cache for gen-only benchmark mode" guard with a time-based gen-count stall watchdog. On current main that watchdog is both superseded and incompatible: main already handles the ADP fill-completion case via _is_benchmark_disagg_fill_complete (per-rank allgather), and the time-based watchdog never fires on a single scheduling iteration, so it regressed tests/unittest/_torch/executor/test_benchmark_disagg.py (TestFailFastDuringBenchmarkFill, TestFillPhaseEndToEnd) which assert an immediate fail-fast when all benchmark requests are fetched (or the fill phase is over) and the scheduler can fit no INIT request. Restore main's immediate guard (fail once all requests are fetched and no INIT request fits, suppressed during warmup) and drop the now-unused benchmark_fill_stall_timeout_s. Verified on 8xB200: test_benchmark_disagg.py, test_py_executor.py and the tp4_pp2_dp_both transceiver cases all pass (240 passed, 0 failed). Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

…nsensus deadlock The sticky-role short-circuits in KvCacheTransceiverV2.check_context_transfer_status and check_gen_transfer_status (sticky markers from NVIDIA#14042) returned early when a transceiver had never opened a send/recv session, to skip the per-iter consensus allgather on pure-role transceivers. Under pipeline parallelism (and ADP request sharding) the per-rank _ever_had_send_session / _ever_had_recv_session markers flip asymmetrically across PP stages, so some ranks short-circuited and skipped the _ctx_consensus pp_allgather barrier while peers entered it, deadlocking the collective. This was observed as a hang in tests/unittest/disaggregated/test_cache_transceiver_single_process.py ::test_cache_transceiver[*-tp4_pp2_dp_both] (8-rank ADP+PP2). Guard the short-circuit with `not self._ctx_need_pp_sync` (pp_size == 1) so it only fires when there is no cross-stage consensus barrier: preserves the pure-role tp_allgather-skip optimization for PP=1 while doing the full symmetric consensus whenever pipeline parallelism is enabled. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…and_schedule_batch The cherry-pick of NVIDIA#14042-era "Fix deepseekv4 stall" replaced main's immediate "Insufficient KV cache for gen-only benchmark mode" guard with a time-based gen-count stall watchdog. On current main that watchdog is both superseded and incompatible: main already handles the ADP fill-completion case via _is_benchmark_disagg_fill_complete (per-rank allgather), and the time-based watchdog never fires on a single scheduling iteration, so it regressed tests/unittest/_torch/executor/test_benchmark_disagg.py (TestFailFastDuringBenchmarkFill, TestFillPhaseEndToEnd) which assert an immediate fail-fast when all benchmark requests are fetched (or the fill phase is over) and the scheduler can fit no INIT request. Restore main's immediate guard (fail once all requests are fetched and no INIT request fits, suppressed during warmup) and drop the now-unused benchmark_fill_stall_timeout_s. Verified on 8xB200: test_benchmark_disagg.py, test_py_executor.py and the tp4_pp2_dp_both transceiver cases all pass (240 passed, 0 failed). Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

…nsensus deadlock The sticky-role short-circuits in KvCacheTransceiverV2.check_context_transfer_status and check_gen_transfer_status (sticky markers from NVIDIA#14042) returned early when a transceiver had never opened a send/recv session, to skip the per-iter consensus allgather on pure-role transceivers. Under pipeline parallelism (and ADP request sharding) the per-rank _ever_had_send_session / _ever_had_recv_session markers flip asymmetrically across PP stages, so some ranks short-circuited and skipped the _ctx_consensus pp_allgather barrier while peers entered it, deadlocking the collective. This was observed as a hang in tests/unittest/disaggregated/test_cache_transceiver_single_process.py ::test_cache_transceiver[*-tp4_pp2_dp_both] (8-rank ADP+PP2). Guard the short-circuit with `not self._ctx_need_pp_sync` (pp_size == 1) so it only fires when there is no cross-stage consensus barrier: preserves the pure-role tp_allgather-skip optimization for PP=1 while doing the full symmetric consensus whenever pipeline parallelism is enabled. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…and_schedule_batch The cherry-pick of NVIDIA#14042-era "Fix deepseekv4 stall" replaced main's immediate "Insufficient KV cache for gen-only benchmark mode" guard with a time-based gen-count stall watchdog. On current main that watchdog is both superseded and incompatible: main already handles the ADP fill-completion case via _is_benchmark_disagg_fill_complete (per-rank allgather), and the time-based watchdog never fires on a single scheduling iteration, so it regressed tests/unittest/_torch/executor/test_benchmark_disagg.py (TestFailFastDuringBenchmarkFill, TestFillPhaseEndToEnd) which assert an immediate fail-fast when all benchmark requests are fetched (or the fill phase is over) and the scheduler can fit no INIT request. Restore main's immediate guard (fail once all requests are fetched and no INIT request fits, suppressed during warmup) and drop the now-unused benchmark_fill_stall_timeout_s. Verified on 8xB200: test_benchmark_disagg.py, test_py_executor.py and the tp4_pp2_dp_both transceiver cases all pass (240 passed, 0 failed). Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

github-actions Bot assigned Shixiaowei02 May 12, 2026

Shixiaowei02 force-pushed the user/xiaoweis/skip-allgather branch from 64791f5 to 684e049 Compare May 12, 2026 08:21

Shixiaowei02 marked this pull request as ready for review May 12, 2026 08:21

Shixiaowei02 requested a review from a team as a code owner May 12, 2026 08:21

Shixiaowei02 requested review from chuangz0, liji-nv and suyoggupta and removed request for a team May 12, 2026 08:21

Shixiaowei02 added the deepseek-v4 label May 12, 2026

Shixiaowei02 requested a review from qiaoxj07 May 12, 2026 08:22

liji-nv approved these changes May 12, 2026

View reviewed changes

qiaoxj07 approved these changes May 12, 2026

View reviewed changes

chuangz0 approved these changes May 12, 2026

View reviewed changes

Skip transceiver tp_allgather when no sessions ever opened

70162aa

Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 force-pushed the user/xiaoweis/skip-allgather branch from 684e049 to 70162aa Compare May 12, 2026 13:30

Shixiaowei02 merged commit b5c1cb7 into NVIDIA:feat/deepseek_v4 May 13, 2026
5 checks passed

Shixiaowei02 deleted the user/xiaoweis/skip-allgather branch May 13, 2026 00:47

lfr-0531 pushed a commit that referenced this pull request May 14, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

c5226ec

…ed (#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

qiaoxj07 mentioned this pull request May 15, 2026

[None][fix] Return 3-tuple from check_gen_transfer_status fast path #14162

Merged

2 tasks

lingjiew mentioned this pull request May 17, 2026

[None][fix] Return 3-tuple in check_gen_transfer_status early-return #14214

Closed

1 task

lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 10, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

a2c3404

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 10, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

9a40e93

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 10, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

867f79e

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 10, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

b587e5d

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 11, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

4c61c1d

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 11, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

0b58b72

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 11, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

d36fd7a

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 11, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

940b849

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 11, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

3429400

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 12, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

a2562f5

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

lfr-0531 pushed a commit to lfr-0531/TensorRT-LLM that referenced this pull request Jun 12, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

f5030aa

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 12, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

7a7113d

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 14, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

f554f79

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 16, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

14369cf

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 17, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

5e69ff2

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 21, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

31385e0

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 22, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

699e1fa

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 23, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

d76abb7

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 23, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

1c473ca

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 23, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

db8f595

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 24, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

d8a3a52

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 24, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

f0a30fa

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Shixiaowei02 added a commit to Shixiaowei02/TensorRT-LLM that referenced this pull request Jun 24, 2026

[None][perf] Skip transceiver tp_allgather when no sessions ever open…

7331953

…ed (NVIDIA#14042) Signed-off-by: yuhangh <58161490+heyuhhh@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][perf] Skip transceiver tp_allgather when no sessions ever opened - #14042

[None][perf] Skip transceiver tp_allgather when no sessions ever opened#14042
Shixiaowei02 merged 1 commit into
NVIDIA:feat/deepseek_v4from
Shixiaowei02:user/xiaoweis/skip-allgather

Shixiaowei02 commented May 12, 2026 •

edited

Loading

Uh oh!

Shixiaowei02 commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

Shixiaowei02 commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

Shixiaowei02 commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Shixiaowei02 commented May 12, 2026 •

edited

Loading