[https://nvbugs/6487038][fix] Stop single-rank disagg errors from crashing all gen ranks by Shixiaowei02 · Pull Request #16834 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-07-24T08:51:29Z

Description

This pull request introduces several improvements and fixes related to disaggregated attention data parallel (ADP) execution and configuration for NIXL cache transceivers. The main changes include making the ADP executor more robust to transient request overshoots, improving thread management and error surfacing in the worker engine, and updating test and configuration files to support new runtime and tuning options.

ADP executor robustness and error handling:

Relaxed a hard assertion in _pad_attention_dp_dummy_request to a warning when active_requests temporarily exceeds expected_num_active_requests, preventing crashes during transient disagg KV-transfer errors and allowing the system to self-correct.
Added a new test test_pad_dummy_tolerates_active_request_overshoot to ensure the executor warns and continues (instead of raising an error) when overshoots occur.
Improved start_thread in the worker engine to avoid masking event loop crashes by checking thread state and surfacing the real error if a restart is attempted after a failure.

Configuration and test tuning for NIXL cache transceiver:

Updated several YAML test configs to set transceiver_runtime: PYTHON and added or increased kv_cache_bounce_size_mb for better memory management and to enable Python runtime for the NIXL backend. [1] [2] [3] [4] [5] [6]
Set the environment variable TRTLLM_KV_TRANSFER_NUM_THREADS=4 in worker environments to control the number of threads used for KV transfer, improving parallelism and performance. [1] [2]

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-07-24T08:57:31Z

Walkthrough

Changes

Transfer admission behavior

Layer / File(s)	Summary
Async transfer admission bypass and coverage `tensorrt_llm/_torch/pyexecutor/py_executor.py`, `tests/unittest/_torch/executor/test_py_executor.py`	Async generation transfers bypass synchronous budget admission. Existing budgeting tests force synchronous overlap behavior, while async-mode coverage verifies candidates remain admitted without rollback.

Executor runtime safeguards

Layer / File(s)	Summary
Active-request padding tolerance `tensorrt_llm/_torch/pyexecutor/py_executor.py`, `tests/unittest/_torch/executor/test_py_executor.py`	Transient active-request overshoot logs a warning instead of raising an assertion, with regression coverage.
Response-thread lifecycle guards `tensorrt_llm/executor/worker.py`	Response-thread startup avoids invalid restarts and re-raises captured event-loop errors.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: lancelly, chienchunhung

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the primary fix for disaggregated execution errors and follows the required NVBugs and type format.
Description check	✅ Passed	The description explains the main changes and references relevant tests, but the dedicated Test Coverage section is left empty.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

Shixiaowei02 · 2026-07-27T05:13:13Z

/bot run --post-merge --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge"

tensorrt-cicd · 2026-07-27T05:19:39Z

PR_Github #61811 [ run ] triggered by Bot. Commit: d290731 Link to invocation

tensorrt-cicd · 2026-07-27T05:19:40Z

PR_Github #61811 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

Shixiaowei02 · 2026-07-27T05:58:37Z

/bot run --post-merge --stage-list "GB200-16_GPUs-4_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE2-GPU8-GEN1-NODE2-GPU8-Post-Merge, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge"

tensorrt-cicd · 2026-07-27T06:04:04Z

PR_Github #61826 [ run ] triggered by Bot. Commit: 9727efc Link to invocation

tensorrt-cicd · 2026-07-27T06:04:05Z

PR_Github #61826 Bot args parsing error: CI requested by --post-merge or an approval-gated stage selector (*, *Post-Merge*, or *PerfSanity*) requires the ci: post-merge approved PR label. Ask a member of NVIDIA/trt-llm-ci-approvers to apply it.

Link to invocation

github-actions · 2026-07-27T06:19:15Z

Removed the "ci: post-merge approved" label because @niukuo could not be verified as an active member of NVIDIA/trt-llm-ci-approvers. Ask a member of that team to apply it.

Shixiaowei02 · 2026-07-28T14:56:19Z

/bot run --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-07-28T15:01:52Z

PR_Github #62209 [ run ] triggered by Bot. Commit: cf19cfd Link to invocation

tensorrt-cicd · 2026-07-28T15:06:06Z

PR_Github #62205 [ run ] completed with state ABORTED. Commit: e076453

Link to invocation

chienchunhung

Thanks for the PR!

tensorrt-cicd · 2026-07-29T06:52:37Z

PR_Github #62209 [ run ] completed with state FAILURE. Commit: cf19cfd
/LLM/main/L0_MergeRequest_PR pipeline #50379 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-07-29T08:31:59Z

/bot run --disable-fail-fast --only-multi-gpu-test

tensorrt-cicd · 2026-07-29T08:44:17Z

PR_Github #62476 [ run ] triggered by Bot. Commit: cf19cfd Link to invocation

tensorrt-cicd · 2026-07-29T11:27:57Z

PR_Github #62476 [ run ] completed with state SUCCESS. Commit: cf19cfd
/LLM/main/L0_MergeRequest_PR pipeline #50627 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Shixiaowei02 · 2026-08-03T06:02:46Z

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-08-03T06:09:13Z

PR_Github #63367 [ run ] triggered by Bot. Commit: ad17083 Link to invocation

tensorrt-cicd · 2026-08-03T10:36:21Z

PR_Github #63367 [ run ] completed with state SUCCESS. Commit: ad17083
/LLM/main/L0_MergeRequest_PR pipeline #51352 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…rt cascade start_thread() restarted the already-finished await-response thread, raising "threads can only be started once" and masking the real cause: the engine event loop crashed (AwaitResponseHelper broadcasts _event_loop_error and returns, so the thread exits). The misleading RuntimeError then cascaded into a peer MPI-collective hang across ranks. Detect the finished thread and re-raise the stashed engine event-loop error instead. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…st overshoot _pad_attention_dp_dummy_request asserted expected_num_active_requests >= len(active_requests). When disagg KV-transfer-error requests linger in active_requests for a tick before cleanup drains them, that assert fires and takes down the generation loop on every attention-DP rank at once, turning a self-correcting transient into a hard failure of the whole gen server. Warn once and continue instead. The padding decision below keys on the schedulable count, so the overshoot does not affect the result. Note: an earlier revision of this commit also bypassed the DisaggTransferAdmissionController budget on the async transfer path. That part was dropped: an isolated A/B on gb300 deepseek-r1 8k1k con4096 e2e (async path, KV-cache bounce enabled on both arms, wheels differing only in that early return) showed no benefit -- 20480/20480 both, 11587 vs 11661 tok/s output and 347.1 s vs 344.5 s median TTFT, i.e. within noise and marginally favouring the unmodified code. With bounce enabled a transfer completes in ~3 ms, so the budget drains immediately and never becomes the bottleneck. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…es to transceiver V2 (PYTHON) The V2 migration (NVIDIA#16669) left these two gb300 cases on the C++ transceiver because they regressed on the Python one. The crash-cascade and admission-throttle fixes in this PR unblock them: - con4096 (8k1k, con=4096): enable PYTHON + KV-cache bounce (2048 MiB) + sender concurrency. Bounce coalesces the scattered per-fragment WRITE that otherwise overflows the transfer queue at this concurrency (652/4096 -> 20480/20480). - con256 (128k8k, con=256): enable PYTHON only. The 128k transfer is bandwidth-bound and measured on par with the C++ transceiver (v1 ~= v2), so no bounce is needed and there is no regression. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

….5 con4096 disagg perf case The gen_only case timed out on the CTX side: 2939 "KV cache transfer timeout: elapsed ~60244ms > kv_transfer_timeout_ms=60000ms" lines across CTX ranks 0-3 and 4096/4096 failed requests. Root cause is the data plane, not the timeout budget. With bounce disabled, each 8k request submits its KV as ~15.6k scattered per-block fragments in a single NIXL transfer request, which UCX cannot stripe across rails, so transfers queue behind one another until the 60s budget expires. Coalescing the request's blocks into one contiguous fabric-VMM buffer restores a single multi-rail write. Reproduced and verified locally on GB300 (20 GPUs / 5 nodes, gen_only), both arms on the same commit, wheel and container: as-is : FAILED - 0/4096 successful, 2879 CTX transfer timeouts + bounce : PASSED - 4096/4096 successful, 0 timeouts, 11580 tok/s output per-transfer latency p50 2.16 ms / p90 2.99 ms "[kv-bounce] coalesced 256 blocks / 274MiB into one region across 1 writer(s)" Bounce engages on the single-writer path here because attention DP is enabled on both sides, so each request has exactly one peer rank (expected_transfers == 1). This mirrors the settings NVIDIA#15966 already applied to the other Kimi disagg cases. The timeout budget is left at the 60s default on purpose: coalesced transfers complete in ~2 ms, four orders of magnitude inside it. Un-waives only the gen_only case, which is the one covered by this bug and the one verified above. The e2e variant stays waived pending its own validation. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-08-03T11:24:33Z

/bot run --add-multi-gpu-test --disable-fail-fast --extra-stage "GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-1, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-2, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-3, GB300-20_GPUs-5_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE4-GPU16-Post-Merge-4, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-1, GB300-12_GPUs-3_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE2-GPU8-Post-Merge-2"

tensorrt-cicd · 2026-08-03T11:31:51Z

PR_Github #63438 [ run ] triggered by Bot. Commit: ca0c515 Link to invocation

Shixiaowei02 requested review from a team as code owners July 24, 2026 08:51

Shixiaowei02 requested review from bo-nv, cascade812, chuangz0 and tongyuantongyu July 24, 2026 08:51

github-actions Bot assigned Shixiaowei02 Jul 24, 2026

Shixiaowei02 requested review from asfiyab-nvidia, chienchunhung and nv-xtf July 24, 2026 08:53

Shixiaowei02 marked this pull request as draft July 24, 2026 08:57

Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch 4 times, most recently from adad095 to d290731 Compare July 27, 2026 05:13

Shixiaowei02 requested a review from litaotju July 27, 2026 05:49

litaotju approved these changes Jul 27, 2026

View reviewed changes

Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch from 05f16c5 to 9727efc Compare July 27, 2026 05:58

niukuo added the ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests label Jul 27, 2026

github-actions Bot removed the ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests label Jul 27, 2026

litaotju added the ci: post-merge approved Approved by TRT-LLM CI approvers for broad post-merge CI requests label Jul 27, 2026

Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch from 9727efc to e14fc1b Compare July 27, 2026 08:37

Shixiaowei02 requested a review from a team as a code owner July 28, 2026 14:52

Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch 2 times, most recently from cf36915 to cf19cfd Compare July 28, 2026 14:54

Shixiaowei02 requested a review from chenfeiz0326 July 28, 2026 14:59

chienchunhung self-requested a review July 28, 2026 17:55

chienchunhung approved these changes Jul 28, 2026

View reviewed changes

asfiyab-nvidia mentioned this pull request Jul 28, 2026

[TRTLLM-13948][feat] Set DeepSeekV3 to use Python KV-cache transceiver V2 by default #16908

Merged

1 task

chenfeiz0326 approved these changes Jul 29, 2026

View reviewed changes

nv-xtf approved these changes Jul 29, 2026

View reviewed changes

LarryXFly approved these changes Jul 30, 2026

View reviewed changes

Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch 2 times, most recently from 8a400e1 to ad17083 Compare August 3, 2026 05:54

Shixiaowei02 changed the title ~~[None][fix] Fix disagg KV-transfer crash cascades and enable KV-cache bounce~~ [https://nvbugs/6487038][fix] Stop single-rank disagg errors from crashing all gen ranks Aug 3, 2026

Shixiaowei02 added 4 commits August 3, 2026 04:20

Shixiaowei02 force-pushed the user/xiaoweis/fix-disagg-async-transfer-throttle branch from ad17083 to ca0c515 Compare August 3, 2026 11:23

Uh oh!

Conversation

Shixiaowei02 commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

Shixiaowei02 commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

Shixiaowei02 commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

tensorrt-cicd commented Jul 27, 2026

Uh oh!

github-actions Bot commented Jul 27, 2026

Uh oh!

Shixiaowei02 commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

tensorrt-cicd commented Jul 28, 2026

Uh oh!

chienchunhung left a comment

Choose a reason for hiding this comment

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

Shixiaowei02 commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

tensorrt-cicd commented Jul 29, 2026

Uh oh!

Shixiaowei02 commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

Shixiaowei02 commented Aug 3, 2026

Uh oh!

tensorrt-cicd commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Shixiaowei02 commented Jul 24, 2026 •

edited

Loading

coderabbitai Bot commented Jul 24, 2026 •

edited

Loading