[None][fix] Avoid TP collective deadlock in ADP disagg error path by Shixiaowei02 · Pull Request #13900 · NVIDIA/TensorRT-LLM

Shixiaowei02 · 2026-05-08T09:55:40Z

@coderabbitai summary

Description

Stabilizes disaggregated serving when ADP is enabled.

ADP dummy request type tracking: Dummy requests now mirror the most recent real request's CTX/GEN type instead of being hardcoded to GEN, and CTX dummies are padded to max_num_tokens so the MoE all-to-all sees aligned token counts across ranks.
TP collective deadlock fix: Moves disagg KV cache transfer error handling to the top of the executor loop behind a TP allgather, ensuring all ranks enter the error path together so the downstream tp_gather in response enqueue no longer deadlocks. Includes matching transceiver/NIXL plumbing and a disagg+ADP accuracy test.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

Shixiaowei02 · 2026-05-08T09:56:18Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-08T10:03:53Z

PR_Github #47381 [ run ] triggered by Bot. Commit: 60c9d81 Link to invocation

tensorrt-cicd · 2026-05-08T17:31:56Z

PR_Github #47381 [ run ] completed with state SUCCESS. Commit: 60c9d81
/LLM/main/L0_MergeRequest_PR pipeline #37312 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Shixiaowei02 · 2026-05-09T01:30:08Z

/bot run --add-multi-gpu-test --disable-fail-fast

tensorrt-cicd · 2026-05-09T01:36:51Z

PR_Github #47460 [ run ] triggered by Bot. Commit: 57977cd Link to invocation

tensorrt-cicd · 2026-05-09T06:00:33Z

PR_Github #47460 [ run ] completed with state SUCCESS. Commit: 57977cd
/LLM/main/L0_MergeRequest_PR pipeline #37381 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

Squashed from: - fix: dis-agg with ADP (77bce45) - update (0f50a39) - remove the single request type restriction (d4146f0) - fix: avoid TP collective deadlock (287c6f6) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 · 2026-05-09T06:17:01Z

/bot run --add-multi-gpu-test --disable-fail-fast

…3900) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…IDIA#13900) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - _count_schedulable_active_requests: revert the GENERATION_TO_COMPLETE exclusion that NVIDIA#13900 added (upstream has no such exclusion). Under V1 ADP it undercounted -> a spurious ADP dummy was inserted on top of a real request -> the batch exceeded max_batch_size=1 and tripped the mamba dummy-mask assert / "No free slots". Restores upstream's exact method (fixes test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance). - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths; ADP dummy-role behavior; GENERATION_TO_COMPLETE counts as active (adp_balance regression). - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr. - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…IDIA#13900) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - _count_schedulable_active_requests: revert the GENERATION_TO_COMPLETE exclusion that NVIDIA#13900 added (upstream has no such exclusion). Under V1 ADP it undercounted -> a spurious ADP dummy was inserted on top of a real request -> the batch exceeded max_batch_size=1 and tripped the mamba dummy-mask assert / "No free slots". Restores upstream's exact method (fixes test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance). - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths; ADP dummy-role behavior; GENERATION_TO_COMPLETE counts as active (adp_balance regression). - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr. - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - _count_schedulable_active_requests: revert the GENERATION_TO_COMPLETE exclusion that NVIDIA#13900 added (upstream has no such exclusion). Under V1 ADP it undercounted -> a spurious ADP dummy was inserted on top of a real request -> the batch exceeded max_batch_size=1 and tripped the mamba dummy-mask assert / "No free slots". Restores upstream's exact method (fixes test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance). - _handle_responses KV-timeout: gate the tp_allgather on disagg (kv_cache_transceiver is not None), not just enable_attention_dp. py_kv_transfer_timed_out is disagg-only, so in non-disagg ADP this added a spurious per-iteration collective that desynced adp_router's tp_allgather (gather_all_rank_states received a bool -> TypeError). Verified by 4-GPU DeepSeek-V3-Lite adp_balance e2e A/B (buggy: timeout hang; fixed: completes). - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths; ADP dummy-role behavior; GENERATION_TO_COMPLETE counts as active (adp_balance regression); CTX dummy padding for disagg idle ranks (incl. awaiting-KV-transfer-only). - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr; _create_kv_slice TokenRange (stub sets py_beam_width for the NVIDIA#14876 path). - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…IDIA#13900) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - _count_schedulable_active_requests: revert the GENERATION_TO_COMPLETE exclusion that NVIDIA#13900 added (upstream has no such exclusion). Under V1 ADP it undercounted -> a spurious ADP dummy was inserted on top of a real request -> the batch exceeded max_batch_size=1 and tripped the mamba dummy-mask assert / "No free slots". Restores upstream's exact method (fixes test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance). - _handle_responses KV-timeout: gate the tp_allgather on disagg (kv_cache_transceiver is not None), not just enable_attention_dp. py_kv_transfer_timed_out is disagg-only, so in non-disagg ADP this added a spurious per-iteration collective that desynced adp_router's tp_allgather (gather_all_rank_states received a bool -> TypeError). Verified by 4-GPU DeepSeek-V3-Lite adp_balance e2e A/B (buggy: timeout hang; fixed: completes). - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths; ADP dummy-role behavior; GENERATION_TO_COMPLETE counts as active (adp_balance regression); CTX dummy padding for disagg idle ranks (incl. awaiting-KV-transfer-only). - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr; _create_kv_slice TokenRange (stub sets py_beam_width for the NVIDIA#14876 path). - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…IDIA#13900) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - _count_schedulable_active_requests: revert the GENERATION_TO_COMPLETE exclusion that NVIDIA#13900 added (upstream has no such exclusion). Under V1 ADP it undercounted -> a spurious ADP dummy was inserted on top of a real request -> the batch exceeded max_batch_size=1 and tripped the mamba dummy-mask assert / "No free slots". Restores upstream's exact method (fixes test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance). - _handle_responses KV-timeout: gate the tp_allgather on disagg (kv_cache_transceiver is not None), not just enable_attention_dp. py_kv_transfer_timed_out is disagg-only, so in non-disagg ADP this added a spurious per-iteration collective that desynced adp_router's tp_allgather (gather_all_rank_states received a bool -> TypeError). Verified by 4-GPU DeepSeek-V3-Lite adp_balance e2e A/B (buggy: timeout hang; fixed: completes). - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths; ADP dummy-role behavior; GENERATION_TO_COMPLETE counts as active (adp_balance regression); CTX dummy padding for disagg idle ranks (incl. awaiting-KV-transfer-only). - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr; _create_kv_slice TokenRange (stub sets py_beam_width for the NVIDIA#14876 path). - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

…IDIA#13900) Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com> Signed-off-by: Fanrong Li <lfr-0531@users.noreply.github.com>

…regation Squash of the post-cherry-pick work layered on top of the 8 DeepSeek-V4 disaggregation cherry-picks. Fixes: - ADP disagg error path: restore per-request hang signal (_event_loop_error), scan all candidates + prefer CTX role for mixed-batch dummy padding, and keep charge_budget=False on KV-transfer timeouts so they don't exhaust the global error budget and shut down the executor. - _count_schedulable_active_requests: revert the GENERATION_TO_COMPLETE exclusion that NVIDIA#13900 added (upstream has no such exclusion). Under V1 ADP it undercounted -> a spurious ADP dummy was inserted on top of a real request -> the batch exceeded max_batch_size=1 and tripped the mamba dummy-mask assert / "No free slots". Restores upstream's exact method (fixes test_ptp_quickstart_advanced_deepseek_v3_lite_4gpus_adp_balance). - _handle_responses KV-timeout: gate the tp_allgather on disagg (kv_cache_transceiver is not None), not just enable_attention_dp. py_kv_transfer_timed_out is disagg-only, so in non-disagg ADP this added a spurious per-iteration collective that desynced adp_router's tp_allgather (gather_all_rank_states received a bool -> TypeError). Verified by 4-GPU DeepSeek-V3-Lite adp_balance e2e A/B (buggy: timeout hang; fixed: completes). - transceiver: only short-circuit the tp_allgather skip when pp_size==1 (_ctx_need_pp_sync) -- the PP>1 path asymmetrically flips send/recv markers across pipeline stages and deadlocks the _ctx_consensus pp_allgather. - py_executor: restore main's immediate benchmark fail-fast guard. - resource_manager: do NOT narrow trim_to_history's except (resize() can raise non-ValueError under v2 SWA + uneven-PP; narrowing leaked KV blocks). Tests (added to existing files): - test_py_executor.py: disagg cache-error sync + ADP no-op paths; ADP dummy-role behavior; GENERATION_TO_COMPLETE counts as active (adp_balance regression); CTX dummy padding for disagg idle ranks (incl. awaiting-KV-transfer-only). - test_kv_cache_v2_scheduler.py: trim_to_history. - test_cache_reuse_adapter.py: trim-to-prompt-history + transceiver ctx mgr; _create_kv_slice TokenRange (stub sets py_beam_width for the NVIDIA#14876 path). - test_router.py: finish_request explicit-session forwarding. - test_agent.py: BindingsNixlTransferStatus + shutdown idempotency (NVIDIA#14137). - transferAgentTest.cpp: status-outlives-agent (weak_ptr UAF safety) + concurrent submitTransferRequests. Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com>

github-actions Bot assigned Shixiaowei02 May 8, 2026

Shixiaowei02 requested a review from qiaoxj07 May 8, 2026 09:56

Shixiaowei02 added the deepseek-v4 label May 8, 2026

qiaoxj07 mentioned this pull request May 8, 2026

[None][fix] Sync disagg KV transfer error handling under ADP across all executor loops #13902

Closed

2 tasks

Shixiaowei02 force-pushed the user/xiaoweis/dsv4-adp-fix branch from b06d7af to 57977cd Compare May 9, 2026 01:30

Shixiaowei02 marked this pull request as ready for review May 9, 2026 04:53

Shixiaowei02 requested review from a team as code owners May 9, 2026 04:53

Shixiaowei02 requested review from StanleySun639, bo-nv, byshiue, chuangz0, dongxuy04, lfr-0531, pcastonguay and reasonsolo and removed request for a team May 9, 2026 04:53

Shixiaowei02 changed the title ~~[None][fix] Fix dis-agg with ADP and avoid TP collective deadlock~~ [None][fix] Avoid TP collective deadlock in ADP disagg error path May 9, 2026

Shixiaowei02 added 2 commits May 9, 2026 14:15

fix overlap and pp

803c68b

Signed-off-by: Xiaowei Shi <39303645+Shixiaowei02@users.noreply.github.com>

Shixiaowei02 force-pushed the user/xiaoweis/dsv4-adp-fix branch from 57977cd to 803c68b Compare May 9, 2026 06:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[None][fix] Avoid TP collective deadlock in ADP disagg error path - #13900

[None][fix] Avoid TP collective deadlock in ADP disagg error path#13900
Shixiaowei02 merged 2 commits into
NVIDIA:feat/deepseek_v4from
Shixiaowei02:user/xiaoweis/dsv4-adp-fix

Shixiaowei02 commented May 8, 2026 •

edited

Loading

Uh oh!

Shixiaowei02 commented May 8, 2026

Uh oh!

tensorrt-cicd commented May 8, 2026

Uh oh!

tensorrt-cicd commented May 8, 2026

Uh oh!

Shixiaowei02 commented May 9, 2026

Uh oh!

tensorrt-cicd commented May 9, 2026

Uh oh!

tensorrt-cicd commented May 9, 2026

Uh oh!

Shixiaowei02 commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Shixiaowei02 commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

Shixiaowei02 commented May 8, 2026

Uh oh!

tensorrt-cicd commented May 8, 2026

Uh oh!

tensorrt-cicd commented May 8, 2026

Uh oh!

Shixiaowei02 commented May 9, 2026

Uh oh!

tensorrt-cicd commented May 9, 2026

Uh oh!

tensorrt-cicd commented May 9, 2026

Uh oh!

Shixiaowei02 commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Shixiaowei02 commented May 8, 2026 •

edited

Loading