[TRTLLM-12228][fix] ADP dummy type mismatch and collective deadlock

[Shixiaowei02]

@coderabbitai summary

Description

Stabilizes disaggregated serving when ADP is enabled.

• ADP dummy request type tracking: Dummy requests now mirror the most recent real request's CTX/GEN type instead of being hardcoded to GEN, and CTX dummies are padded to max_num_tokens so the MoE all-to-all sees aligned token counts across ranks.
• TP collective deadlock fix: Moves disagg KV cache transfer error handling to the top of the executor loop behind a TP allgather, ensuring all ranks enter the error path together so the downstream tp_gather in response enqueue no longer deadlocks. Includes matching transceiver/NIXL plumbing and a disagg+ADP accuracy test.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #45498 [ run ] triggered by Bot. Commit: 6fd59a0 Link to invocation

[tensorrt-cicd]

PR_Github #45499 [ run ] triggered by Bot. Commit: 6fd59a0 Link to invocation

[tensorrt-cicd]

PR_Github #45498 [ run ] completed with state ABORTED. Commit: 6fd59a0

Link to invocation

[tensorrt-cicd]

PR_Github #45499 [ run ] completed with state SUCCESS. Commit: 6fd59a0
/LLM/main/L0_MergeRequest_PR pipeline #35724 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[Shixiaowei02]

/bot --help

[github-actions]

GitHub Bot Help

/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...

Provide a user friendly way for developers to interact with a Jenkins server.

Run /bot [-h|--help] to print this help message.

See details below for each supported subcommand.

Details

run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental) --high-priority]

Launch build/test pipelines. All previously running jobs will be killed.

--reuse-test (optional)pipeline-id (OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.

--disable-reuse-test (OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.

--disable-fail-fast (OPTIONAL) : Disable fail fast on build/tests/infra failures.

--skip-test (OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.

--stage-list "A10-PyTorch-1, xxx" (OPTIONAL) : Only run the specified test stages. Supports wildcard * for pattern matching (e.g., "*PerfSanity*" matches all stages containing PerfSanity). Examples: "A10-PyTorch-1, xxx", "PerfSanity". Note: Does NOT update GitHub check status.

--gpu-type "A30, H100_PCIe" (OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.

--test-backend "pytorch, cpp" (OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.

--only-multi-gpu-test (OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.

--disable-multi-gpu-test (OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.

--add-multi-gpu-test (OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.

--post-merge (OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.

--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" (OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Supports wildcard * for pattern matching. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx", --extra-stage "Post-Merge".

--detailed-log (OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.

--debug (OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in the stage-list parameter to access the appropriate container environment. Note: Does NOT update GitHub check status.

--high-priority (OPTIONAL) : Run the pipeline with high priority. This option is restricted to authorized users only and will route the job to a high-priority queue.

kill

kill

Kill all running builds associated with pull request.

skip

skip --comment COMMENT

Skip testing for latest commit on pull request. --comment "Reason for skipping build/test" is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

reuse-pipeline

reuse-pipeline

Reuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #45574 [ run ] triggered by Bot. Commit: cdcf930 Link to invocation

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #45590 [ run ] triggered by Bot. Commit: c6a60e3 Link to invocation

[tensorrt-cicd]

PR_Github #45590 [ run ] completed with state SUCCESS. Commit: c6a60e3
/LLM/main/L0_MergeRequest_PR pipeline #35806 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces stateful control for ADP dummy request insertion and enhances scheduling diagnostics. Changes add role-locking logic to track dummy request type (context vs. generation), update schedulability accounting to exclude certain request states, provide structured diagnostics for scheduling failures, and expand test coverage for ADP-related behavior.

Changes

Cohort / File(s)	Summary
Core Executor Logic tensorrt_llm/_torch/pyexecutor/py_executor.py	Adds _adp_dummy_is_gen and _adp_dummy_role_locked state fields to control dummy request type; locks role based on first validated request's type (CTX-only → is_gen=False, GEN-only → is_gen=True). Updates schedulability accounting to exclude LlmRequestState.GENERATION_TO_COMPLETE requests. Enhances no-fitting-requests diagnostic paths to log structured output. Extends dummy padding to support context-type dummies with token_nums=[self.max_num_tokens] for MoE alignment.
Diagnostic Utilities tensorrt_llm/_torch/pyexecutor/request_utils.py	Imports LlmRequestState for readable state decoding. Introduces build_no_fitting_reqs_diagnostic() function that aggregates active request state distribution and KV cache capacity metrics (free/in-use slots, free blocks) with defensive exception handling.
Test Infrastructure & Cases tests/unittest/_torch/executor/test_benchmark_disagg.py, tests/unittest/_torch/executor/test_py_executor.py	Adds mock executor state fields (_adp_dummy_is_gen, _adp_dummy_role_locked, max_num_tokens) to test setup. Introduces _StubADPExecutor helper and comprehensive ADP-focused tests validating: role-locking behavior on first validated request, padding conditions and token parameter handling, GENERATION_TO_COMPLETE request filtering, and disagg vs. non-disagg code paths.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	PR description is incomplete and mostly consists of unfilled template placeholders; the Test Coverage section is empty and PR Checklist items lack substantive detail.	Provide explicit test coverage details (which test files/functions safeguard the changes) and complete the Description section with technical rationale beyond the title.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main change: ADP dummy type mismatch in disaggregated contexts, with a specific JIRA ticket [TRTLLM-12228] and [fix] type prefix.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🧹 Nitpick comments (1)

tensorrt_llm/_torch/pyexecutor/request_utils.py (1)

83-105: Narrow the diagnostic helper’s exception handling.

This path catches Exception three times and silently drops two of them, so a broken index_mapper/kv_cache_manager probe will look like “KV cache full” instead of showing that the diagnostic itself failed. Please catch the specific expected failures here and append a fallback marker instead of pass.

As per coding guidelines, "When using Python try-except blocks, limit the except clause to the smallest set of errors possible."

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@tensorrt_llm/_torch/pyexecutor/request_utils.py` around lines 83 - 105, The
enum conversion and diagnostic probes are too broadly swallowing all Exceptions;
narrow them and record fallback markers instead of silent pass: for
LlmRequestState(req.state_value) catch the specific enum/validation error (e.g.
ValueError) and set name to a clear fallback like f"state={req.state_value}
(invalid)"; for the index_mapper probes (idx_mapper.num_free_slots() and
idx_mapper.size()) catch only expected probe errors (e.g. AttributeError,
TypeError, RuntimeError) and append a marker such as ",
indexmapper_probe_failed" when they fail; likewise for
kv_cache_manager.get_num_free_blocks() catch only expected probe errors
(AttributeError, TypeError, RuntimeError) and append ",
free_blocks_probe_failed" instead of silently passing. Ensure you reference
LlmRequestState, idx_mapper, index_mapper.num_free_slots, index_mapper.size, and
kv_cache_manager.get_num_free_blocks when making these changes.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tensorrt_llm/_torch/pyexecutor/py_executor.py`:
- Around line 2845-2857: The ADP dummy role
(_adp_dummy_is_gen/_adp_dummy_role_locked) is being set only after per-rank
routing from validated_requests, causing idle ranks to default to GEN dummies;
change the logic to determine and lock the ADP dummy role before routing by
deriving it from executor configuration or broadcasting a role decision via
kv_cache_transceiver when enable_attention_dp is true (and kv_cache_transceiver
is not None), so that _pad_attention_dp_dummy_request() sees the correct role
even if a CTX worker has no local requests; ensure the new early path sets
_adp_dummy_is_gen and _adp_dummy_role_locked consistently and falls back only if
no broadcast/config value is available, and keep existing checks that reference
validated_requests as a secondary source.

In `@tests/unittest/_torch/executor/test_py_executor.py`:
- Around line 453-470: Tests currently duplicate the role-lock logic in a local
_run_role_lock function instead of exercising the real implementation; replace
the duplication by either invoking the production code path
(PyExecutor._fetch_and_activate_new_requests) with a PyExecutor instance and
minimal mocks/stubs for stub.enable_attention_dp, stub.kv_cache_transceiver, and
validated_requests (or by patching the surrounding interactions) so the real
block runs, or extract the role-lock block into a shared helper (e.g.,
PyExecutor._apply_role_lock or a module-level function) and update both
production to call that helper and tests to call the helper directly; target
symbols: _run_role_lock (remove), PyExecutor._fetch_and_activate_new_requests
(call), and LlmRequestType (use existing enums in tests) to locate the logic to
exercise.

---

Nitpick comments:
In `@tensorrt_llm/_torch/pyexecutor/request_utils.py`:
- Around line 83-105: The enum conversion and diagnostic probes are too broadly
swallowing all Exceptions; narrow them and record fallback markers instead of
silent pass: for LlmRequestState(req.state_value) catch the specific
enum/validation error (e.g. ValueError) and set name to a clear fallback like
f"state={req.state_value} (invalid)"; for the index_mapper probes
(idx_mapper.num_free_slots() and idx_mapper.size()) catch only expected probe
errors (e.g. AttributeError, TypeError, RuntimeError) and append a marker such
as ", indexmapper_probe_failed" when they fail; likewise for
kv_cache_manager.get_num_free_blocks() catch only expected probe errors
(AttributeError, TypeError, RuntimeError) and append ",
free_blocks_probe_failed" instead of silently passing. Ensure you reference
LlmRequestState, idx_mapper, index_mapper.num_free_slots, index_mapper.size, and
kv_cache_manager.get_num_free_blocks when making these changes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1879cd2e-0997-4f88-a3c8-29ed00552115

📥 Commits

Reviewing files that changed from the base of the PR and between dd907c0 and c6a60e3.

📒 Files selected for processing (4)

• tensorrt_llm/_torch/pyexecutor/py_executor.py
• tensorrt_llm/_torch/pyexecutor/request_utils.py
• tests/unittest/_torch/executor/test_benchmark_disagg.py
• tests/unittest/_torch/executor/test_py_executor.py

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #47197 [ run ] triggered by Bot. Commit: 61404c3 Link to invocation

[tensorrt-cicd]

PR_Github #47198 [ run ] triggered by Bot. Commit: 61404c3 Link to invocation

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #47297 [ run ] triggered by Bot. Commit: 46ea67b Link to invocation

[tensorrt-cicd]

PR_Github #47297 [ run ] completed with state FAILURE. Commit: 46ea67b

Link to invocation

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #47299 [ run ] triggered by Bot. Commit: 46ea67b Link to invocation

[Shixiaowei02]

/bot run --add-multi-gpu-test --disable-fail-fast

[tensorrt-cicd]

PR_Github #47371 [ run ] triggered by Bot. Commit: 287c6f6 Link to invocation

[tensorrt-cicd]

PR_Github #47371 [ run ] completed with state FAILURE. Commit: 287c6f6
/LLM/main/L0_MergeRequest_PR pipeline #37302 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation