[https://nvbugs/6007967][fix] fix disagg pp hang issue by bo-nv · Pull Request #12528 · NVIDIA/TensorRT-LLM

bo-nv · 2026-03-25T07:25:39Z

Summary by CodeRabbit

Bug Fixes
- Improved inter-process communication synchronization during distributed batch processing to enhance reliability and prevent potential communication delays in multi-node tensor parallelism inference deployments.
Tests
- Removed test waivers for disaggregated serving scenarios, confirming improved system stability and successful verification of tensor parallelism functionality across various parameter configurations.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-03-25T07:32:32Z

📝 Walkthrough

Walkthrough

This pull request adds conditional MPI send flushing logic to the sample state broadcasting loop in the PyExecutor, forcing an MPI progress-driving wait when the executed batch queue is empty. Additionally, three test waivers for disaggregated serving tests are removed from the waiver list.

Changes

Cohort / File(s)	Summary
MPI Synchronization `tensorrt_llm/_torch/pyexecutor/py_executor.py`	Adds conditional MPI send handle waiting in `_broadcast_sample_state_loop`: when `executed_batch_queue` is empty, calls `wait_on_pp_send_handles(...)` on pending sends for the current microbatch before blocking on queue retrieval.
Test Waivers `tests/integration/test_lists/waives.txt`	Removes three test waivers for `accuracy/test_disaggregated_serving.py::TestLlama3_1_8BInstruct::test_ctx_pp_gen_tp_asymmetric` parameterizations related to GSM8K/MMLU with specific ctx_pp and gen_tp configurations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Description check	⚠️ Warning	The PR description is completely empty, containing only the template structure with no substantive information about the fix, test coverage, or checklist completion.	Add a clear description of the issue being fixed (disagg pp hang), explain the solution, list relevant test cases, and fill out the PR checklist items.

✅ Passed checks (1 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the issue (NVBugs 6007967) and describes the fix (disagg pp hang issue), directly related to the code changes in py_executor.py.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

bo-nv · 2026-03-25T07:41:50Z

/bot run

tensorrt-cicd · 2026-03-25T07:48:18Z

PR_Github #40286 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-25T13:52:27Z

PR_Github #40286 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31399 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-25T14:08:42Z

/bot run

tensorrt-cicd · 2026-03-25T14:14:22Z

PR_Github #40335 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-25T19:49:35Z

PR_Github #40335 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31441 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-25T20:13:15Z

/bot run

tensorrt-cicd · 2026-03-25T20:20:04Z

PR_Github #40365 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-26T00:46:42Z

PR_Github #40365 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31467 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-26T01:13:45Z

/bot run

tensorrt-cicd · 2026-03-26T01:19:51Z

PR_Github #40397 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-26T06:06:39Z

PR_Github #40397 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31492 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-26T06:14:18Z

/bot run

tensorrt-cicd · 2026-03-26T06:20:30Z

PR_Github #40427 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-26T07:46:45Z

PR_Github #40427 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31520 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-28T09:22:45Z

/bot run

tensorrt-cicd · 2026-03-28T09:28:53Z

PR_Github #40535 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-28T09:28:54Z

PR_Github #40535 [ run ] completed with state DISABLED
CI server is currently disabled for scheduled maintenance. Estimated completion time: 9 PM PST on 3/28.

Link to invocation

bo-nv · 2026-03-28T14:58:28Z

/bot run

tensorrt-cicd · 2026-03-28T15:05:11Z

PR_Github #40545 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-28T15:05:12Z

PR_Github #40545 [ run ] completed with state DISABLED
CI server is currently disabled for scheduled maintenance. Estimated completion time: 9 PM PST on 3/28.

Link to invocation

tensorrt-cicd · 2026-03-31T05:51:42Z

PR_Github #40816 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31830 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-31T14:48:38Z

/bot run

tensorrt-cicd · 2026-03-31T14:55:18Z

PR_Github #40953 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-31T17:33:07Z

PR_Github #40953 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31941 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-31T17:49:10Z

/bot run

tensorrt-cicd · 2026-03-31T17:54:46Z

PR_Github #40993 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-03-31T20:49:50Z

PR_Github #40993 [ run ] completed with state SUCCESS. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #31973 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-03-31T21:49:50Z

/bot run

tensorrt-cicd · 2026-03-31T21:55:37Z

PR_Github #41023 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-04-01T00:15:44Z

PR_Github #41023 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #32002 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-04-01T01:25:54Z

/bot run

tensorrt-cicd · 2026-04-01T01:33:19Z

PR_Github #41054 [ run ] triggered by Bot. Commit: 00c7fdb Link to invocation

tensorrt-cicd · 2026-04-01T03:24:22Z

PR_Github #41054 [ run ] completed with state FAILURE. Commit: 00c7fdb
/LLM/main/L0_MergeRequest_PR pipeline #32030 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-04-01T10:03:12Z

/bot run

Signed-off-by: Bo Deng <deemod@nvidia.com>

bo-nv · 2026-04-01T10:08:11Z

/bot run

tensorrt-cicd · 2026-04-01T10:13:58Z

PR_Github #41183 [ run ] triggered by Bot. Commit: 638949c Link to invocation

tensorrt-cicd · 2026-04-01T15:02:06Z

PR_Github #41183 [ run ] completed with state SUCCESS. Commit: 638949c
/LLM/main/L0_MergeRequest_PR pipeline #32146 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-04-01T23:22:08Z

/bot run

tensorrt-cicd · 2026-04-01T23:27:50Z

PR_Github #41279 [ run ] triggered by Bot. Commit: 638949c Link to invocation

tensorrt-cicd · 2026-04-02T07:17:08Z

PR_Github #41279 [ run ] completed with state SUCCESS. Commit: 638949c
/LLM/main/L0_MergeRequest_PR pipeline #32237 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

bo-nv · 2026-04-02T07:33:01Z

/bot run

tensorrt-cicd · 2026-04-02T07:41:01Z

PR_Github #41379 [ run ] triggered by Bot. Commit: 638949c Link to invocation

tensorrt-cicd · 2026-04-02T11:14:52Z

PR_Github #41379 [ run ] completed with state SUCCESS. Commit: 638949c
/LLM/main/L0_MergeRequest_PR pipeline #32319 completed with status: 'SUCCESS'

CI Report

Link to invocation

Signed-off-by: Bo Deng <deemod@nvidia.com>

bo-nv requested a review from a team as a code owner March 25, 2026 07:25

bo-nv requested a review from lancelly March 25, 2026 07:25

github-actions Bot assigned bo-nv Mar 25, 2026

bo-nv requested review from Shixiaowei02, chuangz0 and pcastonguay March 25, 2026 07:25

Shixiaowei02 approved these changes Mar 25, 2026

View reviewed changes

pcastonguay approved these changes Mar 25, 2026

View reviewed changes

bo-nv added 2 commits April 1, 2026 10:05

[https://nvbugs/6007967][fix] fix disagg pp hang issue

cc6ea22

Signed-off-by: Bo Deng <deemod@nvidia.com>

unwaive tests

638949c

Signed-off-by: Bo Deng <deemod@nvidia.com>

bo-nv force-pushed the main-6007967 branch from 00c7fdb to 638949c Compare April 1, 2026 10:07

bo-nv enabled auto-merge (squash) April 1, 2026 10:08

bo-nv merged commit 2b4f54c into NVIDIA:main Apr 2, 2026
5 checks passed

karen-sy pushed a commit to karen-sy/TensorRT-LLM that referenced this pull request Apr 7, 2026

[https://nvbugs/6007967][fix] fix disagg pp hang issue (NVIDIA#12528)

9a24d1a

Signed-off-by: Bo Deng <deemod@nvidia.com>

bo-nv mentioned this pull request Jun 3, 2026

[https://nvbugs/6194812][fix] Restore non-blocking PP send-handle progress in broadcast loop #14850

Open

2 tasks

Uh oh!

Conversation

bo-nv commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (2 warnings)

Uh oh!

bo-nv commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

bo-nv commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

bo-nv commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 25, 2026

Uh oh!

tensorrt-cicd commented Mar 26, 2026

Uh oh!

bo-nv commented Mar 26, 2026

Uh oh!

tensorrt-cicd commented Mar 26, 2026

Uh oh!

tensorrt-cicd commented Mar 26, 2026

Uh oh!

bo-nv commented Mar 26, 2026

Uh oh!

tensorrt-cicd commented Mar 26, 2026

Uh oh!

tensorrt-cicd commented Mar 26, 2026

Uh oh!

bo-nv commented Mar 28, 2026

Uh oh!

tensorrt-cicd commented Mar 28, 2026

Uh oh!

tensorrt-cicd commented Mar 28, 2026

Uh oh!

bo-nv commented Mar 28, 2026

Uh oh!

tensorrt-cicd commented Mar 28, 2026

Uh oh!

tensorrt-cicd commented Mar 28, 2026

Uh oh!

tensorrt-cicd commented Mar 31, 2026

Uh oh!

bo-nv commented Mar 31, 2026

Uh oh!

tensorrt-cicd commented Mar 31, 2026

Uh oh!

tensorrt-cicd commented Mar 31, 2026

Uh oh!

bo-nv commented Mar 31, 2026

Uh oh!

tensorrt-cicd commented Mar 31, 2026

Uh oh!

tensorrt-cicd commented Mar 31, 2026

Uh oh!

bo-nv commented Mar 31, 2026

Uh oh!

tensorrt-cicd commented Mar 31, 2026

Uh oh!

tensorrt-cicd commented Apr 1, 2026

Uh oh!

bo-nv commented Apr 1, 2026

bo-nv commented Mar 25, 2026 •

edited

Loading

coderabbitai Bot commented Mar 25, 2026 •

edited

Loading