[#13076][fix] Destroy torch distributed process groups on PyExecutor shutdown by janbernloehr · Pull Request #12993 · NVIDIA/TensorRT-LLM

janbernloehr · 2026-04-13T08:35:17Z

Summary

trtllm-serve with --backend pytorch and TP>1 hits Aborted! (SIGABRT) during shutdown
The HTTP layer (uvicorn) shuts down cleanly, but the process aborts during NCCL/CUDA teardown
Root cause: PyExecutor.shutdown() tears down model engines, CUDA graphs, and resource managers, but never calls torch.distributed.destroy_process_group(), leaving NCCL communicators to be torn down non-deterministically by Python GC after MPI session shutdown, racing against CUDA context destruction
Fix: add destroy_process_group() at the end of PyExecutor.shutdown(), after all model/resource cleanup but before MPI teardown

Reproduction

Run trtllm-serve with --backend pytorch, --tp_size 4, via mpirun
Send SIGINT to trigger graceful shutdown
Observe Aborted! after "Application shutdown complete" / "Finished server process"

Test plan

Validated with kimi-k2-thinking (nvfp4, tp=4, pp=1) on gb200nvl — no more Aborted! during shutdown

🤖 Generated with Claude Code

Summary by CodeRabbit

Bug Fixes
- Improved shutdown sequence in distributed execution environments to properly clean up process group resources before system shutdown, ensuring more reliable and resource-efficient termination of distributed workloads.

coderabbitai · 2026-04-13T08:39:04Z

📝 Walkthrough

Walkthrough

The PyExecutor.shutdown() method now explicitly checks if PyTorch Distributed is initialized and calls torch.distributed.destroy_process_group() to tear down NCCL process groups before MPI session shutdown.

Changes

Cohort / File(s)	Summary
PyTorch Distributed Cleanup `tensorrt_llm/_torch/pyexecutor/py_executor.py`	Added explicit process group destruction in `shutdown()` method with a conditional check for `torch.distributed.is_initialized()` before calling `destroy_process_group()`.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description lacks the required template structure including ticket reference, type designation, test coverage section, and PR checklist. While the content explains the issue and solution well, it does not follow the repository's mandatory template format.	Restructure the description to follow the template: add ticket reference, ensure [type] prefix in title, complete the 'Test Coverage' section with specific test details, and include the 'PR Checklist' with checkboxes marked as appropriate.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title check	✅ Passed	The PR title directly and specifically describes the main change: adding explicit destruction of torch distributed process groups during PyExecutor shutdown, which matches the core fix in the changeset.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

janbernloehr · 2026-04-15T08:02:40Z

/bot run

tensorrt-cicd · 2026-04-15T08:09:08Z

PR_Github #43446 [ run ] triggered by Bot. Commit: d776e83 Link to invocation

tensorrt-cicd · 2026-04-15T10:32:37Z

PR_Github #43446 [ run ] completed with state SUCCESS. Commit: d776e83
/LLM/main/L0_MergeRequest_PR pipeline #33972 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

venkywonka · 2026-04-16T06:13:49Z

/bot run

janbernloehr · 2026-04-16T18:50:39Z

As far as i can tell the failures are all import errors that are not related to this PR.

tburt-nv · 2026-04-22T17:10:58Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-22T17:16:53Z

PR_Github #44994 [ run ] triggered by Bot. Commit: d776e83 Link to invocation

tensorrt-cicd · 2026-04-22T17:18:43Z

PR_Github #44994 [ run ] completed with state ABORTED. Commit: d776e83

Link to invocation

tburt-nv · 2026-04-22T17:18:48Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-22T17:24:47Z

PR_Github #44997 [ run ] triggered by Bot. Commit: 8cd8549 Link to invocation

tensorrt-cicd · 2026-04-23T06:56:42Z

PR_Github #44997 [ run ] completed with state SUCCESS. Commit: 8cd8549
/LLM/main/L0_MergeRequest_PR pipeline #35315 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

janbernloehr · 2026-04-23T22:13:51Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-04-23T22:19:56Z

PR_Github #45248 [ run ] triggered by Bot. Commit: c64c373 Link to invocation

tensorrt-cicd · 2026-04-24T12:03:01Z

PR_Github #45248 [ run ] completed with state SUCCESS. Commit: c64c373
/LLM/main/L0_MergeRequest_PR pipeline #35508 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

tensorrt-cicd · 2026-05-01T21:47:10Z

PR_Github #46525 [ run ] triggered by Bot. Commit: 2d4ce9e Link to invocation

tensorrt-cicd · 2026-05-02T10:53:10Z

PR_Github #46525 [ run ] completed with state SUCCESS. Commit: 2d4ce9e
/LLM/main/L0_MergeRequest_PR pipeline #36584 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

janbernloehr · 2026-05-07T06:47:25Z

/bot run

tensorrt-cicd · 2026-05-07T06:53:28Z

PR_Github #47137 [ run ] triggered by Bot. Commit: 583ed20 Link to invocation

tensorrt-cicd · 2026-05-07T11:57:47Z

PR_Github #47137 [ run ] completed with state SUCCESS. Commit: 583ed20
/LLM/main/L0_MergeRequest_PR pipeline #37101 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

janbernloehr · 2026-05-08T13:00:55Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-08T13:06:50Z

PR_Github #47402 [ run ] triggered by Bot. Commit: f3ee7ff Link to invocation

tensorrt-cicd · 2026-05-08T21:23:21Z

PR_Github #47402 [ run ] completed with state SUCCESS. Commit: f3ee7ff
/LLM/main/L0_MergeRequest_PR pipeline #37331 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

janbernloehr · 2026-05-11T14:01:10Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-11T14:08:07Z

PR_Github #47751 [ run ] triggered by Bot. Commit: 7e7b55d Link to invocation

tensorrt-cicd · 2026-05-12T14:08:56Z

PR_Github #47751 [ run ] completed with state ABORTED. Commit: 7e7b55d

Link to invocation

trtllm-serve with --backend pytorch and TP>1 hits SIGABRT during shutdown: the HTTP layer stops cleanly but the process aborts during NCCL/CUDA teardown. PyExecutor.shutdown() tears down model engines, CUDA graphs, and resource managers, but never calls torch.distributed.destroy_process_group(). This leaves NCCL communicators to be torn down non-deterministically by Python GC after MPI session shutdown, racing against CUDA context destruction. Add destroy_process_group() in GenerationExecutorWorker.shutdown() (the MPI/standalone worker), after engine cleanup but before MPI teardown. The call is placed here rather than in PyExecutor because PyExecutor does not own the process group — in the Ray executor path, RayWorkerWrapper owns the process group and PyExecutor is a tenant inside it. Destroying the default group from PyExecutor.shutdown() would break subsequent Ray worker calls. Signed-off-by: jbernloehr <jbernloehr@nvidia.com>

janbernloehr · 2026-05-13T14:40:48Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-13T14:47:39Z

PR_Github #48193 [ run ] triggered by Bot. Commit: d410d6c Link to invocation

tensorrt-cicd · 2026-05-14T10:54:09Z

PR_Github #48193 [ run ] completed with state SUCCESS. Commit: d410d6c
/LLM/main/L0_MergeRequest_PR pipeline #38012 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

tburt-nv · 2026-05-18T22:10:39Z

/bot skip --comment "test_fp8_blockscale[throughput_mtp_trtllm] passed on previous pipelines of this PR, it's flaky. unittest/kv_cache_manager_v2_tests/ was fixed by #14140, and also passed in a previous pipeline of this PR"

tensorrt-cicd · 2026-05-18T22:17:11Z

PR_Github #48989 Bot args parsing error: Failed to parse bot args

Link to invocation

tburt-nv · 2026-05-18T22:17:50Z

/bot skip --comment "all failing tests previously passed with the same changes"

tensorrt-cicd · 2026-05-18T22:24:06Z

PR_Github #48991 [ skip ] triggered by Bot. Commit: d410d6c Link to invocation

tensorrt-cicd · 2026-05-18T23:05:04Z

PR_Github #48991 [ skip ] completed with state SUCCESS. Commit: d410d6c
Skipping testing for commit d410d6c

Link to invocation

…cutor shutdown (NVIDIA#12993) Signed-off-by: jbernloehr <jbernloehr@nvidia.com>

janbernloehr requested a review from a team as a code owner April 13, 2026 08:35

janbernloehr requested a review from joyang-nv April 13, 2026 08:35

github-actions Bot assigned janbernloehr Apr 13, 2026

janbernloehr force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from f2917bb to d776e83 Compare April 15, 2026 08:02

janbernloehr changed the title ~~[fix] Destroy torch distributed process groups on PyExecutor shutdown~~ [None][fix] Destroy torch distributed process groups on PyExecutor shutdown Apr 15, 2026

janbernloehr mentioned this pull request Apr 15, 2026

[Bug]: trtllm-serve SIGABRT on shutdown with PyTorch backend and TP>1 #13076

Closed

janbernloehr changed the title ~~[None][fix] Destroy torch distributed process groups on PyExecutor shutdown~~ [#13076][fix] Destroy torch distributed process groups on PyExecutor shutdown Apr 15, 2026

mikeiovine approved these changes Apr 15, 2026

View reviewed changes

juney-nvidia approved these changes Apr 15, 2026

View reviewed changes

tburt-nv enabled auto-merge (squash) April 22, 2026 17:13

tburt-nv force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from d776e83 to 8cd8549 Compare April 22, 2026 17:18

auto-merge was automatically disabled April 23, 2026 22:13
Head branch was pushed to by a user without write access

janbernloehr force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from 8cd8549 to c64c373 Compare April 23, 2026 22:13

janbernloehr requested a review from a team as a code owner April 23, 2026 22:13

janbernloehr requested a review from syuoni April 23, 2026 22:13

janbernloehr force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from 2d4ce9e to 583ed20 Compare May 7, 2026 06:47

janbernloehr force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from 583ed20 to f3ee7ff Compare May 8, 2026 13:00

janbernloehr force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from f3ee7ff to 7e7b55d Compare May 11, 2026 14:01

janbernloehr force-pushed the jbernloehr/fix-pyexecutor-shutdown branch from 7e7b55d to d410d6c Compare May 13, 2026 14:40

tburt-nv enabled auto-merge (squash) May 18, 2026 22:10

tburt-nv disabled auto-merge May 18, 2026 22:17

tburt-nv enabled auto-merge (squash) May 18, 2026 22:17

tburt-nv merged commit 1f1e9aa into NVIDIA:main May 18, 2026
8 checks passed

KleinBlueC pushed a commit to KleinBlueC/TensorRT-LLM that referenced this pull request May 19, 2026

[NVIDIA#13076][fix] Destroy torch distributed process groups on PyExe…

67c5cf6

…cutor shutdown (NVIDIA#12993) Signed-off-by: jbernloehr <jbernloehr@nvidia.com>

bmarimuthu-nv pushed a commit to nv-auto-deploy/TensorRT-LLM that referenced this pull request May 28, 2026

[NVIDIA#13076][fix] Destroy torch distributed process groups on PyExe…

1d6e6ec

…cutor shutdown (NVIDIA#12993) Signed-off-by: jbernloehr <jbernloehr@nvidia.com>

Uh oh!

Conversation

janbernloehr commented Apr 13, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reproduction

Test plan

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

❌ Failed checks (1 warning)

Uh oh!

janbernloehr commented Apr 15, 2026

Uh oh!

tensorrt-cicd commented Apr 15, 2026

Uh oh!

tensorrt-cicd commented Apr 15, 2026

Uh oh!

venkywonka commented Apr 16, 2026

Uh oh!

janbernloehr commented Apr 16, 2026

Uh oh!

tburt-nv commented Apr 22, 2026

Uh oh!

tensorrt-cicd commented Apr 22, 2026

Uh oh!

tensorrt-cicd commented Apr 22, 2026

Uh oh!

tburt-nv commented Apr 22, 2026

Uh oh!

tensorrt-cicd commented Apr 22, 2026

Uh oh!

tensorrt-cicd commented Apr 23, 2026

Uh oh!

janbernloehr commented Apr 23, 2026

Uh oh!

tensorrt-cicd commented Apr 23, 2026

Uh oh!

tensorrt-cicd commented Apr 24, 2026

Uh oh!

tensorrt-cicd commented May 1, 2026

Uh oh!

tensorrt-cicd commented May 2, 2026

Uh oh!

janbernloehr commented May 7, 2026

Uh oh!

tensorrt-cicd commented May 7, 2026

Uh oh!

tensorrt-cicd commented May 7, 2026

Uh oh!

janbernloehr commented May 8, 2026

Uh oh!

tensorrt-cicd commented May 8, 2026

Uh oh!

tensorrt-cicd commented May 8, 2026

Uh oh!

janbernloehr commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 11, 2026

Uh oh!

tensorrt-cicd commented May 12, 2026

Uh oh!

janbernloehr commented May 13, 2026

Uh oh!

tensorrt-cicd commented May 13, 2026

Uh oh!

tensorrt-cicd commented May 14, 2026

Uh oh!

tburt-nv commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

tburt-nv commented May 18, 2026

Uh oh!

tensorrt-cicd commented May 18, 2026

Uh oh!

janbernloehr commented Apr 13, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Apr 13, 2026 •

edited

Loading