[https://nvbugs/6143883][fix] Preserve ip:port for trtllm-serve visual-gen by JunyiXu-nv · Pull Request #14355 · NVIDIA/TensorRT-LLM

JunyiXu-nv · 2026-05-20T09:12:34Z

The L0 Post-Merge test
unittest/_torch/visual_gen/test_trtllm_serve_e2e.py::TestWanTextToVideo::test_health flakes on DGX_B200 with "RuntimeError: Visual-gen server exited unexpectedly." Root cause is the same OCI port-conflict race that already hit the agg-LLM and disagg paths: there is a window between the test-side get_free_port() check and uvicorn's bind() inside trtllm-serve during which another container can grab the port. The visual-gen server makes the window much wider because VisualGen model initialization can take many minutes before uvicorn ever attempts to bind, so any colliding process almost always wins.

Apply the same socket-pre-binding pattern already adopted for the agg LLM server (nvbugs/5703953, PR #9646) and the disagg server (nvbugs/5727517, PR #9859): bind the listening socket before constructing VisualGen, then hand the bound socket to uvicorn via the existing OpenAIServer.call sockets= parameter. This collapses the check-to-bind window from "model load time" to "a few syscalls".

Summary by CodeRabbit

Bug Fixes

Enhanced the visual generation server startup sequence to ensure socket binding completes before model initialization. This improves server startup reliability and prevents potential connection issues during the initialization process, providing more predictable server behavior during launch.

Description

Test Coverage

PR Checklist

Please review the following before submitting your PR:

PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

coderabbitai · 2026-05-20T09:16:19Z

📝 Walkthrough

Walkthrough

The launch_visual_gen_server function is refactored to pre-bind a TCP listening socket before initializing the VisualGen model. The socket is created, bound to the target host and port, then passed to OpenAIServer via the sockets parameter, ensuring port binding occurs before the long-running model setup.

Changes

Socket pre-binding in Visual Gen server

Layer / File(s)	Summary
Socket pre-binding before model initialization `tensorrt_llm/commands/serve.py`	A listening socket is created and bound to host/port before VisualGen initialization, then passed to `OpenAIServer` via `sockets=[s]` to defer binding until after model setup completes.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

zhenhuaw-me

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately references the NVBugs ticket and describes the main fix: preserving the ip:port for the trtllm-serve visual-gen component.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description clearly explains the root cause (port-conflict race during VisualGen initialization) and the solution (socket pre-binding pattern), with references to related issues and PRs.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

JunyiXu-nv · 2026-05-20T09:22:48Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-20T09:28:21Z

PR_Github #49403 [ run ] triggered by Bot. Commit: ff5e568 Link to invocation

tensorrt-cicd · 2026-05-20T15:08:29Z

PR_Github #49403 [ run ] completed with state SUCCESS. Commit: ff5e568
/LLM/main/L0_MergeRequest_PR pipeline #39052 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-05-21T03:09:29Z

/bot run

tensorrt-cicd · 2026-05-21T03:14:53Z

PR_Github #49561 [ run ] triggered by Bot. Commit: ff5e568 Link to invocation

tensorrt-cicd · 2026-05-21T05:12:12Z

PR_Github #49561 [ run ] completed with state SUCCESS. Commit: ff5e568
/LLM/main/L0_MergeRequest_PR pipeline #39189 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-05-21T08:21:01Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-21T08:26:25Z

PR_Github #49655 [ run ] triggered by Bot. Commit: 80f04cb Link to invocation

tensorrt-cicd · 2026-05-21T16:54:24Z

PR_Github #49655 [ run ] completed with state SUCCESS. Commit: 80f04cb
/LLM/main/L0_MergeRequest_PR pipeline #39268 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-05-26T07:54:16Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-05-26T08:04:41Z

PR_Github #50317 [ run ] triggered by Bot. Commit: 8b00ca6 Link to invocation

tensorrt-cicd · 2026-05-26T13:14:33Z

PR_Github #50317 [ run ] completed with state SUCCESS. Commit: 8b00ca6
/LLM/main/L0_MergeRequest_PR pipeline #39846 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-06-02T05:40:22Z

/bot run

tensorrt-cicd · 2026-06-02T05:46:12Z

PR_Github #51513 [ run ] triggered by Bot. Commit: 8b00ca6 Link to invocation

tensorrt-cicd · 2026-06-02T09:36:21Z

PR_Github #51513 [ run ] completed with state SUCCESS. Commit: 8b00ca6
/LLM/main/L0_MergeRequest_PR pipeline #40915 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-06-05T06:08:04Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-05T06:16:04Z

PR_Github #52272 [ run ] triggered by Bot. Commit: 86cb02f Link to invocation

tensorrt-cicd · 2026-06-05T10:46:00Z

PR_Github #52272 [ run ] completed with state SUCCESS. Commit: 86cb02f
/LLM/main/L0_MergeRequest_PR pipeline #41583 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

NVShreyas · 2026-06-05T15:23:38Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-05T15:29:30Z

PR_Github #52363 [ run ] triggered by Bot. Commit: 86cb02f Link to invocation

tensorrt-cicd · 2026-06-05T19:05:05Z

PR_Github #52363 [ run ] completed with state SUCCESS. Commit: 86cb02f
/LLM/main/L0_MergeRequest_PR pipeline #41659 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

NVShreyas · 2026-06-05T19:57:10Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-05T20:04:04Z

PR_Github #52420 [ run ] triggered by Bot. Commit: 86cb02f Link to invocation

tensorrt-cicd · 2026-06-05T21:03:01Z

PR_Github #52420 [ run ] completed with state SUCCESS. Commit: 86cb02f
/LLM/main/L0_MergeRequest_PR pipeline #41713 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-06-08T02:36:44Z

/bot run

tensorrt-cicd · 2026-06-08T02:42:40Z

PR_Github #52638 [ run ] triggered by Bot. Commit: 86cb02f Link to invocation

tensorrt-cicd · 2026-06-08T03:31:56Z

PR_Github #52638 [ run ] completed with state FAILURE. Commit: 86cb02f
/LLM/main/L0_MergeRequest_PR pipeline #41917 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-06-08T08:56:34Z

/bot run

tensorrt-cicd · 2026-06-08T09:07:25Z

PR_Github #52726 [ run ] triggered by Bot. Commit: 86cb02f Link to invocation

tensorrt-cicd · 2026-06-08T09:55:30Z

PR_Github #52726 [ run ] completed with state SUCCESS. Commit: 86cb02f
/LLM/main/L0_MergeRequest_PR pipeline #41991 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…l-gen VisualGen model initialization can take many minutes between the test-side free-port check and uvicorn's bind() call. During that window anything else on the host can grab the port, and trtllm-serve dies at bind() time -- which the visual-gen e2e fixture surfaces as "RuntimeError: Visual-gen server exited unexpectedly." Fix: - launch_visual_gen_server now binds the listening socket *before* constructing VisualGen, then hands the bound socket to uvicorn via the existing OpenAIServer.__call__(..., sockets=[s]) path, collapsing the check-to-bind window to a few syscalls. - RemoteVisualGenServer in the visual-gen e2e test now allocates its port via get_free_port_in_ci, so parallel pytest sessions on the same OCI node fall into disjoint container port sections (CONTAINER_PORT_START / CONTAINER_PORT_NUM); falls back to the plain free-port scan when those env vars are not set. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

The previous commit accidentally renamed three attribute accesses on VisualGenArgs: - args.parallel -> should be args.parallel_config - .dit_cfg_size -> should be .cfg_size - .dit_ulysses_size -> should be .ulysses_size These look like a rebase artifact from a parallel branch where the field was temporarily renamed; main still exposes the original names (see tensorrt_llm/visual_gen/args.py: VisualGenArgs.parallel_config, ParallelConfig.cfg_size / ulysses_size / n_workers). The mismatch caused the visual-gen test to fail in CI: workers came up successfully, then immediately after `Workers ready` the next line `args.parallel.n_workers` raised AttributeError. The exception escaped the `with socket(...)` block, the socket was torn down before uvicorn ever bound it, and the test saw "Visual-gen server exited unexpectedly" / connection refused on /health. The socket pre-bind logic itself is unchanged and matches the agg-LLM (serve.py L313-360) and disagg (serve.py L1245-1276) patterns, which also bind without an explicit s.listen() — uvicorn / asyncio's loop.create_server(sock=...) calls listen() internally. Signed-off-by: JunyiXu-nv <219237550+JunyiXu-nv@users.noreply.github.com>

JunyiXu-nv · 2026-06-09T02:04:33Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-09T02:11:11Z

PR_Github #52916 [ run ] triggered by Bot. Commit: b97e8bd Link to invocation

tensorrt-cicd · 2026-06-09T05:26:25Z

PR_Github #52916 [ run ] completed with state SUCCESS. Commit: b97e8bd
/LLM/main/L0_MergeRequest_PR pipeline #42161 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-06-09T08:30:31Z

/bot run

tensorrt-cicd · 2026-06-09T08:36:28Z

PR_Github #53019 [ run ] triggered by Bot. Commit: b97e8bd Link to invocation

tensorrt-cicd · 2026-06-09T09:24:31Z

PR_Github #53019 [ run ] completed with state SUCCESS. Commit: b97e8bd
/LLM/main/L0_MergeRequest_PR pipeline #42244 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

JunyiXu-nv · 2026-06-10T05:51:53Z

/bot run

tensorrt-cicd · 2026-06-10T05:58:40Z

PR_Github #53247 [ run ] triggered by Bot. Commit: b97e8bd Link to invocation

tensorrt-cicd · 2026-06-10T07:41:12Z

PR_Github #53247 [ run ] completed with state SUCCESS. Commit: b97e8bd
/LLM/main/L0_MergeRequest_PR pipeline #42442 completed with status: 'SUCCESS'

CI Report

Link to invocation

zhenhuaw-me

LGTM

JunyiXu-nv requested a review from a team as a code owner May 20, 2026 09:12

JunyiXu-nv requested a review from syuoni May 20, 2026 09:12

github-actions Bot assigned JunyiXu-nv May 20, 2026

JunyiXu-nv force-pushed the user/junyix/fix-nvbug-6143883 branch from 7677e22 to ff5e568 Compare May 20, 2026 09:19

JunyiXu-nv requested a review from a team as a code owner May 20, 2026 09:19

JunyiXu-nv requested a review from chang-l May 20, 2026 09:23

JunyiXu-nv force-pushed the user/junyix/fix-nvbug-6143883 branch from ff5e568 to 80f04cb Compare May 21, 2026 08:18

JunyiXu-nv force-pushed the user/junyix/fix-nvbug-6143883 branch from 80f04cb to 8b00ca6 Compare May 26, 2026 03:22

NVShreyas approved these changes Jun 5, 2026

View reviewed changes

JunyiXu-nv added 2 commits June 9, 2026 10:04

JunyiXu-nv force-pushed the user/junyix/fix-nvbug-6143883 branch from 86cb02f to b97e8bd Compare June 9, 2026 02:04

zhenhuaw-me approved these changes Jun 11, 2026

View reviewed changes

zhenhuaw-me merged commit 5e3f012 into NVIDIA:main Jun 11, 2026
7 checks passed

luyiyun1021 mentioned this pull request Jul 7, 2026

[None][fix] trtllm-serve VisualGen: bind HTTP port only on rank 0 in multi-rank launch #16049

Merged

1 task

Uh oh!

Conversation

JunyiXu-nv commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Bug Fixes

Description

Test Coverage

PR Checklist

GitHub Bot Help

Uh oh!

coderabbitai Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested reviewers

Review ran into problems

Uh oh!

JunyiXu-nv commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

tensorrt-cicd commented May 20, 2026

Uh oh!

JunyiXu-nv commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

JunyiXu-nv commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

tensorrt-cicd commented May 21, 2026

Uh oh!

JunyiXu-nv commented May 26, 2026

Uh oh!

tensorrt-cicd commented May 26, 2026

Uh oh!

tensorrt-cicd commented May 26, 2026

Uh oh!

JunyiXu-nv commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

tensorrt-cicd commented Jun 2, 2026

Uh oh!

JunyiXu-nv commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

NVShreyas commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

NVShreyas commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

tensorrt-cicd commented Jun 5, 2026

Uh oh!

JunyiXu-nv commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

JunyiXu-nv commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

tensorrt-cicd commented Jun 8, 2026

Uh oh!

JunyiXu-nv commented Jun 9, 2026

JunyiXu-nv commented May 20, 2026 •

edited

Loading

coderabbitai Bot commented May 20, 2026 •

edited

Loading