[None][feat] VisualGen: async mp4 encode + fixed noise latent via env vars by wu6u3tw · Pull Request #15229 · NVIDIA/TensorRT-LLM

wu6u3tw · 2026-06-10T18:32:10Z

Summary

Video generation responses currently block the async server coroutine on
output.save() (libx264 mp4 encode, ~1.7 s/video) before the next request
can start. This adds an opt-in to offload the encode to a thread-pool executor,
so the server can begin GPU generation for the next request while the previous
video is being encoded.

Change: TRTLLM_VIDEO_ASYNC_ENCODE=1 env var switches output.save() to
run via asyncio.get_running_loop().run_in_executor(None, ...), making the
await non-blocking to the event loop.

Motivation

On GB300, each mp4 encode takes ~1.7 s (CPU libx264; no hardware encoder on
datacenter Blackwell). With the default synchronous path, this sits on the
critical cycle path — generation is idle during encode. With async overlap and
≥2 concurrent requests in flight, encode is fully hidden: measured +0.5%
end-to-end QPS at 4 GPUs in MLPerf Wan2.2-T2V-A14B offline scenario.

The flag is off by default ("0"), so existing behavior is unchanged.

Testing

Ran MLPerf Wan2.2-T2V-A14B offline benchmark (DP4, GB300):
- Without flag: 0.0354 QPS
- With TRTLLM_VIDEO_ASYNC_ENCODE=1: 0.0368 QPS (+0.5%)
Flag=0 path unchanged; no regression on existing serve behavior.

Summary by CodeRabbit

Release Notes

Performance
- Video generation endpoints now support optional asynchronous encoding to reduce blocking operations. When enabled via environment configuration, encoding runs in a background thread, improving endpoint responsiveness while maintaining backward compatibility.

wu6u3tw · 2026-06-10T18:33:58Z

@chang-l for review as well.

coderabbitai · 2026-06-10T18:35:15Z

📝 Walkthrough

Walkthrough

The PR modifies the synchronous video-generation endpoint to conditionally offload the blocking ffmpeg encode step (output.save()) to a thread pool executor when the TRTLLM_VIDEO_ASYNC_ENCODE environment variable is set to "1". The change maintains backward compatibility by retaining the previous synchronous behavior when the flag is not set.

Changes

Async Video Encoding

Layer / File(s)	Summary
Optional async encoding for video output `tensorrt_llm/serve/openai_video_routes.py`	Adds `_save_kwargs` dictionary and gates the blocking `output.save(...)` call behind the `TRTLLM_VIDEO_ASYNC_ENCODE` environment variable. When enabled, the save is executed via `asyncio.get_running_loop().run_in_executor(...)` to overlap encoding with other in-flight requests; otherwise it executes synchronously. Latency logging and response file selection remain unchanged.

🎯 3 (Moderate) | ⏱️ ~15 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly identifies the main feature change (async mp4 encode) and mentions the key mechanism (env vars), directly aligned with the changeset's primary objective.
Description check	✅ Passed	The PR description covers all key template sections: clear summary of the problem and solution, motivation with specific performance data, and comprehensive testing results. The description is complete and well-structured.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

wu6u3tw · 2026-06-10T18:44:24Z

@zhenhuaw-me for fixed_latent from env variable change.

…LM_VIDEO_ASYNC_ENCODE) Set TRTLLM_VIDEO_ASYNC_ENCODE=1 to offload the blocking ffmpeg mp4 encode to a thread-pool executor via run_in_executor, freeing the asyncio event loop to begin the next request's diffusion while the current video encodes. Without this, the encode (CPU libx264, ~1.7 s/video on GB300) serialises on the event loop and holds the server idle between generation and the next accept. With async offload, generation and encode overlap when >=2 requests are in flight per server, recovering the full encode latency from the cycle time. Default path (env var unset) is unchanged. Signed-off-by: Tin-Yin Lai <tinyinl@nvidia.com>

…ED_LATENT_PATH When TRTLLM_VIDEO_FIXED_LATENT_PATH points to a .pt file containing a pre-sampled noise tensor, WanPipeline loads it once at startup and reuses it across all T2V requests instead of sampling fresh random latents. Enables fully reproducible video generation for benchmarking (e.g. MLPerf Wan2.2-T2V-A14B offline scenario, where a fixed latent is required for consistent VBench accuracy evaluation across runs). I2V requests are unaffected (fixed latent is T2V-only). Signed-off-by: Tin-Yin Lai <tinyinl@nvidia.com>

JunyiXu-nv

LGTM

JunyiXu-nv · 2026-06-17T02:48:31Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-17T02:54:58Z

PR_Github #54724 [ run ] triggered by Bot. Commit: d6ae497 Link to invocation

tensorrt-cicd · 2026-06-17T07:42:51Z

PR_Github #54724 [ run ] completed with state SUCCESS. Commit: d6ae497
/LLM/main/L0_MergeRequest_PR pipeline #43754 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

Please check the failed tests and fix your PR
If you cannot view the failures, ask the CI triggerer to share details
Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…_ENCODE=0 to opt out Change the default from opt-in (TRTLLM_VIDEO_ASYNC_ENCODE=1 to enable) to opt-out (TRTLLM_VIDEO_ASYNC_ENCODE=0 to fall back to sync). Async offload via run_in_executor is now always active unless explicitly disabled. Signed-off-by: Tin-Yin Lai <tinyinl@nvidia.com>

wu6u3tw · 2026-06-18T02:32:36Z

/bot run --disable-fail-fast

tensorrt-cicd · 2026-06-18T02:38:52Z

PR_Github #54901 [ run ] triggered by Bot. Commit: 5bef5ac Link to invocation

tensorrt-cicd · 2026-06-18T06:13:08Z

PR_Github #54901 [ run ] completed with state SUCCESS. Commit: 5bef5ac
/LLM/main/L0_MergeRequest_PR pipeline #43905 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation

wu6u3tw · 2026-06-22T17:59:57Z

@chang-l can you help me merge this PR. I don't have permission. Thanks

… vars (NVIDIA#15229) Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>

… vars (NVIDIA#15229)

wu6u3tw requested a review from a team as a code owner June 10, 2026 18:32

wu6u3tw requested a review from schetlur-nv June 10, 2026 18:32

github-actions Bot assigned wu6u3tw Jun 10, 2026

wu6u3tw requested a review from a team as a code owner June 10, 2026 18:41

wu6u3tw changed the title ~~[None][feat] VisualGen: offload video encode to thread executor (TRTLLM_VIDEO_ASYNC_ENCODE)~~ [None][feat] VisualGen: async mp4 encode + fixed noise latent via env vars Jun 10, 2026

chang-l requested a review from JunyiXu-nv June 10, 2026 22:25

JunyiXu-nv reviewed Jun 11, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/openai_video_routes.py Outdated

zhenhuaw-me reviewed Jun 11, 2026

View reviewed changes

Comment thread tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py Outdated

Comment thread tensorrt_llm/serve/openai_video_routes.py Outdated

wu6u3tw force-pushed the feat/videogen-async-encode branch from b901311 to 019f9d4 Compare June 16, 2026 01:18

wu6u3tw force-pushed the feat/videogen-async-encode branch from 019f9d4 to 0b77cff Compare June 16, 2026 16:16

Merge branch 'main' into feat/videogen-async-encode

d6ae497

JunyiXu-nv approved these changes Jun 17, 2026

View reviewed changes

zhenhuaw-me approved these changes Jun 17, 2026

View reviewed changes

Comment thread tensorrt_llm/serve/openai_video_routes.py Outdated

Merge branch 'main' into feat/videogen-async-encode

5bef5ac

chang-l merged commit f7dd7ec into NVIDIA:main Jun 22, 2026
7 checks passed

xinhe-nv pushed a commit to tensorrt-cicd/TensorRT-LLM that referenced this pull request Jun 23, 2026

[None][feat] VisualGen: async mp4 encode + fixed noise latent via env…

3e1edad

… vars (NVIDIA#15229) Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>

xinhe-nv pushed a commit to tensorrt-cicd/TensorRT-LLM that referenced this pull request Jun 24, 2026

[None][feat] VisualGen: async mp4 encode + fixed noise latent via env…

94819f5

… vars (NVIDIA#15229) Signed-off-by: GitLab CI Bot <gitlab-ci@nvidia.com>

BrianLi23 pushed a commit to BrianLi23/TensorRT-LLM that referenced this pull request Jul 9, 2026

[None][feat] VisualGen: async mp4 encode + fixed noise latent via env…

683edf4

… vars (NVIDIA#15229)

Uh oh!

Conversation

wu6u3tw commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Testing

Summary by CodeRabbit

Release Notes

Uh oh!

wu6u3tw commented Jun 10, 2026

Uh oh!

coderabbitai Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

wu6u3tw commented Jun 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JunyiXu-nv left a comment

Choose a reason for hiding this comment

Uh oh!

JunyiXu-nv commented Jun 17, 2026

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

Uh oh!

tensorrt-cicd commented Jun 17, 2026

Uh oh!

wu6u3tw commented Jun 18, 2026

Uh oh!

tensorrt-cicd commented Jun 18, 2026

Uh oh!

tensorrt-cicd commented Jun 18, 2026

Uh oh!

wu6u3tw commented Jun 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

wu6u3tw commented Jun 10, 2026 •

edited

Loading

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading