[None][feat] Wan 2.2 serve: extend video generation API

[wu6u3tw]

Description

Adds four serve-side capabilities to the Wan 2.2 video generation pipeline.
All are independent, backward-compatible additive extensions to existing
public-facing schemas (no internal API changes).

1. VideoGenerationRequest knobs: guidance_scale_2, boundary_ratio,
   num_frames. guidance_scale_2/boundary_ratio are routed through
   extra_params to match WanPipeline.infer's extra.get(...) reads.
   Also defaults guidance_scale_2 = 3.0 on Wan 2.2 A14B when unset
   (was = guidance_scale, which is wrong for two-stage CFG).

2. fixed_latent_path: VisualGenArgs option to load a pre-computed
   latent tensor and substitute it for sampled latents at request time —
   used for MLPerf-deterministic generation. Lazy-loaded on first
   inference (pipeline construction runs inside MetaInitMode, where
   torch.load on a meta tensor would raise). Mirrors the existing
   extra_attrs pattern used by spatial_upsampler_path /
   distilled_lora_path. Warmup passes _use_fixed_latent=False.

3. response_format = "video_path": server returns JSON with the
   on-disk path instead of streaming bytes — for filesystem-shared
   deployments (e.g. Lustre). Avoids holding HTTP open through encode

   • transfer of multi-MB payloads. Default remains "video_bytes".

4. output_format = "raw": uncompressed uint8 RGB byte dump
   (T·H·W·3 layout, .rgb.bin). No ffmpeg dependency, no encode
   latency. Worst-case payload ~224 MB at 720x1280x81. Default remains
   "auto".

Test Coverage

• Existing tests/unittest/_torch/visual_gen/test_wan22_t2v_pipeline.py
  exercises the WAN pipeline path; new fields are additive and do not
  change existing test behavior.
• Manual verification: trtllm-serve <Wan2.2-T2V-A14B-Diffusers> then
  POST VideoGenerationRequest with each new field combination.

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why.

• PR Follows TRT-LLM CODING GUIDELINES.

• Test cases provided for new code paths (additive backward-compatible fields; no new code paths require dedicated tests beyond manual server verification).

• Any new dependencies have been scanned for license and vulnerabilities (none added).

• CODEOWNERS updated if ownership changes (no ownership change).

• Documentation updated as needed.

• Update tava architecture diagram if there is a significant design change in PR (n/a).

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR adds MLPerf-deterministic video generation support via fixed latent tensors, extends the video generation API with diffusion controls (guidance_scale_2, boundary_ratio, num_frames), and introduces raw RGB binary video export. Changes flow through config, model pipeline, OpenAI protocol, server, and utilities layers.

Changes

Fixed Latent Support

Layer / File(s)	Summary
Config & Field Definition tensorrt_llm/_torch/visual_gen/config.py	VisualGenArgs adds fixed_latent_path: Optional[str] to specify a pre-computed latent tensor path. Field is propagated to extra_attrs in from_pretrained for downstream access.
Pipeline Implementation tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py	Wan pipeline initializes fixed_latent_path and fixed_latent attributes before superclass setup. Forward method signature adds _use_fixed_latent: bool = True control flag. Latent preparation lazily loads fixed latent when available and enabled; warmup disables fixed latent usage. guidance_scale_2 default behavior adjusted to 3.0 for Wan 2.2 A14B two-stage when undefined.

Extended Diffusion Controls Protocol

Layer / File(s)	Summary
Protocol Definition tensorrt_llm/serve/openai_protocol.py	VideoGenerationRequest gains guidance_scale_2 (optional float), boundary_ratio (optional float with range constraints), and num_frames (optional int with minimum constraint) to enable extended diffusion control parameters.
Server Request Parsing & Response tensorrt_llm/serve/openai_server.py	Multipart/form-data parsing in _parse_video_generation_request now extracts new numeric fields (guidance_scale_2, boundary_ratio, num_frames). Synchronous video generation path gains support for response_format == "video_path" to return JSON (video_id and path) instead of streaming bytes.
Parameter Routing tensorrt_llm/serve/visual_gen_utils.py	parse_visual_gen_params propagates guidance_scale_2 and boundary_ratio from VideoGenerationRequest into extra_params. Adds logic to use num_frames from request when provided, otherwise compute from seconds and fps.

Raw Video Export

Layer / File(s)	Summary
Format Registration & Implementation tensorrt_llm/serve/media_storage.py	resolve_video_format adds "raw" format returning extension ".rgb.bin". MediaStorage.save_video new "raw" branch dumps video tensor as uint8 RGB bytes without encoding, dropping audio and normalizing tensor to uint8 if needed.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the main change: extending the video generation API for Wan 2.2 serve with new parameters and features.
Description check	✅ Passed	PR description comprehensively covers all changes with clear organization, rationale, and testing approach that aligns with the provided template structure.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

• Generate code and open pull requests
• Plan features and break down work
• Investigate incidents and troubleshoot customer tickets together
• Automate recurring tasks and respond to alerts with triggers
• Summarize progress and report instantly

Built for teams:

• Shared memory across your entire org—no repeating context
• Per-thread sandboxes to safely plan and execute work
• Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)

tensorrt_llm/serve/openai_server.py (2)

1951-1957: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Handle raw outputs with the correct content type.

resolve_video_format("raw") produces a .rgb.bin file, but this branch still maps everything except .mp4 to video/x-msvideo. For output_format="raw" the sync endpoint will stream arbitrary RGB bytes with an AVI media type.

Suggested fix

             actual_path = Path(actual_output_path)
-            media_type = "video/mp4" if actual_path.suffix == ".mp4" else "video/x-msvideo"
+            if actual_path.suffix == ".mp4":
+                media_type = "video/mp4"
+            elif actual_path.suffix == ".avi":
+                media_type = "video/x-msvideo"
+            else:
+                media_type = "application/octet-stream"
 
             return FileResponse(
                 actual_output_path,
                 media_type=media_type,
                 filename=actual_path.name,

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/serve/openai_server.py` around lines 1951 - 1957, The current
FileResponse branch always uses "video/x-msvideo" for non-.mp4 files, causing
raw outputs like ".rgb.bin" (from resolve_video_format("raw")) to be served with
the wrong MIME type; update the logic around actual_path / actual_output_path to
detect raw output (e.g., check actual_path.suffix == ".rgb.bin" or the local
variable output_format == "raw") and set media_type to a generic binary type
such as "application/octet-stream" (or a more specific raw/RGB MIME if desired)
before returning FileResponse so raw RGB byte streams are served with the
correct content type.

---

1997-2000: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Parse response_format from multipart requests too.

The new response_format="video_path" mode is only honored for JSON bodies right now. Multipart requests never copy response_format into data, so they always fall back to the model default and cannot opt into the new path-returning behavior.

Suggested fix

-            for field in ["model", "size", "negative_prompt", "output_format"]:
+            for field in ["model", "size", "negative_prompt", "output_format", "response_format"]:
                 if field in form and form[field]:
                     data[field] = form[field]

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tensorrt_llm/serve/openai_server.py` around lines 1997 - 2000, The multipart
parsing loop that copies optional string fields into data (for field in
["model", "size", "negative_prompt", "output_format"]) omits the new
response_format option, so multipart requests never get data["response_format"];
update that loop to include "response_format" (or otherwise copy
form["response_format"] into data when present) so the
response_format="video_path" mode is honored for multipart requests; look for
the variables form and data in openai_server.py to apply the change.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py`:
- Around line 483-495: The loaded fixed latent (in the _use_fixed_latent branch
where self.fixed_latent_path is used to set self.fixed_latent) must be validated
against the current expected latent shape (batch, channels, latent_frames,
latent_h, latent_w) before reuse; after torch.load and before assigning latents,
check self.fixed_latent.shape matches the computed target shape (derived from
the current batch size, self.latent_channels, computed latent_frames from
num_frames, and latent spatial dims from input resolution) and if it mismatches
either raise a clear ValueError indicating the expected vs actual shape or
optionally reshape/replicate the tensor only if a safe, documented
transformation is possible; update the logic around
self.fixed_latent.to(device=self.device, dtype=self.dtype) to run after
validation so you never pass an incompatible tensor into the denoiser/decoder
(refer to symbols: _use_fixed_latent, fixed_latent_path, fixed_latent, latents,
device, dtype).

In `@tensorrt_llm/serve/openai_server.py`:
- Around line 2013-2020: The multipart parsing currently treats form values by
truthiness and thus drops valid zero values (e.g., "0" or "0.0"); update the
checks for guidance_scale_2, boundary_ratio, guidance_rescale, and num_frames so
they only skip when the form key is missing or the value is an empty string (not
when the value is "0" or "0.0"), then parse and assign into the data dict as
before; locate the block in openai_server.py that sets data["guidance_scale_2"],
data["boundary_ratio"], data["guidance_rescale"], and data["num_frames"] and
replace the truthiness guards with explicit existence/empty-string checks before
casting.

---

Outside diff comments:
In `@tensorrt_llm/serve/openai_server.py`:
- Around line 1951-1957: The current FileResponse branch always uses
"video/x-msvideo" for non-.mp4 files, causing raw outputs like ".rgb.bin" (from
resolve_video_format("raw")) to be served with the wrong MIME type; update the
logic around actual_path / actual_output_path to detect raw output (e.g., check
actual_path.suffix == ".rgb.bin" or the local variable output_format == "raw")
and set media_type to a generic binary type such as "application/octet-stream"
(or a more specific raw/RGB MIME if desired) before returning FileResponse so
raw RGB byte streams are served with the correct content type.
- Around line 1997-2000: The multipart parsing loop that copies optional string
fields into data (for field in ["model", "size", "negative_prompt",
"output_format"]) omits the new response_format option, so multipart requests
never get data["response_format"]; update that loop to include "response_format"
(or otherwise copy form["response_format"] into data when present) so the
response_format="video_path" mode is honored for multipart requests; look for
the variables form and data in openai_server.py to apply the change.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 98e8c215-c95f-4c65-9382-a6fe519d8698

📥 Commits

Reviewing files that changed from the base of the PR and between 28b3471 and 6a16992.

📒 Files selected for processing (6)

• tensorrt_llm/_torch/visual_gen/config.py
• tensorrt_llm/_torch/visual_gen/models/wan/pipeline_wan.py
• tensorrt_llm/serve/media_storage.py
• tensorrt_llm/serve/openai_protocol.py
• tensorrt_llm/serve/openai_server.py
• tensorrt_llm/serve/visual_gen_utils.py

[zhenhuaw-me]

@wu6u3tw Can you clarify the motivation of this change? We are refactoring the Python APIs, once we have a "beta" python API, we plan to get back to the serving endpoint changes.

[wu6u3tw]

@zhenhuaw-me
I am adding support for adding param: gaidence_scale_2 and fixed_latened pt into trtllm-serve which is a supported param in non-serve run

[wu6u3tw]

@zhenhuaw-me Let's sync offline for the api change plan.

[wu6u3tw]

Fixed based on the reviews.

[wu6u3tw]

@zhenhuaw-me friendly ping — when you have a moment, could you take another look?

Since your last round of feedback, I've pushed:

• a42d32d — made fixed_latent_path a hybrid: env var TLLM_MLPERF_FIXED_LATENT_PATH now takes precedence over the VisualGenArgs.fixed_latent_path config field, so the debug/benchmark workflow you suggested is supported. (Left the discussion open for your call on whether the hybrid is acceptable, or you'd prefer env-var-only.)
• 933ae99 — added shape validation for the lazy-loaded fixed latent (CodeRabbit's suggestion, confirmed by them).
• 76d3adc — zero-value-safe multipart parsing for guidance_scale_2 / boundary_ratio / num_frames (CodeRabbit's suggestion, confirmed by them).
• bc5dd2f — replaced the raw RGB byte dump with torch.save (.pt), matching your pt comment.
• a796cf1 — made num_frames docstring explicit that it overrides both seconds and fps, per your suggestion.

All CI checks are green. Thanks!

[chang-l]

Approve to unblock — please also wait for @zhenhuaw-me to sign off as well

[wu6u3tw]

@zhenhuaw-me I am wondering in you API change, is the extra param wired into your new API?

[wu6u3tw]

Covered in other PRs