[None][feat] Add HunyuanVideo 1.5 text-to-video support to VisualGen

[jloftin-nv]

Summary by CodeRabbit

• New Features

  • Added HunyuanVideo 1.5 text-to-video generation with FP8 quantization support for single-GPU configurations.

• Documentation

  • Updated documentation with HunyuanVideo 1.5 model details and feature matrix including limitations.
  • Added example usage instructions and configuration files for HunyuanVideo 1.5 video generation.

• Tests

  • Added comprehensive test coverage for HunyuanVideo 1.5 implementation and quantization features.

Description

Added support for HunyuanVideo1.5 T2V pipeline along with unit tests and example to VisualGen. Wanted to get this merged then follow up with cache, parallelism, and maybe sage attention support

Test Coverage

Added test_hunyuan_video1_5_transformer.py and test_hunyuan_video1_5_pipeline.py

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds HunyuanVideo 1.5 single-GPU text-to-video support to TensorRT-LLM: a new hunyuan_video1_5 package containing a 3D diffusion transformer backbone, dual-encoder (Qwen2.5-VL + ByT5) pipeline with FP8/NVFP4 quantization, pipeline registry wiring, example CLI scripts and YAML configs, and unit/integration tests.

Changes

HunyuanVideo 1.5 Text-to-Video Pipeline

Layer / File(s)	Summary
Pipeline registry and module exports tensorrt_llm/_torch/visual_gen/pipeline_registry.py, tensorrt_llm/_torch/visual_gen/models/__init__.py, tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/__init__.py	Adds GUIDER to PipelineComponent, maps HunyuanVideo15 checkpoint class names to HunyuanVideo15Pipeline in _detect_from_checkpoint, and re-exports the pipeline and transformer classes from package init files.
Timestep and guidance conditioning embeddings tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/timestep_embedding.py	Adds CombinedTimestepTextProjEmbeddings and CombinedTimestepGuidanceTextProjEmbeddings modules that combine sinusoidal timestep, optional guidance, and pooled text projections into a single conditioning tensor.
HunyuanVideo 1.5 3D transformer backbone tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/transformer_hunyuan_video1_5.py	Implements rotary positional embeddings, adaptive LayerNorm modulation, fused-QKV joint attention with optional RoPE and encoder cross-attention, Qwen token refiner, ByT5 text projection, dual-stream transformer blocks, HunyuanVideo15Transformer3DModel forward pass with multi-modality token reordering, and HuggingFace weight loading with FP8/NVFP4 quantization support.
HunyuanVideo 1.5 inference pipeline tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/pipeline_hunyuan_video1_5.py	Adds HunyuanVideo15Pipeline with Qwen2.5-VL and ByT5 prompt encoding, standard component loading (tokenizers, text encoders, VAE, scheduler, guider), latent preparation, VAE decoding, and the main forward() denoising loop with CFG routing via extra_cfg_tensors.
Example scripts, configs, and docs examples/visual_gen/models/hunyuan_t2v.py, examples/visual_gen/configs/hunyuan-t2v-fp8-1gpu.yaml, examples/visual_gen/serve/configs/hunyuan.yml, examples/visual_gen/README.md, docs/source/models/visual-generation.md	Adds a CLI example script with _output_paths helper, an FP8 single-GPU YAML config, a serving config with cfg_size: 1 / ulysses_size: 1, README command examples, and HunyuanVideo 1.5 entries in the supported models table and feature matrix with a new footnote.
Transformer unit and pipeline integration tests tests/unittest/_torch/visual_gen/test_hunyuan_video1_5_transformer.py, tests/unittest/_torch/visual_gen/test_hunyuan_video1_5_pipeline.py, tests/integration/test_lists/test-db/l0_b200.yml	Adds transformer tests for structure, forward shape, and HF cosine-similarity comparison; pipeline tests for end-to-end TRT-LLM vs HF correctness, batch shape, FP8 attention, and per-quant-mode (BF16/FP8/FP8_BLOCK_SCALES/NVFP4) dtype, weight format, memory, and accuracy assertions; registers both test files in the l0_b200 integration test list.

Sequence Diagram

sequenceDiagram
    participant CLI as hunyuan_t2v.py
    participant VisualGen
    participant HunyuanVideo15Pipeline
    participant QwenEncoder as Qwen2.5-VL Encoder
    participant ByT5Encoder as ByT5 Encoder
    participant Transformer as HunyuanVideo15Transformer3DModel
    participant VAE

    CLI->>VisualGen: generate(prompt, height, width, num_frames)
    VisualGen->>HunyuanVideo15Pipeline: infer(req)
    HunyuanVideo15Pipeline->>QwenEncoder: _get_mllm_prompt_embeds(prompt)
    QwenEncoder-->>HunyuanVideo15Pipeline: prompt_embeds, prompt_embeds_mask
    HunyuanVideo15Pipeline->>ByT5Encoder: _get_byt5_prompt_embeds(glyph_texts)
    ByT5Encoder-->>HunyuanVideo15Pipeline: prompt_embeds_2, prompt_embeds_mask_2
    HunyuanVideo15Pipeline->>HunyuanVideo15Pipeline: _prepare_latents()
    loop Denoising steps
        HunyuanVideo15Pipeline->>Transformer: forward_fn(latents, cond_latents, timestep, embeds)
        Transformer-->>HunyuanVideo15Pipeline: noise_pred
        HunyuanVideo15Pipeline->>HunyuanVideo15Pipeline: scheduler.step()
    end
    HunyuanVideo15Pipeline->>VAE: decode(latents)
    VAE-->>HunyuanVideo15Pipeline: video frames
    HunyuanVideo15Pipeline-->>VisualGen: PipelineOutput
    VisualGen-->>CLI: saved video path(s)

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: adding HunyuanVideo 1.5 text-to-video support to VisualGen, which aligns with the changeset across documentation, examples, pipeline implementation, and tests.
Description check	✅ Passed	The PR description includes a clear explanation of what was added (HunyuanVideo1.5 T2V pipeline, tests, and examples), test coverage is documented (two test files listed), and the PR checklist is completed. However, the description lacks details about why this change was made and doesn't explain the limitations mentioned in documentation.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@examples/visual_gen/models/hunyuan_t2v.py`:
- Line 1: The file hunyuan_t2v.py is missing the required NVIDIA copyright
header that must appear at the very beginning of all new Python files in the
repository. Add the NVIDIA copyright header preamble with the current year
before the existing module docstring that starts with "HunyuanVideo 1.5
Text-to-Video generation." The header should be placed as the first lines of the
file, preceding all other content including imports and docstrings.
- Line 25: The main function is missing an explicit return type annotation,
which violates the project's Python typing guidelines. Update the main function
signature to include an explicit return type annotation of None, since this
function does not return anything. Change the function definition from `def
main():` to include `-> None` in the signature to satisfy the typing
requirement.

In
`@tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/transformer_hunyuan_video1_5.py`:
- Around line 958-959: The condition on line 958 uses `if timestep_r` to check
if the tensor is None, but since timestep_r is typed as
Optional[torch.LongTensor], this raises a RuntimeError when timestep_r contains
multiple elements because the truth value of multi-element tensors is ambiguous.
Replace the implicit truth value check with an explicit None check by changing
`if timestep_r` to `if timestep_r is not None` in the line that assigns
timestep_r to self.config.dtype.

In `@tests/unittest/_torch/visual_gen/test_hunyuan_video1_5_pipeline.py`:
- Around line 219-224: The TestHunyuanVideo15BatchGeneration class docstring
claims to test batch generation with list prompts, but the existing tests only
cover single-prompt behavior. Add a new test method to this class that
explicitly validates the current pipeline contract by passing multiple prompts
as input and asserting that a ValueError is raised (because effective batch size
> 1 is not supported). This ensures the test coverage aligns with the documented
class behavior and validates the expected error handling for batch inputs.
- Around line 149-164: The issue is that if _load_trtllm_pipeline fails on the
first line of the try block, trtllm_pipe is never assigned, causing an
UnboundLocalError in the finally block when _teardown_pipeline attempts to
reference it, which masks the original setup failure. Initialize trtllm_pipe to
None before the try block so the finally block can safely call
_teardown_pipeline even if setup fails. Apply the same pattern to all similar
try-finally blocks in the file (at lines 166-180, 230-247, and 265-283) where
other pipeline variables need initialization before their respective try blocks.
- Around line 386-401: The test_parameter_dtypes method requires CUDA to be
available but lacks a guard to skip the test when CUDA is unavailable, causing
it to fail on CPU-only environments. Add the same CUDA availability guard
decorator that is used by adjacent tests in the file to the
test_parameter_dtypes method to ensure it is skipped when CUDA is not available.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 24b4a132-6d82-4f00-8513-dee5c36edf15

📥 Commits

Reviewing files that changed from the base of the PR and between 07270b7 and e8446d4.

📒 Files selected for processing (14)

• docs/source/models/visual-generation.md
• examples/visual_gen/README.md
• examples/visual_gen/configs/hunyuan-t2v-fp8-1gpu.yaml
• examples/visual_gen/models/hunyuan_t2v.py
• examples/visual_gen/serve/configs/hunyuan.yml
• tensorrt_llm/_torch/visual_gen/models/__init__.py
• tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/__init__.py
• tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/pipeline_hunyuan_video1_5.py
• tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/timestep_embedding.py
• tensorrt_llm/_torch/visual_gen/models/hunyuan_video1_5/transformer_hunyuan_video1_5.py
• tensorrt_llm/_torch/visual_gen/pipeline_registry.py
• tests/integration/test_lists/test-db/l0_b200.yml
• tests/unittest/_torch/visual_gen/test_hunyuan_video1_5_pipeline.py
• tests/unittest/_torch/visual_gen/test_hunyuan_video1_5_transformer.py

[jloftin-nv]

/bot run

[zhenhuaw-me]

Not a blocker, cross-cutting across both your VG ports: this AdaLayerNormContinuous / timestep-embedding / FeedForward stack is a near-duplicate of what Qwen-Image already has and what the GlmImage PR (#16018) adds again. visual_gen/modules/ has no home for these yet — could the most uniform ones (timestep embeddings first) move into a shared module instead of another copy, or is there a port-fidelity reason to keep per-model copies? Worth a team-sync call either way.

[jloftin-nv]

Mainly just following precedent with each model having repeat/similar AdaNorms. Is it ok if I resolve this comment here and we can do a separate MR unifying timestep and AdaNorms?