Skip to content

[None][test] Add e2e example tests for flux1/2, ltx2, wan_i2v, and cosmos3 - #15126

Merged
chang-l merged 2 commits into
NVIDIA:mainfrom
chang-l:dev-visgen-example-e2e-tests
Jun 10, 2026
Merged

[None][test] Add e2e example tests for flux1/2, ltx2, wan_i2v, and cosmos3#15126
chang-l merged 2 commits into
NVIDIA:mainfrom
chang-l:dev-visgen-example-e2e-tests

Conversation

@chang-l

@chang-l chang-l commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Five example scripts were recently added under examples/visual_gen/models/
(flux1.py, flux2.py, ltx2.py, wan_i2v.py, cosmos3_ti2v.py) but none
had integration test coverage. This PR adds one smoke test per script and wires
them into CI.

Tests added (tests/integration/defs/examples/test_visual_gen.py):

  • test_flux1_example — runs flux1.py + configs/flux1-dev-fp4-1gpu.yaml
  • test_flux2_example — runs flux2.py + configs/flux2-dev-fp4-1gpu.yaml
  • test_ltx2_example — runs ltx2.py + configs/ltx2-t2v-fp4-1gpu.yaml (passes --text_encoder_path separately since the shared YAML omits it intentionally)
  • test_wan_i2v_example — runs wan_i2v.py + configs/wan2.2-i2v-fp4-1gpu.yaml (uses NVFP4 pre-quantized checkpoint, default cat_piano.png input image)
  • test_cosmos3_example — runs cosmos3_ti2v.py + configs/cosmos3-nano-1gpu.yaml

Each test:

  1. Skips gracefully if the model checkpoint is absent under LLM_MODELS_ROOT
  2. Passes the shared YAML config from examples/visual_gen/configs/
  3. Asserts the expected output file (.png or .mp4) is produced

CI registration (tests/integration/test_lists/test-db/l0_b200.yml):
All five tests registered in the B200 post-merge stage alongside the existing test_wan_t2v_example.

Test plan

  • Pre-merge CI (B200 post-merge stage) runs all 5 new tests
  • Tests with missing checkpoints skip cleanly rather than fail
  • Validate test list entries exist in source files (AST) pre-commit hook passes (already verified locally)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Tests
    • Added comprehensive end-to-end integration tests for visual generation examples, covering Flux1, Flux2, LTX-2, Wan 2.2 I2V, and Cosmos3.
    • New tests validate example scripts, configurations, and output generation for each model.

…smos3

The five newly added example scripts (flux1.py, flux2.py, ltx2.py,
wan_i2v.py, cosmos3_ti2v.py) under examples/visual_gen/models/ had no
integration test coverage.  Add one smoke test per script that:
 - loads the local checkpoint via LLM_MODELS_ROOT (skips if absent),
 - passes the shared per-model YAML from examples/visual_gen/configs/,
 - runs the script end-to-end,
 - asserts the expected output file is produced.

Register all five new tests in l0_b200.yml so they run in the B200
post-merge CI stage alongside the existing test_wan_t2v_example.

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@chang-l
chang-l requested a review from a team as a code owner June 8, 2026 22:57
@coderabbitai

coderabbitai Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

This PR expands VisualGen integration test coverage by adding a model checkpoint constant and five end-to-end example tests (Flux1, Flux2, LTX-2, Wan I2V, Cosmos3), then registering those tests in the pre-merge CI configuration.

Changes

VisualGen Example Tests

Layer / File(s) Summary
Model checkpoint constant definition
tests/integration/defs/examples/test_visual_gen.py
Defines WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH constant for the Wan I2V A14B NVFP4 checkpoint.
Visual-generation example integration tests
tests/integration/defs/examples/test_visual_gen.py
Implements five integration tests (test_flux1_example, test_flux2_example, test_ltx2_example, test_wan_i2v_example, test_cosmos3_example) that validate checkpoint resolution, confirm example scripts and YAML configs exist, execute examples with model-specific CLI arguments, and verify output media files are produced.
Test matrix configuration update
tests/integration/test_lists/test-db/l0_b200.yml
Registers the five new example tests in the l0_b200 pre-merge PyTorch test suite (1 GPU, Ubuntu).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

  • NVIDIA/TensorRT-LLM#14685: Extends VisualGen integration test coverage in tests/integration/defs/examples/test_visual_gen.py by adding end-to-end example tests.
  • NVIDIA/TensorRT-LLM#14981: Introduces the Wan I2V example script that the new test_wan_i2v_example integration test validates.
  • NVIDIA/TensorRT-LLM#14976: Introduces LTX-2 example artifacts that the new test_ltx2_example test exercises and validates.

Suggested labels

VisualGen

Suggested reviewers

  • laikhtewari
  • venkywonka
  • karljang
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and specifically summarizes the main change: adding end-to-end integration tests for five visual generation example scripts (flux1/2, ltx2, wan_i2v, cosmos3).
Description check ✅ Passed The pull request description is comprehensive and well-structured, covering the summary of changes, test details, CI registration, and test plan checklist.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
tests/integration/defs/examples/test_visual_gen.py (1)

1423-1429: 💤 Low value

Consider using _lpips_model_path() helper for consistency.

While the current checkpoint resolution pattern (direct os.path.join(conftest.llm_models_root(), ...)) matches the existing test_wan_t2v_example, the other four new tests in this PR use the _lpips_model_path() helper. For consistency within the new code, consider:

-    scratch_space = conftest.llm_models_root()
-    model_path = os.path.join(scratch_space, WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH)
-    if not os.path.isdir(model_path):
-        pytest.skip(
-            f"Model not found: {model_path} "
-            f"(set LLM_MODELS_ROOT or place {WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH} under models root)"
-        )
+    model_path = _lpips_model_path(WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH)
+    _skip_if_missing(model_path, "Wan 2.2 I2V checkpoint", is_dir=True)

This would align with test_flux1_example, test_flux2_example, test_ltx2_example, and test_cosmos3_example.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/integration/defs/examples/test_visual_gen.py` around lines 1423 - 1429,
Replace the ad-hoc model path construction in
tests/integration/defs/examples/test_visual_gen.py that uses scratch_space =
conftest.llm_models_root() and os.path.join(...,
WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH) with the existing helper _lpips_model_path()
used by the other new tests; call
_lpips_model_path(WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH) (or the equivalent helper
import) to obtain model_path and keep the same pytest.skip behavior if the
returned path is missing so the test matches test_flux1_example,
test_flux2_example, test_ltx2_example, and test_cosmos3_example.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@tests/integration/defs/examples/test_visual_gen.py`:
- Around line 1423-1429: Replace the ad-hoc model path construction in
tests/integration/defs/examples/test_visual_gen.py that uses scratch_space =
conftest.llm_models_root() and os.path.join(...,
WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH) with the existing helper _lpips_model_path()
used by the other new tests; call
_lpips_model_path(WAN22_I2V_A14B_NVFP4_MODEL_SUBPATH) (or the equivalent helper
import) to obtain model_path and keep the same pytest.skip behavior if the
returned path is missing so the test matches test_flux1_example,
test_flux2_example, test_ltx2_example, and test_cosmos3_example.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 91452b27-922f-4ae1-aa7d-31d32381173f

📥 Commits

Reviewing files that changed from the base of the PR and between 9827c21 and a4566c6.

📒 Files selected for processing (2)
  • tests/integration/defs/examples/test_visual_gen.py
  • tests/integration/test_lists/test-db/l0_b200.yml

@chang-l

chang-l commented Jun 8, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52875 [ run ] triggered by Bot. Commit: a4566c6 Link to invocation

@yibinl-nvidia yibinl-nvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the LTX-2 example test

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52875 [ run ] completed with state SUCCESS. Commit: a4566c6
/LLM/main/L0_MergeRequest_PR pipeline #42127 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

…e2e tests

- test_ltx2_example: pass specific safetensors file (ltx-2-19b-dev.safetensors)
  instead of directory; LTX-2 VisualGen loader requires a checkpoint file path
- test_cosmos3_example: set TRTLLM_DISABLE_COSMOS3_GUARDRAILS=1 to avoid
  cosmos_guardrail package dependency that is absent in CI

Signed-off-by: Chang Liu <9713593+chang-l@users.noreply.github.com>
@chang-l

chang-l commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52947 [ run ] triggered by Bot. Commit: 137f0c6 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator

PR_Github #52947 [ run ] completed with state SUCCESS. Commit: 137f0c6
/LLM/main/L0_MergeRequest_PR pipeline #42190 completed with status: 'SUCCESS'

CI Report

Link to invocation

@chang-l
chang-l merged commit 27b52b3 into NVIDIA:main Jun 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants