[TRTLLM-13027][ci] Relocate under-using tests to right-sized stages#14684
Conversation
📝 WalkthroughWalkthroughThis PR updates test configurations across Jenkins CI infrastructure and multiple GPU-specific test matrices. It adds B200 2-GPU SLURM testing, expands B200 single-GPU tests (MoE and VisualGen), reorganizes the B200 multi-GPU test matrix with new 2-GPU pre_merge tests, adjusts existing 4-GPU/8-GPU blocks, and relocates a Ray disaggregated serving test from B200 to H100. ChangesB200 and H100 Test Matrix Configuration
🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_lists/test-db/l0_dgx_h100.yml`:
- Line 335: The QA list is missing an entry for the test node
examples/test_ray.py::test_ray_disaggregated_serving[tp2] which only appears in
l0_dgx_h100.yml and waives.txt; either add that node id to the appropriate QA
list (likely qa/llm_function_core.txt) so H100 relocation is covered, or update
waives.txt or a short README to document why this test is intentionally excluded
from QA; modify the relevant test list file to include the exact node id string
or add a one-line justification to waives.txt/QA doc accordingly.
- Line 335: Move the test entry
examples/test_ray.py::test_ray_disaggregated_serving[tp2] out of the H100 2-GPU
block (system_gpu_count: gte: 2, lte: 2) and add it to the H100 4-GPU
Ray/PyTorch block (system_gpu_count: gte: 4, lte: 4) in
tests/integration/test_lists/test-db/l0_dgx_h100.yml so the test can see the
required 4 GPUs (tp_size=2 → needs 4 GPUs).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 732fb4e8-c307-4858-8a8c-12b00038bc3b
📒 Files selected for processing (4)
jenkins/L0_Test.groovytests/integration/test_lists/test-db/l0_b200.ymltests/integration/test_lists/test-db/l0_dgx_b200.ymltests/integration/test_lists/test-db/l0_dgx_h100.yml
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
tests/integration/test_lists/test-db/l0_b200.yml (1)
306-314: QA list follow-up is optional for this PR.These changes are in
tests/integration/test_lists/test-db/(CI execution source) and do not add/modifytests/integration/defs/; scheduled QA list updates undertests/integration/test_lists/qa/are optional unless you explicitly want these newly placed single-GPU VisualGen cases in periodic QA runs.As per coding guidelines, "If the change adds or materially alters an integration test under tests/integration/defs/ (or otherwise affects what QA should run on a schedule), call out whether an entry is needed under tests/integration/test_lists/qa/."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/test_lists/test-db/l0_b200.yml` around lines 306 - 314, This change removes single-GPU VisualGen cases from the CI list in tests/integration/test_lists/test-db/l0_b200.yml (the entries referencing examples/test_visual_gen.py::test_visual_gen_quickstart and other test_visual_gen_* functions and visual_gen/test_visual_gen_benchmark.py::*), so either add corresponding scheduled QA entries under tests/integration/test_lists/qa/ if these tests should be run periodically, or explicitly confirm in the PR that no integration defs under tests/integration/defs/ were added/modified and that QA list updates are intentionally omitted; update the repo by adding QA list entries for those specific test identifiers if periodic execution is desired or add a PR note stating QA follow-up is not required.tests/integration/test_lists/test-db/l0_dgx_b200.yml (1)
3-28: QA list update is not required by default here.This is a CI matrix rebalance in
test-db(no new/changed integration defs), sotests/integration/test_lists/qa/*updates are not mandatory in this PR; add them only if you want these 2-GPU placements mirrored into scheduled QA cadence.As per coding guidelines, "Keep feedback actionable: suggest concrete list file names and whether coverage is sufficient, insufficient, or needs follow-up outside the PR."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/integration/test_lists/test-db/l0_dgx_b200.yml` around lines 3 - 28, Summary: QA list update not required by default for this 2-GPU CI matrix rebalance. If you decide to mirror these 2-GPU placements into scheduled QA, create a QA list file named qa_2gpu_b200.yml and copy the condition block (system_gpu_count: gte/lte 2, wildcards gpu '*b200*', linux_distribution_name ubuntu*, cpu x86_64, terms stage: pre_merge/backend: pytorch/orchestrator: mpi) and the exact test entries (e.g., unittest/_torch/misc/test_autotuner.py::test_autotuner_distributed_strategy, accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_bf16[tp2-CUTLASS], kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2DSv3Lite::test_mtp_draft_tokens, unittest/_torch/visual_gen/test_flux_pipeline.py::TestFluxParallelism::test_ulysses_2gpu_correctness, etc.); otherwise leave tests/integration/test_lists/qa/* untouched since coverage is sufficient for CI-only rebalance.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/integration/test_lists/test-db/l0_b200.yml`:
- Around line 306-314: This change removes single-GPU VisualGen cases from the
CI list in tests/integration/test_lists/test-db/l0_b200.yml (the entries
referencing examples/test_visual_gen.py::test_visual_gen_quickstart and other
test_visual_gen_* functions and visual_gen/test_visual_gen_benchmark.py::*), so
either add corresponding scheduled QA entries under
tests/integration/test_lists/qa/ if these tests should be run periodically, or
explicitly confirm in the PR that no integration defs under
tests/integration/defs/ were added/modified and that QA list updates are
intentionally omitted; update the repo by adding QA list entries for those
specific test identifiers if periodic execution is desired or add a PR note
stating QA follow-up is not required.
In `@tests/integration/test_lists/test-db/l0_dgx_b200.yml`:
- Around line 3-28: Summary: QA list update not required by default for this
2-GPU CI matrix rebalance. If you decide to mirror these 2-GPU placements into
scheduled QA, create a QA list file named qa_2gpu_b200.yml and copy the
condition block (system_gpu_count: gte/lte 2, wildcards gpu '*b200*',
linux_distribution_name ubuntu*, cpu x86_64, terms stage: pre_merge/backend:
pytorch/orchestrator: mpi) and the exact test entries (e.g.,
unittest/_torch/misc/test_autotuner.py::test_autotuner_distributed_strategy,
accuracy/test_llm_api_pytorch.py::TestQwen3_5_35B_A3B::test_bf16[tp2-CUTLASS],
kv_cache/test_kv_cache_v2_scheduler.py::TestKVCacheV2DSv3Lite::test_mtp_draft_tokens,
unittest/_torch/visual_gen/test_flux_pipeline.py::TestFluxParallelism::test_ulysses_2gpu_correctness,
etc.); otherwise leave tests/integration/test_lists/qa/* untouched since
coverage is sufficient for CI-only rebalance.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 10c06e61-eba8-4173-b776-293699bffa4e
📒 Files selected for processing (4)
jenkins/L0_Test.groovytests/integration/test_lists/test-db/l0_b200.ymltests/integration/test_lists/test-db/l0_dgx_b200.ymltests/integration/test_lists/test-db/l0_dgx_h100.yml
|
/bot run |
|
PR_Github #50780 [ run ] triggered by Bot. Commit: |
|
PR_Github #50780 [ run ] completed with state
|
|
/bot run |
|
PR_Github #50933 [ run ] triggered by Bot. Commit: |
|
PR_Github #50933 [ run ] completed with state
|
|
/bot run |
|
PR_Github #51009 [ run ] triggered by Bot. Commit: |
|
PR_Github #51009 [ run ] completed with state
|
|
/bot run |
|
PR_Github #51234 [ run ] triggered by Bot. Commit: |
|
PR_Github #51234 [ run ] completed with state
|
|
/bot run |
|
PR_Github #51291 [ run ] triggered by Bot. Commit: |
|
PR_Github #51291 [ run ] completed with state
|
|
/bot run |
|
PR_Github #51579 [ run ] triggered by Bot. Commit: |
|
PR_Github #51579 [ run ] completed with state
|
|
/bot run |
1 similar comment
|
/bot run |
|
PR_Github #51722 [ run ] triggered by Bot. Commit: |
|
PR_Github #51722 [ run ] completed with state
|
|
/bot run |
|
PR_Github #51784 [ run ] triggered by Bot. Commit: |
|
PR_Github #51784 [ run ] completed with state
|
|
/bot run |
|
PR_Github #51838 [ run ] triggered by Bot. Commit: |
|
PR_Github #51838 [ run ] completed with state
|
|
/bot run |
…n DGX_B200 Source-verified each test's actual GPU requirement, then rebalanced placement so each test runs on a stage whose reserved GPU count matches what it uses. l0_dgx_b200.yml: - Add new 2-GPU pre-merge pytorch/mpi condition for 8 tests previously on a 4-GPU stage but using 2 GPU (test_autotuner_distributed_strategy, two TestQwen3_5_35B_A3B::test_bf16[tp2-*], three TestKVCacheV2DSv3Lite::test_mtp_*, two TestFlux* 2-GPU pipeline tests). - Move TestDeepSeekV32::test_nvfp4_attn_multi_gpus from the 8-GPU post-merge stage to the 4-GPU post-merge stage (test uses tp=4 per @skip_less_mpi_world_size(4)). - Remove test_configurable_moe_single_gpu -k "MEGAMOE_DEEPGEMM" and 9 unconditional 1-GPU visual_gen tests; they now live in their right-sized 1-GPU stage (see below). l0_b200.yml: - Add the single-GPU MEGAMOE_DEEPGEMM row next to the existing test_configurable_moe_single_gpu CUTLASS/TRTLLM/CUTEDSL/DEEPGEMM/DENSEGEMM rows in the 1-GPU pre-merge pytorch condition. - Add 9 visual_gen 1-GPU tests (test_visual_gen_quickstart, test_visual_gen_api_walkthrough, five LPIPS golden tests, two visual_gen_benchmark tests) to the 1-GPU post-merge pytorch condition. All use VisualGenArgs without parallel_config or explicit cfg_size=1 / ulysses_size=1. L0_Test.groovy: - Add DGX_B200-2_GPUs-PyTorch-1 stage to x86SlurmTestConfigs (single split, 2 GPU, dgx-b200-flex pool). Note: 6 conditional visual_gen tests under l0_dgx_b200.yml condition #5 (test_wan_t2v_example, four test_vbench_dimension_score_wan*, two test_vbench_dimension_score_ltx2_*) were considered but kept in place. They call _generate_wan_video / _generate_ltx2_video, which append --cfg_size 2 only when torch.cuda.device_count() >= 2. Moving to a 1-GPU stage would silently drop the cfg_size=2 code path from CI coverage. Note: test_disaggregated_deepseek_v3_lite_fp8_nixl[DeepSeek-V3-Lite-fp8] was considered for the 2-GPU stage but kept on the 4-GPU stage beside its ucx twin. Both load a ctxtp2+gentp2 disaggregated config (context tp2 + generation tp2 = 4 GPUs), not 2. Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
|
/bot run --stage-list "DGX_B200-2_GPUs-PyTorch-1" |
|
PR_Github #51949 [ run ] triggered by Bot. Commit: |
|
PR_Github #51949 [ run ] completed with state |
|
/bot skip --comment "B200 2GPU stage passed" |
|
PR_Github #51996 [ skip ] triggered by Bot. Commit: |
|
PR_Github #51996 [ skip ] completed with state |
…VIDIA#14684) Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
3 participants
Source-verified each test's actual GPU requirement, then rebalanced placement so each test runs on a stage whose reserved GPU count matches what it uses.
l0_dgx_b200.yml:
l0_b200.yml:
l0_dgx_h100.yml:
L0_Test.groovy:
Summary by CodeRabbit
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.