[None][test] Add GB300 DISAGG NIXL CI Perf Test Back - #13594
Conversation
📝 WalkthroughWalkthroughThis PR enables GB300 disaggregated performance sanity tests for multi-node configurations (2, 3, 5, and 9 nodes) by activating stage configurations in the Jenkins pipeline, removing UCX environment variable prefixes from pytest command generation, clearing container-injected UCX variables at SLURM script startup, and uncommenting disabled test entries across multiple test list configurations. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Review rate limit: 9/10 reviews remaining, refill in 6 minutes. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@jenkins/scripts/slurm_run.sh`:
- Around line 7-10: The UCX env reset is applied unconditionally; wrap the
unset/export lines in a guard so they only run for the GB300/NIXL disagg
flow—check the environment variable your pipeline uses to indicate that stage
(e.g., GB300_DISAGG, SERVING_MODE, or a stage-specific flag) and only perform
the unset UCX_NET_DEVICES UCX_TLS 2>/dev/null || true and export BASH_ENV= when
that flag indicates the GB300 disagg path; place the current lines inside an if
[ "<flag check>" ]; then ... fi block so unrelated Slurm stages keep their UCX
settings.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 6b36a7e1-390a-4b2f-995e-ea21c9dcc67a
📒 Files selected for processing (7)
jenkins/L0_Test.groovyjenkins/scripts/perf/disaggregated/submit.pyjenkins/scripts/slurm_run.shtests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node1_gpu4.ymltests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node2_gpu8.ymltests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node4_gpu16.ymltests/integration/test_lists/test-db/l0_gb300_multi_nodes_perf_sanity_ctx1_node1_gpu4_gen1_node8_gpu32.yml
6e9f3c6 to
a469160
Compare
|
/bot run --disable-fail-fast --stage-list "GB300Disagg-PerfSanity" |
|
PR_Github #46061 [ run ] triggered by Bot. Commit: |
|
PR_Github #46061 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB300*Disagg-PerfSanity*" |
|
PR_Github #46128 [ run ] triggered by Bot. Commit: |
|
PR_Github #46128 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB300Disagg-PerfSanity" |
|
PR_Github #46298 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast --stage-list "*GB300*Disagg-PerfSanity*" |
|
PR_Github #46314 [ run ] triggered by Bot. Commit: |
|
PR_Github #46298 [ run ] completed with state |
|
PR_Github #46314 [ run ] completed with state
|
5cdef01 to
e4ff569
Compare
|
/bot run --disable-fail-fast --stage-list "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1" |
|
PR_Github #47705 [ run ] triggered by Bot. Commit: |
|
PR_Github #47705 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1" |
|
PR_Github #47753 [ run ] triggered by Bot. Commit: |
|
PR_Github #47753 [ run ] completed with state
|
67151ae to
4909c22
Compare
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
|
/bot run --disable-fail-fast --stage-list "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1" |
|
PR_Github #47953 [ run ] triggered by Bot. Commit: |
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
|
/bot run --disable-fail-fast --stage-list "GB300-4_GPUs-PyTorch-PerfSanity-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-PerfSanity-Node2-GPU8-Post-Merge-1,GB300-8_GPUs-2_Nodes-PyTorch-Disagg-PerfSanity-CTX1-NODE1-GPU4-GEN1-NODE1-GPU4-Post-Merge-1" |
|
PR_Github #47956 [ run ] triggered by Bot. Commit: |
|
/bot run --disable-fail-fast --stage-list "GB300-*" |
|
PR_Github #47982 [ run ] triggered by Bot. Commit: |
|
/bot skip --comment "Only add new GB300 perf tests, no need to run the whole CI pipeline" |
|
PR_Github #48106 [ skip ] triggered by Bot. Commit: |
|
PR_Github #48106 [ skip ] completed with state |
Signed-off-by: Chenfei Zhang <chenfeiz@nvidia.com>
3 participants
Summary by CodeRabbit
Tests
Bug Fixes
Description
Test Coverage
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.