[https://nvbugs/6221055][fix] added four GB300 rows to agg_unit_mem_df.csv - #14785
Conversation
The GB300-4_GPUs-PyTorch-Post-Merge-1 stage was timing out at the 4h Slurm budget because test_unittests_v2 fell back to num_workers=1 on GB300 nodes. The fallback was triggered by a missing lookup row in tests/integration/defs/agg_unit_mem_df.csv: pynvml reports the GB300 device name as 'NVIDIA B300 SXM6 AC' but the CSV only had entries for B200 / H100 / B200-bring-up. With xdist disabled, the 5,011-case unittest/_torch/thop/parallel suite alone needs ~3h and the stage cannot finish in time. Add 'NVIDIA B300 SXM6 AC' rows for the unittest/_torch cases scheduled on GB300 stages (l0_gb300.yml, l0_gb300_multi_gpus.yml), mirroring the parallel_factor values already used for B200/H100. test_unittests_v2 clamps num_workers to 8, so 16 for thop/parallel matches the other Blackwell stages. Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
📝 WalkthroughWalkthroughRemoved the ChangesWaiver list update
Estimated code review effort: 1 (Trivial) | ~2 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch
NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch |
/bot run --help |
|
/bot run --stage-list "GB300-PyTorch-1" |
|
PR_Github #58585 [ run ] triggered by Bot. Commit: |
|
PR_Github #58584 Bot args parsing error: usage: /bot [-h] |
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
|
/bot run --stage-list "GB300-PyTorch-1" |
|
PR_Github #58612 [ run ] triggered by Bot. Commit: |
|
PR_Github #58585 [ run ] completed with state |
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
|
/bot run --stage-list "GB300-PyTorch-1" |
Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
|
/bot run --stage-list "GB300-PyTorch-1" |
|
PR_Github #58634 [ run ] triggered by Bot. Commit: |
|
PR_Github #58635 [ run ] triggered by Bot. Commit: |
|
PR_Github #58634 [ run ] completed with state |
|
PR_Github #58612 [ run ] completed with state |
|
PR_Github #58635 [ run ] completed with state |
|
/bot skip --comment "GB300 stage passed" |
|
PR_Github #58655 [ skip ] triggered by Bot. Commit: |
|
PR_Github #58655 [ skip ] completed with state |
Summary
NVIDIA B300 SXM6 AC, so test_unittests_v2 falls back to serial (num_workers=1) and the 5011-case unittest/_torch/thop/parallel suite cannot fit in the 4h Slurm budget.NVIDIA B300 SXM6 ACrows toagg_unit_mem_df.csvfor the_torchcases scheduled on GB300 stages (thop/parallel=16, thop/serial=4, attention=4, executor=4), matching B200/H100 values;test_unittests_v2already capsnum_workersat 8.Test plan
Links
Summary by CodeRabbit
full:GB300/unittest/_torch/thop/paralleltest, allowing it to run as part of the test suite.