Skip to content

[https://nvbugs/6221055][fix] added four GB300 rows to agg_unit_mem_df.csv - #14785

Merged
QiJune merged 6 commits into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6221055
Jul 10, 2026
Merged

[https://nvbugs/6221055][fix] added four GB300 rows to agg_unit_mem_df.csv#14785
QiJune merged 6 commits into
NVIDIA:mainfrom
tensorrt-cicd:repair-bot-bug6221055

Conversation

@tensorrt-cicd

@tensorrt-cicd tensorrt-cicd commented May 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Root cause: agg_unit_mem_df.csv has no row for GB300 device name NVIDIA B300 SXM6 AC, so test_unittests_v2 falls back to serial (num_workers=1) and the 5011-case unittest/_torch/thop/parallel suite cannot fit in the 4h Slurm budget.
  • Fix: Added four NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch cases scheduled on GB300 stages (thop/parallel=16, thop/serial=4, attention=4, executor=4), matching B200/H100 values; test_unittests_v2 already caps num_workers at 8.
  • Automated fix generated by repair-bot

Test plan

  • Verify fix on the same GPU type as the original failure
  • Check for regressions in related tests

Links

Summary by CodeRabbit

  • Tests
    • Removed a waiver for the full:GB300/unittest/_torch/thop/parallel test, allowing it to run as part of the test suite.

The GB300-4_GPUs-PyTorch-Post-Merge-1 stage was timing out at the 4h
Slurm budget because test_unittests_v2 fell back to num_workers=1 on
GB300 nodes. The fallback was triggered by a missing lookup row in
tests/integration/defs/agg_unit_mem_df.csv: pynvml reports the GB300
device name as 'NVIDIA B300 SXM6 AC' but the CSV only had entries for
B200 / H100 / B200-bring-up. With xdist disabled, the 5,011-case
unittest/_torch/thop/parallel suite alone needs ~3h and the stage
cannot finish in time.

Add 'NVIDIA B300 SXM6 AC' rows for the unittest/_torch cases scheduled
on GB300 stages (l0_gb300.yml, l0_gb300_multi_gpus.yml), mirroring the
parallel_factor values already used for B200/H100. test_unittests_v2
clamps num_workers to 8, so 16 for thop/parallel matches the other
Blackwell stages.

Signed-off-by: tensorrt-cicd <90828364+tensorrt-cicd@users.noreply.github.com>
QiJune added 2 commits July 10, 2026 11:49
Signed-off-by: QI JUN <22017000+QiJune@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Removed the full:GB300/unittest/_torch/thop/parallel waiver entry from the integration test waiver list.

Changes

Waiver list update

Layer / File(s) Summary
Remove GB300 test waiver
tests/integration/test_lists/waives.txt
Removes the waiver for the GB300 thop parallel integration test.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

Suggested reviewers: dongxuy04, mzweilz, yuxianq, jiaganc

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning It includes the main details, but it does not follow the required template sections or include the PR checklist. Reformat it to the template with @coderabbitai summary, Description, Test Coverage, and PR Checklist sections.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is specific, concise, and aligned with the main change.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@QiJune QiJune changed the title [https://nvbugs/6221055][fix] Added four NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch [https://nvbugs/6221055][fix] added four NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch Jul 10, 2026
@QiJune QiJune changed the title [https://nvbugs/6221055][fix] added four NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv for the _torch [https://nvbugs/6221055][fix] added four NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv Jul 10, 2026
@QiJune

QiJune commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

/bot run --help

@QiJune

QiJune commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

/bot run --stage-list "GB300-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58585 [ run ] triggered by Bot. Commit: f8b9ca9 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58584 Bot args parsing error: usage: /bot [-h]
{run,kill,skip,submit,reviewers,reuse-pipeline,reuse-review} ...
/bot: error: unrecognized arguments: --help

Link to invocation

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
@QiJune

QiJune commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

/bot run --stage-list "GB300-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58612 [ run ] triggered by Bot. Commit: c89e103 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58585 [ run ] completed with state ABORTED. Commit: f8b9ca9

Link to invocation

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
@QiJune QiJune changed the title [https://nvbugs/6221055][fix] added four NVIDIA B300 SXM6 AC rows to agg_unit_mem_df.csv [https://nvbugs/6221055][fix] added four GB300 rows to agg_unit_mem_df.csv Jul 10, 2026
@QiJune

QiJune commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

/bot run --stage-list "GB300-PyTorch-1"

Signed-off-by: junq <22017000+QiJune@users.noreply.github.com>
@QiJune

QiJune commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

/bot run --stage-list "GB300-PyTorch-1"

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58634 [ run ] triggered by Bot. Commit: ebf9a26 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58635 [ run ] triggered by Bot. Commit: ebf9a26 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58634 [ run ] completed with state ABORTED. Commit: ebf9a26

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58612 [ run ] completed with state ABORTED. Commit: c89e103

Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58635 [ run ] completed with state SUCCESS. Commit: ebf9a26
/LLM/main/L0_MergeRequest_PR pipeline #47223 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

@QiJune

QiJune commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

/bot skip --comment "GB300 stage passed"

@QiJune
QiJune enabled auto-merge (squash) July 10, 2026 09:48
@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58655 [ skip ] triggered by Bot. Commit: ebf9a26 Link to invocation

@tensorrt-cicd

Copy link
Copy Markdown
Collaborator Author

PR_Github #58655 [ skip ] completed with state SUCCESS. Commit: ebf9a26
Skipping testing for commit ebf9a26

Link to invocation

@QiJune
QiJune merged commit aedf23c into NVIDIA:main Jul 10, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants