Skip to content

Skip MXFP8 MFSDP tests on hopper - #3163

Merged
vthumbe1503 merged 2 commits into
NVIDIA:mainfrom
vthumbe1503:mcore_fsdp_bug
Jul 3, 2026
Merged

Skip MXFP8 MFSDP tests on hopper#3163
vthumbe1503 merged 2 commits into
NVIDIA:mainfrom
vthumbe1503:mcore_fsdp_bug

Conversation

@vthumbe1503

Copy link
Copy Markdown
Collaborator

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

Signed-off-by: Varun Thumbe <vthumbe@nvidia.com>
@vthumbe1503
vthumbe1503 marked this pull request as ready for review July 1, 2026 22:48
@greptile-apps

greptile-apps Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds an early-exit guard to qa/L1_pytorch_mcore_fsdp_integration/test.sh that skips the entire MXFP8 MFSDP test suite on hardware below Blackwell (compute capability < 10.0), since --fp8-recipe mxfp8 is unsupported on Hopper and older.

  • Queries nvidia-smi for the raw compute capability, strips the decimal with sed, and compares the resulting integer against 100 (representing 10.0), exiting cleanly with an informative message when the check fails.
  • The guard handles missing nvidia-smi gracefully: an empty DEVICE_ARCH also triggers the skip, printing compute_cap=unknown.

Confidence Score: 5/5

The change is a small, self-contained guard that only adds a skip-path; it cannot affect test execution on Blackwell or cause regressions on passing hardware.

The guard logic is correct for all known GPU generations: decimal stripping and integer comparison handle Hopper (9.0→90), Ada (8.9→89), and Blackwell (10.0→100) properly. The one improvement noted (checking the minimum across all GPUs rather than just GPU 0) matters only on heterogeneous nodes, which are uncommon in CI, so the change is safe to merge as-is.

No files require special attention; the single changed file is a QA shell script with straightforward logic.

Important Files Changed

Filename Overview
qa/L1_pytorch_mcore_fsdp_integration/test.sh Adds a Blackwell-only guard using nvidia-smi compute_cap; correctly strips the decimal and compares as integer, skips with informative message when below 10.0. One minor concern: only the first GPU's capability is checked while the test runs across all GPUs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[test.sh starts] --> B[nvidia-smi query compute_cap of GPU 0]
    B --> C[Strip non-digits: e.g. '9.0' → '90', '10.0' → '100']
    C --> D{DEVICE_ARCH empty\nor < 100?}
    D -- Yes --> E[Print skip message\nwith raw compute_cap\nexit 0]
    D -- No --> F[Continue: setup Megatron-LM\nclone, vocab, env vars]
    F --> G[Launch distributed GPT\npretrain with --fp8-recipe mxfp8]
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[test.sh starts] --> B[nvidia-smi query compute_cap of GPU 0]
    B --> C[Strip non-digits: e.g. '9.0' → '90', '10.0' → '100']
    C --> D{DEVICE_ARCH empty\nor < 100?}
    D -- Yes --> E[Print skip message\nwith raw compute_cap\nexit 0]
    D -- No --> F[Continue: setup Megatron-LM\nclone, vocab, env vars]
    F --> G[Launch distributed GPT\npretrain with --fp8-recipe mxfp8]
Loading

Reviews (2): Last reviewed commit: "Update qa/L1_pytorch_mcore_fsdp_integrat..." | Re-trigger Greptile

Comment thread qa/L1_pytorch_mcore_fsdp_integration/test.sh Outdated
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Signed-off-by: vthumbe1503 <vthumbe@nvidia.com>

@cspades cspades left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I didn't add any device checks to this test.

@ptrendx ptrendx added 2.18 and removed 2.17 labels Jul 2, 2026
@ptrendx

ptrendx commented Jul 2, 2026

Copy link
Copy Markdown
Member

/te-ci L1

@timmoon10 timmoon10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vthumbe1503
vthumbe1503 merged commit dc57958 into NVIDIA:main Jul 3, 2026
47 of 55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants