Skip to content

[None][doc] Update gpt-oss doc on MoE support matrix #6908

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
litaotju merged 1 commit into from
Aug 15, 2025
Merged

[None][doc] Update gpt-oss doc on MoE support matrix #6908

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 10 additions & 6 deletions examples/models/core/gpt_oss/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,16 @@ GPT-OSS is a reasoning model with MoE weights quantized with mxfp4. All the othe

## MoE Support Matrix

In MoE, the weights are pre-quantized to mxfp4. The activation can be in either bf16 (Hopper) or mxfp8 (Blackwell), with similar accuracy.

| device | Activation | Weight | Supported moe_backend |
|----------|----------|----------|----------|
| Hopper | bf16 | mxfp4 | **TRITON**, CUTLASS |
| Blackwell | mxfp8 | mxfp4 | CUTLASS, TRTLLM |
In MoE, the weights are pre-quantized to mxfp4. The activation can be in either bf16 (Hopper) or mxfp8 (Blackwell), with similar accuracy. FP8 activation with per-tensor scaling factor has limited support. Note that the per-tensor scaling factor needs to be calculated dynamically during inference with the official mxfp4 checkpoints, which may negatively impact perf. The configs in **bold** are the recommended configs for the official checkpoints.

Comment on lines +9 to +10

@coderabbitai coderabbitai Bot Aug 14, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Clarify activation support and avoid slang ("perf").

The intro sentence omits that bf16 activations are also supported on Blackwell (see Line 18), and “perf” is informal. Suggest rewording for accuracy and tone.

Apply this diff:

-In MoE, the weights are pre-quantized to mxfp4. The activation can be in either bf16 (Hopper) or mxfp8 (Blackwell), with similar accuracy. FP8 activation with per-tensor scaling factor has limited support. Note that the per-tensor scaling factor needs to be calculated dynamically during inference with the official mxfp4 checkpoints, which may negatively impact perf. The configs in **bold** are the recommended configs for the official checkpoints.
+In MoE, weights are pre-quantized to mxfp4. Activations are supported in bf16 (Hopper, Blackwell) and mxfp8 (Blackwell) with similar accuracy. Standard FP8 activations with per‑tensor scaling have limited support; with the official mxfp4 checkpoints, per‑tensor scales must be computed dynamically at inference time, which can negatively impact performance. Configurations in **bold** are recommended for the official checkpoints.
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
In MoE, the weights are pre-quantized to mxfp4. The activation can be in either bf16 (Hopper) or mxfp8 (Blackwell), with similar accuracy. FP8 activation with per-tensor scaling factor has limited support. Note that the per-tensor scaling factor needs to be calculated dynamically during inference with the official mxfp4 checkpoints, which may negatively impact perf. The configs in **bold** are the recommended configs for the official checkpoints.
In MoE, weights are pre-quantized to mxfp4. Activations are supported in bf16 (Hopper, Blackwell) and mxfp8 (Blackwell) with similar accuracy. Standard FP8 activations with per‑tensor scaling have limited support; with the official mxfp4 checkpoints, per‑tensor scales must be computed dynamically at inference time, which can negatively impact performance. Configurations in **bold** are recommended for the official checkpoints.
🤖 Prompt for AI Agents 
In examples/models/core/gpt_oss/README.md around lines 9 to 10, update the
paragraph to clarify activation support and remove informal language: state that
activations can be bf16 (supported on Hopper and Blackwell) or mxfp8 (Blackwell)
with similar accuracy, note that FP8 activation with per-tensor scaling factor
has limited support and that the per-tensor scaling factor must be calculated
dynamically during inference with official mxfp4 checkpoints which may
negatively impact performance (replace "perf" with "performance"), and keep the
note that the configs in bold are the recommended configs for the official
checkpoints.

| device | Activation | Weight | Supported moe_backend | MMA|
|----------|----------|----------|----------|----------|
| Hopper | **bf16** | mxfp4 | **TRITON**, CUTLASS | simulated mxfp4, HGMMA |
| Hopper | fp8 | mxfp4 | CUTLASS (not enabled) | simulated mxfp4, QGMMA |
| Blackwell | **mxfp8** | mxfp4 | **CUTLASS, TRTLLM** | UTCQMMA |
| Blackwell | fp8 | mxfp4 | CUTLASS, TRTLLM | UTCQMMA |
| Blackwell | fp8 | mxfp4 | TRITON (experimental) | NA |
| Blackwell | bf16 | mxfp4 | TRTLLM | simulated mxfp4, UTCHMMA |


| moe_backend | TP | EP | AlltoAll |
Expand Down