Skip to content

[https://nvbugs/6433809][test] Guard Cutlass MoE TorchBind schema arity - #16405

Closed
nv-yna wants to merge 1 commit into
NVIDIA:mainfrom
nv-yna:nvbug-6433809-cutlass-schema-contract
Closed

[https://nvbugs/6433809][test] Guard Cutlass MoE TorchBind schema arity#16405
nv-yna wants to merge 1 commit into
NVIDIA:mainfrom
nv-yna:nvbug-6433809-cutlass-schema-contract

Conversation

@nv-yna

@nv-yna nv-yna commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Description

NVBug 6433809 exposed a Python/TorchBind ABI regression in the Cutlass MoE path: the rc19/rc20 Python caller supplied 36 positional arguments while the loaded FusedMoeRunner.run_moe schema required 43, so TP4/EP4 WideEP initialization failed at the first omitted argument (_37).

Main already contains the production correction from #15330. This PR adds the missing regression guard:

  • execute the real CutlassMoEOp.compute_moe Python call path with a capture runner;
  • compare the supplied positional arity with the live TorchBind FusedMoeRunner.run_moe schema; and
  • enroll that specific contract test in the H100 pre-merge test database.

This catches future C++ schema extensions that are not reflected in the Python caller without requiring a heavyweight model load.

Test Coverage

  • Exact rc.4 image before the caller correction: contract test failed as expected with supplied 36 arguments ... schema requires 43.
  • Same exact image with the corrected caller mounted: 1 passed.
  • Exact TP4/EP4 WideEP configuration on four GB200 GPUs crossed executor initialization with zero _37 errors and zero failed executor ranks, then registered dynamo.prefill.generate.
  • pre-commit run --files tests/unittest/_torch/modules/moe/test_cutlass_moe_op_smoke.py tests/integration/test_lists/test-db/l0_h100.yml — passed all applicable hooks.
  • Focused Python compilation, Ruff, Ruff format, YAML parsing/selector uniqueness, repository legacy lint, and git diff --check — passed.

The workstation's standalone pytest environment is missing nvtx, so local collection stops in the repository-wide conftest before reaching this test. The exact-image pytest A/B result above exercises the test itself; CI supplies the built extension and runtime dependencies.

PR Checklist

  • I reviewed the checklist in the repository PR template. This change adds no API, dependency, ownership, or architecture-documentation changes.

Summary by CodeRabbit

  • Tests
    • Added coverage for the Cutlass MoE smoke-test suite on H100 environments.
    • Added validation that MoE execution calls remain compatible with the registered TorchBind interface.
    • Improved detection of argument mismatches in CUDA/TensorRT-LLM MoE integration.

Signed-off-by: Yuewei Na <nv-yna@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0703857f-96ee-4336-bfec-57bc1df7d505

📥 Commits

Reviewing files that changed from the base of the PR and between d3436a0 and 30324df.

📒 Files selected for processing (2)
  • tests/integration/test_lists/test-db/l0_h100.yml
  • tests/unittest/_torch/modules/moe/test_cutlass_moe_op_smoke.py

📝 Walkthrough

Walkthrough

Added a CUDA/TensorRT-LLM smoke test that compares FusedMoeRunner.run_moe callsite arity with its TorchBind schema, then registered the test in the H100 pre-merge test list.

Changes

MoE schema validation

Layer / File(s) Summary
Capture and validate MoE schema arity
tests/unittest/_torch/modules/moe/test_cutlass_moe_op_smoke.py
Adds fused-runner stubs, TorchBind schema lookup, and a gated test asserting matching positional argument counts.
Register the H100 smoke test
tests/integration/test_lists/test-db/l0_h100.yml
Adds the new schema-arity smoke test to the H100 pre-merge PyTorch MPI test list.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: brb-nv

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is specific and matches the PR's Cutlass MoE TorchBind schema-arity regression test change.
Description check ✅ Passed The description includes the required Description, Test Coverage, and PR Checklist sections with relevant details.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@yiqingy0
yiqingy0 removed their request for review July 16, 2026 02:22
@brnguyen2

Copy link
Copy Markdown
Collaborator

Non-blocking, out of scope for this PR: the 43-arg FusedMoeRunner.run_moe signature is worth cleaning up as a follow-up. The natural Python approach would be a dataclass bundling the args, but since this is a TorchBind method the type has to be something C++ can receive — a paired C++ struct registered as a custom class, or grouping related args into existing registered types. Either way it would make arity mismatches like the one this PR guards against less likely to slip through in the first place.

@nv-yna nv-yna closed this Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants