Emit MatMulNBits accuracy_level=4 (int8 path) — 2.3x CPU decode - #402
Merged
Conversation
MatMulNBits was emitted without accuracy_level, so ORT's MLAS CPU kernel ran the slow fp32 dequant+GEMM path instead of the int8 dynamic-quant + int8 dot-product path (ARM SDOT / x86 VNNI) that makes ORT competitive with llama.cpp. default_int4_accuracy_level=4 already existed on cpu/webgpu EpCapabilities but was dead config. Plumb it via _accuracy_level_attrs() into both MatMulNBits emission sites (Q/K/V/O, MLP, tied + non-tied head); emit when >0, omit at 0 (portable default preserved). Measured (Qwen2.5-0.5B Q4, CPU, decode tok/s, coherent): baseline 39.3 -> accuracy_level=4 91.8 (2.33x); quantized-head + acc4 = 194.7, exceeding LM Studio CPU (157). fp16/bf16 levels regress on M1 (no native GEMM). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bd351e74-2569-444d-a7da-6456f3b5df82
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR plumbs EpCapabilities.default_int4_accuracy_level into emitted com.microsoft::MatMulNBits nodes by adding an _accuracy_level_attrs() helper, enabling ORT’s faster int8 compute path for INT4 weight-only quantized models (notably improving CPU decode throughput).
Changes:
- Add
_accuracy_level_attrs()and apply it to bothMatMulNBitsemission sites (standard quantized linear + tied quantized LM head). - Add unit tests to assert
accuracy_levelis omitted without a build context and emitted as4under a CPU EP build context.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/mobius/components/_quantized_linear.py | Adds EP-context-driven accuracy_level attribute emission for MatMulNBits. |
| src/mobius/components/_quantized_linear_test.py | Adds coverage for attribute presence/absence based on build context / CPU EP. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Member
Author
|
@jambayk could you review? Thanks! |
…e guard Addresses PR #402 review: accuracy_level (from default_int4_accuracy_level) was emitted on ALL MatMulNBits regardless of bits. Its int8-accumulation semantics are INT4-specific, so 2/8-bit models could change behavior or hit undefined paths. _accuracy_level_attrs(bits) now returns {} unless bits==4; both call sites pass self._bits. Added test_cpu_ep_omits_accuracy_level_for_non_int4 (8-bit under CPU ctx omits it) and a for/else fail guard to test_matmulnbits_attributes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: bd351e74-2569-444d-a7da-6456f3b5df82
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause of onnx-genai being ~3.6x slower than llama.cpp on CPU: MatMulNBits was emitted with no
accuracy_level, so ORT's MLAS kernel ran the fp32 dequant+GEMM path instead of the int8 dynamic-quant + int8 dot-product path (ARM SDOT / x86 AVX-VNNI) — the same class of kernel llama.cpp uses. ORT's parity claim holds; we just weren't opting in.default_int4_accuracy_level=4already existed on the cpu/webgpuEpCapabilitiesbut was dead config. This plumbs it via a new_accuracy_level_attrs()helper into bothMatMulNBitsemission sites incomponents/_quantized_linear.py(Q/K/V/O, MLP, tied + non-tied head). Emits when>0, omits at0(portable default preserved).Measured (Qwen2.5-0.5B Q4, CPU EP, decode tok/s, all coherent)
fp16/bf16 levels regress on M1 (no native GEMM). End-to-end verified:
--ep cpustamps all 168 nodes with accuracy_level=4;--ep defaultomits it.lintrunner clean; pytest 518 passed (+2 tests).