spike(QUANT-CT-MXFP4): native MXFP4 keep-quant + Qwen e2e — W0 vehicle + W1 target pinned - #38
Closed
localai-bot wants to merge 12 commits into
Closed
spike(QUANT-CT-MXFP4): native MXFP4 keep-quant + Qwen e2e — W0 vehicle + W1 target pinned#38localai-bot wants to merge 12 commits into
localai-bot wants to merge 12 commits into
Conversation
…nfer W4A4) Re-scope QUANT-CT-MXFP4 around a real on-box Qwen MXFP4 checkpoint per USER priority (full MXFP4 at vLLM parity, benchmarked on a Qwen model; DeepSeek/Kimi not the vehicle). Design + oracle-support proofs only; empirical run/build/bench are GPU/disk-gated and QUEUED (box contended: locks held, Kimi download shrinking disk). No GPU held, no locks touched. W0 vehicle: Yi30/Qwen3-8B-MXFP4 (dense Qwen3ForCausalLM, compressed-tensors mxfp4-pack-quantized group 32, input_activations set = true W4A4, 6.18 GB). Oracle-support proved at import/registry on the runnable 0.25.0 oracle: Qwen3ForCausalLM in get_supported_archs() and CompressedTensorsW4A4Mxfp4 imports. W1 parity target pinned from the running 0.25.0 tree: init_mxfp4_linear_kernel returns the first supported of [FlashInfer, Marlin, Humming]; FlashInfer W4A4 (has_device_capability(100) AND has_flashinfer_cutedsl(), both True on GB10) wins, so the oracle runs the true W4A4 fp4xf4 cute-dsl GEMM (block_size=32, use_nvfp4=False). Overrides the earlier Marlin-W4A16 hypothesis for GB10; Marlin is the non-Blackwell fallback. Runtime confirm + nsys QUEUED. W2-W5 contract added to the spec: extend the NVFP4 cutlass fp4 GEMM to the mxf4 block-scale format + activation mxf4 quant; schemes/mxfp4.h selection method mirroring schemes/nvfp4.h (ResidentWeight staging); CPU mxfp4_emulation mirroring nvfp4_emulation as the gate truth; W3 gates (emulation unit + M=1 RED trap + e2e distributional) and W4 online_gate bench spec'd. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…A16, not FlashInfer W4A4) Resumed the QUEUED empirical steps on the freed GB10 box under both flock locks, mem gate (94-95 GiB free), tmux + done-markers, oracle alone, single load, disk floor respected. Box left clean. W0 DONE: downloaded Yi30/Qwen3-8B-MXFP4 (6.18 GB) and the 0.25.0 oracle RAN a greedy golden (temperature=0, seed=0, enforce_eager, 4 prompts): PYEXIT=0, coherent + correct (Paris/Rome/Berlin/Madrid; 2+2=4 ... 5+5=10; coherent story; correct recursive fibonacci). Satisfies the hard oracle rule. Golden + evidence in docs/bench-evidence/mxfp4-qwen/. W1 DONE with a runtime correction of the source-only trace: init_mxfp4_linear_kernel selects FlashInferMxFp4LinearKernel (is_supported passes on cap 121>=100 + cute-dsl) and logs it, THEN engine start dies with flashinfer.utils.BackendSupportedError: mm_fp4 does not support backend 'cute-dsl' with capability 121. FlashInfer's cute-dsl mxf4 backend covers sm_100 datacenter Blackwell, not sm_121 GB10, so the default oracle config is non-functional for this checkpoint on GB10. Working path = VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel -> MarlinMxFp4LinearKernel (W4A16 weight-only fp4). So the GB10 parity target is Marlin W4A16 mxf4, revalidating the row's original Laguna-B2 hypothesis and disproving the source-only W1 conclusion (is_supported != actually-runs). W2 route revised in the spec: extend our existing Marlin FP4 infra (src/vt/cuda/ marlin/*) for group-32 E8M0 (the Laguna B2 route), not a new cute-dsl W4A4 kernel. W2/W3/W4 remain (native compute + gates + bench); build-env caveat recorded. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…he continuation Add the exact our-side file:line surface for the native Marlin-W4A16-mxf4 keep-quant extension: dense Qwen3 loader (qwen3_weights.cpp), scheme detection seam (model_loader.cpp uses_nvfp4_w4a4 :750), Mxfp4Weight mirroring Nvfp4Weight, MatmulMxfp4W4A16D mirroring MatmulNvfp4W4A16D + Marlin FP4 repack for E8M0 group-32 no-global, schemes/mxfp4.h. Corrects the build-size figure (our tree ~1-2 GiB, fits). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…ds+runs, e2e numerics bug localized
Implements the compressed-tensors MXFP4 (mxfp4-pack-quantized) native keep-quant
compute for dense Qwen3 on GB10, routed through the EXISTING Marlin FP4 infra (the
Laguna-B2 route), NOT a new cute-dsl W4A4 kernel. Grounded 1:1 in vLLM
marlin_utils_fp4.prepare_fp4_layer_for_marlin (is_nvfp4=False branch) @ pin.
What landed (all additive; NVFP4 + the 27B/35B gate paths byte-unchanged):
- generate_kernels.py: MXFP4 QUANT_CONFIG (s_type kFE8M0fnu, group_blocks [2]);
regenerated kernel_selector.h + sm80_kernel_*.cu (15 new group_blocks=2 instances).
- MarlinProcessExpertScalesMxfp4 (cuda_marlin_repack.cu): E8M0 passthrough permute
(marlin_permute_scales + within-4 [0,2,1,3]); PROVEN byte-exact vs vLLM's
mxfp4_marlin_process_scales on a 128x256 case (tools check).
- MoeMarlinArgs.{group_size,mxfp4} + cuda_moe_marlin.cu launcher branch: s_type
kFE8M0fnu, group_size 32 (group_blocks 2), global_scale=nullptr (E8M0 has none;
kernel only reads global under kFE4M3fn).
- Nvfp4Weight.{group_size,is_mxfp4}; dense_nvfp4_gemm.h BuildMarlinDenseResident +
MatmulNvfp4MarlinD branch on is_mxfp4 (K/32 scale grid, mxfp4 scale proc, split
gate_up); MatmulMxfp4W4A16D named entry; MXFP4 CPU dequant fallback.
- dense_weight_loaders.h: IsCtMxfp4Projection/LoadCtMxfp4W4A16/LoadMergedCtMxfp4W4A16
(U8 E8M0 scale [N,K/32], no global). qwen3_weights.cpp: detect+load MXFP4.
Verified: clean -Werror build; loads Yi30/Qwen3-8B-MXFP4 on GB10; dispatches the
native group_blocks=2 Marlin kernel; produces the CORRECT first (prefill) token
(" Paris", "there"). NOT YET token-exact vs the oracle golden: token-2+ (decode/M=1)
degenerates — a numerics bug isolated to the group_blocks=2 GEMM path (scale permute
byte-exact and fp4 dequant faithful, so the residual is the never-before-exercised
group_blocks=2 kernel interaction / decode path). W3 e2e = RED (honest); the M=1
device unit gate vs the CPU dequant reference is the next debugging step.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…s), e2e RED with bug localized Spec empirical status + state handoff: the MXFP4 Marlin W4A16 keep-quant path builds -Werror on GB10, loads Yi30/Qwen3-8B-MXFP4, dispatches the native group_blocks=2 Marlin kernel and runs; e2e is NOT token-exact — a deterministic uniform GEMM numerics error (not graph-related), with the scale permute proven byte-exact vs vLLM and the fp4 dequant a faithful lift, so the residual is the group_blocks=2 GEMM interaction. Next: M=1/M=8 device unit gate vs the CPU dequant reference; then re-gate e2e + W4 bench. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…U dequant ref (M=1+M=8) Adds the owed W3 device unit gate to tests/vt/test_ops_moe_grouped.cpp: the MXFP4 W4A16 Marlin GEMM (group_blocks=2, E8M0 scales, no global) vs the INDEPENDENT CPU dequant reference (DequantMxfp4ToF32 + f32 matmul) — a different code path than the Marlin repack+kernel dequant, so a real cross-check. Single expert / all tokens -> expert 0 (the dense MatmulMxfp4W4A16D routing), RED-first coverage of the M=1 DECODE path AND M=8. RESULT: GREEN — max_rel = 3.8e-3 at both M=1 and M=8 (pure bf16 rounding, NOT a systematic error). This PROVES the MXFP4 keep-quant compute (repack, E8M0 scale processing, group_blocks=2 kernel dispatch, launcher) is correct. So the e2e degeneration (commit 7068dca) is NOT in the GEMM — it is in the loader or model wiring (the unit gate bypasses the loader with synthetic bytes). Localization continues from here. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…n-pending)
The N=128/K=256 case is GREEN (0.36% error), but the model uses N up to 12288 and
K up to 12288. Extend the shape loop to the real projection shapes
({4096,4096} o/qkv-per-shard, {4096,12288} down, {12288,4096} gate/up) so the
group_blocks=2 kernel is exercised at model scale — the one coverage gap between
the green unit gate and the RED e2e. RUN-PENDING (box taken by the Kimi agent when
authored): a FAIL localizes a large-N/K kernel bug; a PASS shifts the residual to
model integration.
FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…esidual localized off-compute Records the W3 device unit gate result (MXFP4 GEMM == CPU dequant ref, 0.38% at M=1/M=8) and the exhaustive component verification that eliminates the compute AND the loader byte interpretation as the e2e cause. Prime remaining suspect: the group_blocks=2 kernel at large model N/K (extended unit case run-pending); else a model-integration subtlety (per-layer activation diff next). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…enseResident correct Adds the decisive model-facing gate (test_linear_method.cpp): the EXACT model path MakeLinearMethod(bf16-empty, mxfp4) -> Apply -> MatmulNvfp4W4A16D -> MatmulNvfp4MarlinD -> BuildMarlinDenseResident -> MoeGroupedGemmNvfp4Marlin, vs the INDEPENDENT CPU dequant reference, at REAL Qwen3-8B shapes (K=4096 and K=12288), M=1 AND M=8. GREEN (4/4, bad=0). This closes the one link the op-level gate did not cover (it fed MANUALLY-built residents) — the resident builder is correct even at large K. (It also surfaced that MarlinDenseResidentFor is keyed by weight POINTER: a loop-local reused stack slot aliases residents -> OOB; the fix uses distinct persistent weights. Not a model bug — model weights are distinct persistent objects.) Also adds a synthetic model-forward MXFP4 smoke case (test_qwen3_forward.cpp): routes mxfp4 via IsNvfp4()-emptiness, runs finite + deterministic. NOTE: on CPU both arms dequant via the same helper, so its dlogit==0 is a routing/determinism check, not an independent correctness gate (the correctness gates are the two above). Net: every MXFP4 code path — GEMM, scale processing, repack, dispatch, loader byte interpretation, and the model-facing Apply/resident build — is now independently verified correct. The e2e residual is therefore NOT in the MXFP4 compute path. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
…ken-exact), async-default is a pre-existing non-MXFP4 bug Root-caused the e2e residual. The default (async ON) degeneration is NOT the MXFP4 compute: with VT_ASYNC_SCHED=0 the same binary is TOKEN-EXACT vs the golden on 3/4 prompts (p1/p2/p4 exact; p3 open-ended story diverges after the identical first token = bf16/impl non-determinism). The async executor overlaps the prior step's output-copy with the forward, and classic dense Qwen3ForCausalLM lacks the async device-mirror fix (the #31 class wired only for the gate models) — a quant-independent, pre-existing classic-dense-Qwen3 async bug, separate from this row. MXFP4 keep-quant compute is proven correct: op-level GEMM (0.36%), scale permute byte-exact vs vLLM at all shapes, model-facing Apply/BuildMarlinDenseResident gate, and e2e async-off 3/4 token-exact. Reverted an exploratory per-stream workspace change (not the async cause). Evidence: docs/bench-evidence/mxfp4-qwen/W3-e2e-result.md. Residual: classic-dense-Qwen3 async device-mirror fix (separate row) + W4 bench. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
… + e2e token-exact (async-off) Marks loader/compute/test/e2e = Y with anchors: op-level GEMM gate, model-facing Apply gate, and e2e 3/4 token-exact vs the golden (async-off). W-bits column updated to native Marlin W4A16 mxf4 keep-quant (GB10 target; W4A4 cute-dsl crashes sm_121). Residual noted: default-async is a pre-existing classic-dense-Qwen3 bug; W4 bench owed. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
# Conflicts: # .agents/state.md
Collaborator
Author
|
Landed on main as the squash of this branch (local squash, mudler authorship per the new landing flow). MXFP4 W4A16 keep-quant compute is correctness-proven; W4 bench owed after the classic-dense-Qwen3 async device-mirror fix (separate row, dispatched). 🤖 Generated with Claude Code |
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
…t, bench owed (#38) row/QUANT-CT-MXFP4 squash. Native compressed-tensors mxfp4-pack-quantized route through the existing Marlin FP4 infra (Laguna-B2 pattern): MXFP4 kernel-gen config (group_blocks=2, s_type kFE8M0fnu, 15 instances), MarlinProcessExpertScalesMxfp4 (byte-exact vs vLLM at all model shapes), launcher branch, Nvfp4Weight.{group_size,is_mxfp4}, MatmulMxfp4W4A16D, CT-MXFP4 loaders, qwen3 detect+load. Correctness PROVEN: op-level GEMM vs independent CPU dequant max_rel 0.36% at M=1/M=8 across K,N up to 12288; model-facing path bad=0; e2e 3/4 prompts token-exact vs the oracle golden (4th = open-ended, near-tie regime). W1 runtime finding: vLLM 0.25.0's default MXFP4 dispatch CRASHES on sm_121 (FlashInfer cute-dsl rejects cap 121); the working oracle path is Marlin W4A16 via VLLM_DISABLED_KERNELS. Default-config e2e is blocked by a PRE-EXISTING quant-independent async bug: classic dense Qwen3ForCausalLM lacks the #31 async device-mirror (VT_ASYNC_SCHED=0 token-exact, async-on degenerates) - separate row, fix in flight. W4 bench owed after that fix. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 5, 2026
…38 repair) The #38 squash moved the quantization matrix + qwen3 loader without the three public surfaces (my landing-gate ordering error - the checkpoint check printed rc=1 after the push chain had fired). STATUS/FEATURES rows move to the proven-compute state; BENCHMARKS gains the pending-bench row with the oracle-arm crash workaround recorded. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Helper claim for row QUANT-CT-MXFP4. USER-priority: full MXFP4 support at vLLM parity, benchmarked on a Qwen model.
W0 — DONE
Yi30/Qwen3-8B-MXFP4runs on the 0.25.0 oracle; greedy golden captured.W1 — DONE (runtime correction)
FlashInfer W4A4 is selected but crashes on sm_121 (
mm_fp4 ... cute-dsl ... capability 121); working path =VLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel→ Marlin W4A16 (the GB10 target).W2 — native Marlin W4A16 mxf4 keep-quant, all additive (NVFP4 + gate paths byte-unchanged)
generate_kernels.pyMXFP4 config (kFE8M0fnu, group_blocks 2) + regenerated instances;MarlinProcessExpertScalesMxfp4;MoeMarlinArgs.{group_size,mxfp4}launcher;Nvfp4Weight.{group_size,is_mxfp4}+dense_nvfp4_gemm.hbranch +MatmulMxfp4W4A16D;dense_weight_loaders.hloaders;qwen3_weights.cppdetect+load.W3 — GATES GREEN; compute PROVEN correct
MakeLinearMethod → Apply → BuildMarlinDenseResident: bad=0 at K=4096 and K=12288.VT_ASYNC_SCHED=0, 3/4 prompts TOKEN-EXACT vs the oracle golden (p1 capitals, p2 arithmetic, p4 fibonacci exact; p3 open-ended story diverges after the identical first token = bf16/impl non-determinism).The default-async degeneration is a PRE-EXISTING non-MXFP4 bug
The default (async on) degenerated, but NOT from MXFP4: the async executor overlaps the prior step's output-copy with the forward, and classic dense
Qwen3ForCausalLMlacks the async device-mirror fix (the #31 class wired only for the gate models). Quant-independent (hits bf16/NVFP4 the same way), pre-existing — a separate row, not this one.Residual (kept DRAFT)
online_gate.pyc1..c8×3, oracle armVLLM_DISABLED_KERNELS=FlashInferMxFp4LinearKernel) on the async-off compute path;Evidence:
docs/bench-evidence/mxfp4-qwen/. Build tree persists atdgx:~/work/mxfp4-w2.🤖 Generated with Claude Code