Skip to content

row/QUANT-CT-MXFP4-FUSED-GLUE: W0 — glue-into-Marlin kernel SOURCE-REFUTED; residual is flash-dominant - #67

Closed
localai-bot wants to merge 1 commit into
mainfrom
row/QUANT-CT-MXFP4-FUSED-GLUE
Closed

row/QUANT-CT-MXFP4-FUSED-GLUE: W0 — glue-into-Marlin kernel SOURCE-REFUTED; residual is flash-dominant#67
localai-bot wants to merge 1 commit into
mainfrom
row/QUANT-CT-MXFP4-FUSED-GLUE

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

W0 of the funded MXFP4 glue-fusion kernel campaign

Charter: build a from-scratch Marlin fusing add+RMSNorm+quant into the GEMM prologue and silu+mul into the epilogue, framed as mirroring vLLM's Inductor "GEMM pro/epilogue fusion", to close MXFP4 c2-c8 0.962-0.969 to >=1.0. Vehicle Yi30/Qwen3-8B-MXFP4 (dense W4A16 Marlin).

W0 precondition: dump vLLM's actual kernels, never trust the label. The GPU dumps (TORCH_LOGS=output_code, same-tool decode-window) are box-contended by the active row/H3-FP4-GPU-E2E (both flock locks) and are OWED — but the vLLM compilation source at the parity pin 555967922 is decisive on its own and REFUTES the funded premise.

Findings (source-conclusive)

  1. RMSNormQuantFusionPass / ActivationQuantFusionPass are activation-quant fusions — every registered pattern needs an fp8/nvfp4 quant op (rms_quant_fusion.py:629-669, act_quant_fusion.py:296-320). W4A16 MXFP4 keeps bf16 activations (apply_gptq_marlin_linear quantizes only int8/fp8, marlin_utils.py:704-717) → neither pass fires for this model.
  2. ops.marlin_gemm is a register_fake extern op (_custom_ops.py:1200-1247) Inductor cannot fuse elementwise into. The triton_*_marlin_gemm_* kernels are Inductor-native elementwise fusions named after the adjacent extern call; vLLM's Marlin runs as a separate void marlin::Marlin CUDA kernel (144 launches in same-tool row/KERNEL-MARLIN-DENSE-EXEC: VT_MARLIN_DENSE default ON — all four gates green, c2-c8 to 0.96-0.97 #57).
  3. Same-tool row/KERNEL-MARLIN-DENSE-EXEC: VT_MARLIN_DENSE default ON — all four gates green, c2-c8 to 0.96-0.97 #57 shows vLLM runs more glue launches than us (299 vs 255), not fewer — no glue-into-GEMM collapse to mirror; the +195us glue delta is per-kernel efficiency.
  4. The dominant marlin-tiling term row/QUANT-CT-MXFP4-C8-DIFF: first same-tool c8 node-trace — residual is MARLIN-structural (58%) #52/row/KERNEL-MARLIN-DENSE-EXEC: VT_MARLIN_DENSE default ON — all four gates green, c2-c8 to 0.96-0.97 #57 measured (+1,177us) already closed on main by the dense-direct default efa6e40d (VT_MARLIN_DENSE ON, dense_nvfp4_gemm.h:106-127) — which is why the terminal rose row/QUANT-CT-MXFP4-CLOSERS: byte-exact slivers ship — c1 FULL PARITY; c2-c8 0.925-0.953 #51 0.925-0.953 → row/KERNEL-MARLIN-DENSE-EXEC: VT_MARLIN_DENSE default ON — all four gates green, c2-c8 to 0.96-0.97 #57 0.962-0.969. In that regime the c8 residual is FLASH-dominant: ~+784us flash (71%) / ~+195us glue (18%) / ~+226us residual-marlin.

Verdict

Do NOT build the funded glue Marlin. It mirrors a fusion vLLM does not perform (the beyond-vLLM surpass rung), targets ~18% of the residual, and even glue→0 leaves c8 ~0.977 (<1.0). The real unexhausted lever is the flash +784us term on an identical grid — a STRUCTURAL-lens context/residency audit the FA2 num_splits refutation did not address. A costed surpass-rung proposal is recorded (scoped, not built).

Owed GPU work (box-contended; short + batched when the flock frees)

  • TORCH_LOGS=output_code on the oracle to visually confirm the Triton kernels are elementwise-only over the extern marlin.
  • A fresh same-tool nsys decode-window on current main (dense-direct default) c1..c8 to re-decompose and confirm the flash-dominant split.
  • The flash same-kernel/different-time audit (+784us).

Records-only PR; no code shipped — building the glue Marlin would gold-plate a source-refuted hypothesis. Full forensic entry + per-span table in .agents/benchmark-record.md ("QUANT-CT-MXFP4-FUSED-GLUE W0"). Pre-existing gate reds (check-env-doc VT_DFLASH_*, check-fusion-consistency minimax_h3_video_vae_device) are on main, not introduced here.

🤖 Generated with Claude Code

https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys

@mudler
record(mxfp4): FUSED-GLUE W0 — glue-into-Marlin kernel SOURCE-REFUTED…
1532993 
…, residual is flash-dominant

W0 of the funded MXFP4 glue-fusion kernel campaign. The precondition is to dump
vLLM's actual kernels and never trust the label. GPU dumps are box-contended by
the active H3 GPU campaign (both flock locks) and are OWED, but the vLLM
compilation source at the parity pin 555967922 is decisive and REFUTES the
funded premise (a from-scratch Marlin fusing add+RMSNorm+quant into the prologue
and silu+mul into the epilogue, framed as mirroring vLLM's Inductor GEMM
pro/epilogue fusion):

- RMSNormQuantFusionPass / ActivationQuantFusionPass are ACTIVATION-QUANT fusions:
  every registered pattern needs an fp8/nvfp4 quant op (rms_quant_fusion.py:629-669,
  act_quant_fusion.py:296-320). W4A16 MXFP4 keeps bf16 activations
  (apply_gptq_marlin_linear quantizes only int8/fp8, marlin_utils.py:704-717), so
  NEITHER pass fires for this model.
- ops.marlin_gemm is a register_fake extern op (_custom_ops.py:1200-1247) Inductor
  cannot fuse elementwise into; the triton_*_marlin_gemm_* kernels are
  Inductor-NATIVE elementwise fusions named after the adjacent extern call —
  vLLM's Marlin runs as a SEPARATE void marlin::Marlin CUDA kernel (144 launches
  in the same-tool #57 trace).
- Same-tool #57 already shows vLLM runs MORE glue launches than us (299 vs 255),
  not fewer — no glue-into-GEMM collapse to mirror.
- The dominant marlin-tiling term #52/#57 measured (+1,177us) was already closed on
  main by the dense-direct default efa6e40 (VT_MARLIN_DENSE ON,
  dense_nvfp4_gemm.h:106-127), which is why the terminal rose #51 0.925-0.953 ->
  #57 0.962-0.969. In that regime the c8 residual is FLASH-dominant: ~+784us flash
  (71%) / ~+195us glue (18%) / ~+226us residual-marlin.

VERDICT: do NOT build the funded glue Marlin — it mirrors a fusion vLLM does not
perform (the beyond-vLLM surpass rung), targets ~18% of the residual, and even
glue->0 leaves c8 ~0.977 (<1.0). The real unexhausted lever is the flash +784us
term on an IDENTICAL grid — a STRUCTURAL-lens context/residency audit the FA2
num_splits refutation did not address. Costed surpass-rung proposal recorded (not
built). Owed GPU work (box-contended): TORCH_LOGS=output_code visual confirm, a
fresh same-tool decode-window on current main (dense-direct default) c1..c8, and
the flash same-kernel/different-time audit.

Records-only; no code shipped. Box untouched; H3 campaign uninterrupted.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
@localai-bot

localai-bot commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator Author

Landed on main as the squash (mudler authorship). The flash +784us audit + fresh dense-direct binding follow as the next campaign once the box frees.

🤖 Generated with Claude Code

https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys

mudler added a commit that referenced this pull request Aug 6, 2026
@mudler
record(quant): MXFP4 glue-into-Marlin kernel SOURCE-REFUTED — residua…
307ce6b 
…l is FLASH-dominant (#67)

row/QUANT-CT-MXFP4-FUSED-GLUE squash, records-only W0. The funded
fused-glue Marlin kernel mirrors a fusion vLLM does not perform for
this model: RMSNormQuantFusionPass/ActivationQuantFusionPass are
activation-quant patterns that never fire for W4A16 bf16 activations
(rms_quant_fusion.py:629-669, act_quant_fusion.py:296-320,
marlin_utils.py:704-717); ops.marlin_gemm is a register_fake'd extern
op Inductor cannot decompose (_custom_ops.py:1200-1247); and the
same-tool #57 trace shows vLLM runs MORE glue launches than us (299 vs
255). The prior "Inductor GEMM-epilogue fusion" attribution came from a
cross-tool read. Honest c8 decomposition under the dense-direct default
(+951us marlin already banked by efa6e40): flash +784us (71%) / glue
+195us (18%) / marlin residual +226us — even glue->0 leaves ~0.977.
Public surfaces corrected (STATUS/BENCHMARKS/NOW). The REAL unexhausted
lever: same flash_fwd_splitkv kernel, identical 1x3x64 grid, +22us/call
slower — a STRUCTURAL context/residency audit the num_splits refutation
never addressed. Surpass-rung fused-Marlin scoped + costed
(~150-198us, sub-parity) — not recommended as the parity path. Owed on
the box: fresh same-tool c1-c8 bind on current main + the flash audit.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
@localai-bot localai-bot closed this Aug 6, 2026
@localai-bot localai-bot mentioned this pull request Aug 6, 2026
Closed
mudler added a commit that referenced this pull request Aug 6, 2026
@mudler
perf(flash): -use_fast_math on the vendored FA2 TUs — closes the MXFP…
2e56699 
…4 flash +807us decode term (#68)

QUANT-CT-MXFP4-FLASH-AUDIT closes the #67-OWED flash audit: why the IDENTICAL-grid
flash_fwd_splitkv MXFP4-8B decode kernel runs ~+22us/call slower than the vLLM oracle.

W1 (fresh same-tool reconciliation, analyze_decode.py on the #57 c8 traces): flash MAIN
ours 178.8 vs vLLM 156.3 us/call (+807us/step), combine tied, totals reconcile — REAL,
not a reconciliation artifact.

W2 (cuobjdump/ptxas arbiter, nvcc/ptxas 13.0.88 both sides): kernel-version REFUTED —
vLLM v0.25.0 pins vllm-flash-attn @ 2c839c33, the exact commit we vendored, so flash
SOURCE is byte-identical. The real difference is a BUILD FLAG: vLLM's flash-attn is
compiled --use_fast_math; our vendored FA2 TUs were not. Decode kernel <128,64,128,4>:
ours 5448 instrs/REG246 -> 4832/REG255 with fast-math (= vLLM's 4880/REG255), with
HMMA(768)/LDSM(408)/LDGSTS(120) byte-identical. The +616 scalar instrs (precise
div/rsqrt) = +11.6% ~ the +14.3% per-call time at the latency-bound decode tile. hd256
(27B/35B) also leans 4928->4560. Ruled out: -maxrregcount=255 (5424/246); our own
nvcc-13 compute_80 PTX (5440).

FIX: CMakeLists flash TU COMPILE_OPTIONS += -use_fast_math (mirrors flash-attn's own
build). Real sm_121a build's flash object VERIFIED = 4832 instrs/REG255. Non-byte-exact
(fast-math div/rsqrt) => runtime nsys A/B + SACRED battery OWED (box-contended by the
active H3 campaign); NO default flipped. Full forensics in .agents/benchmark-record.md
(QUANT-CT-MXFP4-FLASH-AUDIT) + spec; STATUS/BENCHMARKS/NOW/state updated.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler added a commit that referenced this pull request Aug 6, 2026
@mudler
record(quant): FLASH-AUDIT #68 — -use_fast_math REJECTED (measured fl…
927b60f 
…ash regression); gap is occupancy/L2, not the SASS

QUANT-CT-MXFP4-FLASH-AUDIT runs the #67-OWED flash audit to a MEASURED verdict.
W1: fresh same-tool c8 decode-window — the flash gap on CURRENT main is +12.5us/call
(+450us/step) ours 168.8 vs vLLM 156.3, SMALLER than #57's +807 (leaner marlin/glue).
W2 lens 1 (cuobjdump HYPOTHESIS): vLLM's flash-attn is --use_fast_math, ours was not
(same 2c839c33 src; +fast-math -> 4832/REG255 = vLLM's 4880/255). W2 lens 2 (MEASURED,
the arbiter): a controlled same-build nsys A/B REFUTES it — -use_fast_math makes flash
+21us/call SLOWER (168.8->189.8); the kernel is memory-latency-bound (ncu occ 8.3%,
register-limited), so the higher reg count (246->255) lowers occupancy, which dominates.
cuobjdump was necessary but NOT sufficient. So the flag is REVERTED (a CMakeLists NOTE
records why); no functional code ships. Real residual = occupancy/L2 (ncu: 38%
smem-scoreboard + 37% barrier stalls); vLLM-side ncu owed (box OOM-rebooted twice under
3-way contention). No default flip (no throughput win). Records + spec updated.

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@localai-bot @mudler