From 90d9c71d29cd6ae4cc9bae3dd6556b88c1d4023f Mon Sep 17 00:00:00 2001
From: Ettore Di Giacinto <mudler@localai.io>
Date: Thu, 6 Aug 2026 04:31:20 +0000
Subject: [PATCH] =?UTF-8?q?bench(quant):=20dense-direct-A=20marlin=20ARBIT?=
 =?UTF-8?q?ER=20=E2=80=94=20grouped=E2=86=92dense=20port=20is=20a=20measur?=
 =?UTF-8?q?ed=20NO-GO=20(#50)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Arbitrated the named MXFP4 residual-#1 lever (route E=1 dense projections
through vLLM's DENSE marlin_gemm direct-A / m_block_size_8 instead of our
MoeGroupedGemmNvfp4Marlin E=1) with the same-tool per-shape ubench the lever
protocol mandates BEFORE a large port — did NOT vendor the ~2000-line dense
kernel.

Re-ran dgx:~/mxfp4-nsys/mxfp4_marlin_ubench.py (reference apply_fp4_marlin_linear
= the exact kernel a port would vendor, vs ops.moe_wna16_marlin_gemm E=1 = the
1:1 twin of our production kernel) 3x fresh on the idle GB10; with #46 = 4
same-tool runs. Per-STEP moe/dense mean: M=1 0.997, M=2 1.007, M=4 1.010 (all
noise, sign of delta flips run-to-run) and M=8 1.039 (the only reproducible
signal: m_block_size_8 8-row tile vs our moe_block_size=16, ~+0.33ms/step =
~0.8% of a c8 step). The +7-9% cross-tool figure (#47/#48) is the
nsys-vs-torch-profiler tool bias, NOT a recoverable kernel/dispatch cost.

VERDICT: NO-GO. The port cannot reach >=1.0x on any axis (recovers ~0.8pp of
the 5.8% c8 gap at M=8 only, nothing at c1-c4); vendoring 2000 lines of delicate
dense marlin + a NON-byte-exact reroute of the CUDA-graph-captured decode for
that would gold-plate a measured near-dead hypothesis (#46 caution).
Recoverable slivers named (all cheaper than the port): force moe_block_size=8
at M=8 in DenseAlignFor (engages our already-vendored m_block_size_8 kernels,
~5 lines, no new vendoring); drop the per-call defensive workspace memset; the
~0.7ms/step host/sched slice. Box left clean (both locks free, worker down,
disk 21G).

FOLLOWING_AGENTS_PROTOCOL
Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
---
 .agents/benchmark-record.md | 64 +++++++++++++++++++++++++++++++++++++
 1 file changed, 64 insertions(+)

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
index 0083fcd95..b62a8b2a2 100644
--- a/.agents/benchmark-record.md
+++ b/.agents/benchmark-record.md
@@ -12490,3 +12490,67 @@ uses indirect `sorted_token_ids` gather + fp32 `C_tmp` reduce vs vLLM's dense `m
 direct-A addressing; per-shape parity was shown at M≤8 (#46 microbench), so this is a delicate
 grouped→dense-direct-A port, not a config knob. (2) **~0.7ms/step host/sched slice**. No single
 lever reaches ≥1.0x. NEXT lever candidate: the grouped→dense-direct-A marlin decode path.
+
+## KERNEL-MARLIN-DENSE-DIRECT (#50): dense-direct-A marlin ARBITER re-run (4 same-tool runs) — the +7-9% is NOT a recoverable kernel/dispatch cost; port is a measured NO-GO (only M=8 shows a real ~+0.33ms/step signal, m_block_size_8, ~0.8% of a c8 step)
+
+The named residual-#1 lever (route the E=1 dense MXFP4/NVFP4 projections through vLLM's DENSE
+`marlin_gemm` — direct-A + `m_block_size_8` at prob_m≤8 — instead of our
+`MoeGroupedGemmNvfp4Marlin` E=1) was arbitrated per parity-lever-protocol (per-shape MEASUREMENT
+decides BEFORE a large port), NOT ported. The task's step-5 ubench ("the new dense op vs grouped at
+the model shapes — expect the +7-9% back") already exists as `dgx:~/mxfp4-nsys/mxfp4_marlin_ubench.py`
+(#46): it times the REFERENCE `apply_fp4_marlin_linear` (dense `ops.marlin_gemm`, the EXACT kernel a
+port would vendor) vs `ops.moe_wna16_marlin_gemm` E=1 (the 1:1 upstream twin of our production
+kernel) — same tool, both the oracle's compiled ops, cuda-event timed — at the Qwen3-8B **MXFP4**
+decode shapes {qkv 6144×4096, o 4096×4096, gate/up 12288×4096, down 4096×12288} × M∈{1,2,4,8}.
+Re-ran it 3× fresh on the idle GB10 under BOTH locks (worker down, free-g 93, disk unchanged 21G,
+tmux+done-marker); with #46 that is 4 same-tool runs.
+
+PER-STEP moe/dense ratio (dense = the port target; moe = our production twin; sum of 5 GEMMs ×36
+layers), 4 runs [#46, r1, r2, r3] and mean:
+| M | #46 | r1 | r2 | r3 | mean | delta_step_ms mean (moe−dense) | sign consistent? |
+|---|-----|-----|-----|-----|------|-------------------------------|------------------|
+| 1 | 1.023 | 1.008 | 0.952 | 1.006 | **0.997** | −0.030 | NO (flips ±) |
+| 2 | 0.986 | 0.983 | 1.028 | 1.032 | **1.007** | +0.057 | NO (flips ±) |
+| 4 | 1.077 | 1.023 | 0.964 | 0.975 | **1.010** | +0.071 | NO (flips ±) |
+| 8 | 1.020 | 1.016 | 1.044 | 1.077 | **1.039** | +0.327 | YES (all + ) |
+delta_step_ms per run: M=1 [+0.184,+0.061,−0.416,+0.052]; M=2 [−0.117,−0.142,+0.228,+0.258];
+M=4 [+0.617,+0.186,−0.312,−0.206]; M=8 [+0.169,+0.135,+0.367,+0.637]. Outputs match to bf16
+last-bit (maxdiff col = bf16 reduction-order near-tie, same class as the moe path's own near-tie).
+
+READING (the arbiter):
+- **M=1,2,4 (concurrencies c1/c2/c4): the port recovers ZERO.** moe/dense mean is within ±1% of
+  parity and the sign of delta_step FLIPS run-to-run (M=2 even favors MoE overall) — a true-zero
+  effect (benchmark-gate-statistics). The reference dense kernel a port would vendor is NOT faster
+  than the grouped-E1 twin we already run at these shapes.
+- **M=8 (c8) is the ONLY reproducible, mechanistically-real signal: ~+0.33ms/step, all 4 runs
+  positive.** Cause (per #46): at prob_m=8 dense picks `m_block_size_8`=TRUE (an 8-row tile) while
+  our grouped path picks `moe_block_size`=16 (m8=FALSE → a 16-row tile with 8 padding rows, ~2×
+  useful-row work). The clearest single shape is qkv M=8 (moe/dense 1.111/1.130/1.132/1.119 across
+  the 4 runs); gate/up/down partly wash so the per-STEP mean dilutes to 1.039. Magnitude: +0.33ms
+  out of a ~39.5ms c8 decode step (#46 TPOT) ≈ **0.8% of the c8 step** — recovers ~0.8pp of the
+  5.8% c8 gap and leaves c8 ≈0.95x, still BELOW 1.0x; at c1/c2/c4 it yields nothing.
+
+VERDICT = **NO-GO on the full dense-marlin port as an MXFP4 parity lever.** The +7-9% cross-tool
+figure (#47/#48 nsys-ours 120us vs torch-vLLM 113us) is the tool bias, NOT a recoverable
+kernel/dispatch cost — the same-tool per-shape arbiter (now 4 runs) reads parity/noise at M≤4 and
+only a ~0.8%-of-step M=8 sliver. Vendoring ~2000 lines of delicate dense marlin (`marlin_template.h`
+2081L + generated kernels + the `marlin_mm`/`marlin_gemm` launcher) behind a new `kMarlinDenseGemm`
+op and re-routing the CUDA-graph-captured decode through a NON-byte-exact (near-tie) path — for at
+most ~0.8pp at c8 only and nothing at c1–c4 — is gold-plating a measured near-dead hypothesis (the
+explicit #46 caution). "Never accept a ceiling" does not apply: the specific difference was FOUND
+and measured to be ~0 at M≤4. Provenance confirmed anyway: dense and MoE are genuinely separate
+`Marlin<>` templates (dense `MARLIN_KERNEL_PARAMS` has `lda`+`max_shared_mem`, direct A; moe has
+`sorted_token_ids/expert_ids/topk_weights`), so a real port IS ~2000 lines, not a shim — the cost
+side of the ledger is high, the win side is ~0.
+
+RESIDUAL MAP (refined): the recoverable slivers are NOT the dense port. (a) **M=8-only, CHEAP:** force
+`moe_block_size`=8 (not 16) for the E1 dense case in `dense_nvfp4_gemm.h::DenseAlignFor` →
+engages our ALREADY-VENDORED `m_block_size_8` grouped kernels (kernel_selector lines 3–8/33–38) →
+captures the ~0.33ms/step at c8 with a ~5-line change and NO new vendoring; its own small near-tie
++ SACRED gate. Legitimate because we run E1 (free to pick the block heuristic); recovers ~0.8pp of
+c8 only. (b) the per-call defensive `d.b.Memset(ws)` in `MatmulNvfp4MarlinD`/`GateUpFusedMarlinD`
+(marlin leaves its workspace zeroed; vLLM dense+moe both reuse without re-zeroing) — one tiny
+launch/GEMM, micro. (c) the **~0.7ms/step host/sched slice** — the real remaining term, host-side,
+not a kernel port. No single lever reaches ≥1.0x; the MXFP4 parity goal stays BELOW-FLOOR (best c8
+0.942). Evidence: `dgx:~/mxfp4-nsys/{mxfp4_marlin_ubench.py,ubench.log,ubench_3x.log}`. Box left
+clean (both locks free, GPU idle, worker down, disk 21G, tmux gone).