mudler · localai-bot · Aug 7, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -18,7 +18,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint
 | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool |
 | Invocation-parity prevention | CI guard + checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx |
 | MiniMax-H3 lane | **fl2va COHERENT; ref2va grid DIAGNOSED (#95): NO loader bug; bf16 13-shard DiT INDEXES** | residual = community-NVFP4 quant fidelity §8.12; no bf16 render yet |
-| Kimi-Linear-48B (KDA/NoPE-MLA/MoE) | **`vt::KdaGatedDeltaRule` LANDED + GB10-MEASURED** (§15, `row/KIMI-KDA-DEVICE-KERNEL`): **106→122/128 + 1.35→4.24 tok/s (3.1×)**, beats §14's 120. OFF (122≠STRICT) | close p7: chunked-prefill/paged-MLA/incremental |
+| Kimi-Linear-48B (KDA/NoPE-MLA/MoE) | device-KDA **122/128 + 4.24 tok/s** best (§15); device NoPE-MLA `VT_KIMI_DEVICE_MLA` **MEASURED-NEGATIVE 122→109** (§16, #107 — f32-softmax≠FA2, coin-flips). Both OFF ≠STRICT | STRICT owes ACTUAL kernels: chunk_kda + paged FA2 MLA + incremental |
 | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
 | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity |

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -14609,3 +14609,20 @@ The §14 named residual ("a NEW per-channel-decay GDN kernel `g[T,H,D]`; `vt::Gd
 RESULT (the §14 thesis CONFIRMED): the device recurrence — vLLM's ACTUAL f32-on-bf16 arithmetic — moves **106→122/128** (prompts 0-6 all 16/16; only p7 diverges at pos-6, got `387` vs golden `11`, a comma near-tie) AND is **3.1× FASTER (1.35→4.24 tok/s)**. It beats BOTH the control (106) AND §14's host-precision best (120, which needed both bf16 knobs), and FIXES the p2 divergence the f64 host path had — because it runs the right arithmetic, not a coin-flip. The §14 bf16 knobs are now SUPERSEDED + COUNTERPRODUCTIVE: device-KDA + bf16 REGRESSES 122→90 (reintroducing p3's `163586×` repeat loop) — they were tuned to compensate for the f64 host island's over-precision. The speed win = the device recurrence kills the per-step host Download/f64-recompute/upload round-trip and runs the O(T²) recurrence in parallel on the GPU.
 
 VERDICT: device-KDA (122/128, 4.24 tok/s) is the NEW BEST on BOTH axes but STILL a DIVERGENCE (STRICT required, K=3-deterministic golden) → `VT_KIMI_DEVICE_KDA` STAYS OFF (parity-enablers). The residual is now a SINGLE near-tie (p7 pos-6). NAMED next brick to STRICT (+ more speed): the KDA chunked-prefill kernel family (vLLM processes the PROMPT with `chunk_kda`, we still run the recurrent form — regen a Triton-AOT cubin for sm_121a via `scripts/regen-triton-aot.sh`, or a native `chunk_kda` port) + paged `mla::ForwardMlaAttentionBlock` for the 7 NoPE-MLA layers + paged-incremental decode (persistent KDA state + MLA-KV, kills the remaining O(n²)). Row STAYS ACTIVE.
+
+## 2026-08-07 — Kimi-Linear-48B: device NoPE-MLA attention lever `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE (122→109/128 AND slower); STRICT still owes vLLM's ACTUAL FA2/chunk_kda kernels (`row/KIMI-STRICT-CLOSE`, #107)
+
+The §15 residual (d) — "the 7 NoPE-MLA layers still use a host f64 softmax island; closing p7 needs paged `mla::ForwardMlaAttentionBlock`" — was attempted in its device-COMPUTE form (the §15 device-KDA pattern applied to the MLA half) and MEASURED NEGATIVE on GB10. Additive knob `VT_KIMI_DEVICE_MLA` + `MlaAttnCoreDevice` (`kimi_linear_device.cpp`): the NoPE causal softmax over per-head `[k_nope|k_pe(shared)]`/`v` runs through the shared device op `vt::Attention` (f32 online max-subtracted softmax — vLLM's FA2 accumulation regime) instead of the f64 host `MlaSoftmaxIsland`. `vt::Attention` carries ONE head-dim for q/k/v while MLA is asymmetric (`qk = qk_nope+qk_rope = 192`, `v = 128`), so the value is zero-PADDED to `qk` — the weighted sum over the zero tail is 0, so `out[:, :, :v]` is byte-exact (softmax weights depend only on `q·k`). MLA dims VERIFIED from the real 48.9B `config.json` (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn `[4,8,12,16,20,24,27]`, 20 KDA) — not the K3 numbers.
+
+Unit gate (RED-first, CPU) GREEN: `test_kimi_linear_forward` **14/14·825** (was 13/13·656) — NEW case (g2) `KimiMlaAttnCoreDevice` (pad-V + `vt::Attention`) == a from-first-principles f64 causal-softmax reference at the Kimi MLA geometry (rtol 3e-3); RED-first verified (a perturbed scale fails 108 assertions). Env-gated whole-forward runs green (`VT_KIMI_DEVICE_MLA=1` alone and with `VT_KIMI_DEVICE_KDA=1`). Same on the GB10 CUDA binary (clean Release, CUTLASS-NVFP4 + FA2 + Triton-AOT, 210 GDN + 23 KDA syms linked).
+
+Full 48.9B GB10 gate vs the §12 STRICT `greedy_ids.npy` (single-load per config, `flock $HOME/gpu.lock`, `drop_caches` before wall-clock, min-avail **21 GiB**, no reboot, worker parked+restored):
+
+| Config | env (all `VT_KIMI_DEVICE_COMPUTE=1`) | /128 | tok/s | verdict |
+|---|---|---|---|---|
+| control (device-KDA) | `DEVICE_KDA=1` | 122 | 4.24 | reproduces §15 EXACTLY (p0-p6 16/16, p7 10/16) |
+| **+ device-MLA** | `DEVICE_KDA=1 DEVICE_MLA=1` | **109** | **3.89** | **REGRESSION on BOTH axes** |
+
+WHY NEGATIVE (the §14 razor, re-proven). device-KDA WORKS (106→122) because the recurrence is the SAME algorithm as vLLM's decode kernel, only f32-on-bf16 — it MATCHES. But vLLM's MLA prefill uses FA2 (a specific flash tiling/reduction ORDER); `vt::Attention`'s plain f32 online-softmax is the right MATH but a DIFFERENT reduction order, so — exactly like §14's host-precision plateau — it COIN-FLIPS near-ties: it BREAKS p3 16/16→3/16 (got `220,41938,382,1810,…163586,163586` — the same `163586×` degenerate repeat §14's bf16 knobs caused) while p7 stays diverged at 10/16. And it is SLOWER (4.24→3.89): the per-`(t,h)` key/value build copies + the 192-dim pad-V waste add overhead to the O(n²) recompute path. An approximation of vLLM's kernel is not enough — only the ACTUAL kernel matches.
+
+VERDICT: `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (parity-lever precedent: §14's `ISLAND_F32ACC`). device-KDA (122/128, 4.24 tok/s) remains the best config, itself default OFF (122 ≠ STRICT). The one-brick STRICT-close did NOT land. STRICT residual, SHARPENED: needs vLLM's ACTUAL kernels, not a device approximation — (c) the **chunk_kda** prefill family (`chunk_kda_scaled_dot_kkt`+`recompute_w_u`+`chunk_gla_fwd_o_gk`+`fused_kda_gate_chunk_cumsum`, FLA `ops/kda.py`) via a Triton-AOT regen for sm_121a (`scripts/regen-triton-aot.sh` + new `triton_kernels/*.py`), the named prime suspect; (d) the paged FA2 `mla::ForwardMlaAttentionBlock` (NOT the `vt::Attention` approximation tried here); (e) paged-incremental decode (needs a decode/paged-attn op with `query_len≠key_len`, which `vt::Attention` cannot express; kills the O(n²)). Each is a substantial multi-kernel brick, recorded as the named follow-on. Row STAYS ACTIVE. HONEST bar stays HW-forced-indirect (vLLM cannot serve this bf16 on one GB10 with KV headroom — §14).
diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md
diff --git a/.agents/specs/kimi-linear.md b/.agents/specs/kimi-linear.md
@@ -913,6 +913,65 @@ aot.sh`), or a native `chunk_kda` port. Row STAYS `ACTIVE`.
 
 ---
 
+## 16. DEVICE NoPE-MLA attention lever MEASURED-NEGATIVE; STRICT still owed the ACTUAL FA2/chunk_kda kernels (2026-08-07, `row/KIMI-STRICT-CLOSE`, #107)
+
+The §15 residual (d) — "the 7 NoPE-MLA layers still use a host f64 softmax island … closing p7
+needs paged `mla::ForwardMlaAttentionBlock`" — was attempted in its device-COMPUTE form (the §15
+device-KDA pattern applied to the MLA half) and MEASURED-NEGATIVE on GB10. The one-brick STRICT-close
+did NOT land; the honest verdict re-confirms §14's razor.
+
+**Implementation (`kimi_linear_device.cpp`, additive, default OFF).** New knob `VT_KIMI_DEVICE_MLA` +
+helper `MlaAttnCoreDevice`: the NoPE causal softmax over per-head `[k_nope|k_pe(shared)]`/`v` runs
+through the shared device op `vt::Attention` (f32 online max-subtracted softmax — vLLM's FA2
+accumulation regime) instead of the f64 host `MlaSoftmaxIsland`. `vt::Attention` carries a single
+head-dim for q/k/v while MLA is asymmetric (`qk = qk_nope+qk_rope = 192`, `v = 128`), so the value is
+PADDED to `qk` with zeros — the weighted sum over the zero tail is 0, so `out[:, :, :v]` is byte-exact
+to the unpadded math (softmax weights depend only on `q·k`). q views `dq` directly as `[T,nah,192]`;
+key is built per `(t,h)` as `[k_nope | k_pe(broadcast)]`. Wired into both the f32 and bf16
+`MlaSoftmaxIsland` paths. MLA dims VERIFIED from the real 48.9B `config.json` (not the K3 numbers):
+`nah=32, qk_nope=128, qk_rope=64, v_head_dim=128, kv_lora=512, q_lora=None`; 7 full-attn/MLA layers
+(`full_attn_layers=[4,8,12,16,20,24,27]`), 20 KDA.
+
+**Unit gate (RED-first, CPU) GREEN.** `test_kimi_linear_forward` **14/14·825** (was 13/13·656) —
+NEW case (g2) `KimiMlaAttnCoreDevice` (pad-V + `vt::Attention`) == a from-first-principles f64
+causal-softmax reference at the Kimi MLA geometry (rtol 3e-3). RED-first verified: a perturbed scale
+fails 108 assertions. Env-gated whole-forward runs green (`VT_KIMI_DEVICE_MLA=1` alone and with
+`VT_KIMI_DEVICE_KDA=1`, 14/14·825). Same on the GB10 CUDA binary (210 GDN + 23 KDA syms linked).
+
+**Full 48.9B GB10 gate — MEASURED NEGATIVE (single-load per config, `flock $HOME/gpu.lock`, min-avail
+21 GiB, no reboot; the golden is the §12 STRICT `greedy_ids.npy`).**
+
+| Config | env (all `VT_KIMI_DEVICE_COMPUTE=1`) | /128 | tok/s | verdict |
+|---|---|---|---|---|
+| control (device-KDA) | `DEVICE_KDA=1` | 122 | 4.24 | reproduces §15 EXACTLY (p0-p6 16/16, p7 10/16) |
+| **+ device-MLA** | `DEVICE_KDA=1 DEVICE_MLA=1` | **109** | **3.89** | **REGRESSION both axes** |
+
+**Why negative (the §14 razor, re-proven).** device-KDA WORKS (106→122) because the recurrence is the
+SAME algorithm as vLLM's decode kernel, just f32-on-bf16 — it matches. But vLLM's MLA prefill uses
+**FA2** (a specific flash tiling/reduction order); `vt::Attention`'s plain f32 online-softmax is the
+right MATH but a DIFFERENT reduction ORDER, so — exactly like §14's host-precision-matching plateau —
+it COIN-FLIPS near-ties: it BREAKS p3 16/16→3/16 (into the same `163586×` degenerate repeat the §14
+bf16 knobs caused) while p7 stays diverged at 10/16. And it is SLOWER (4.24→3.89): the per-`(t,h)`
+key/value build copies + the 192-dim pad-V waste add overhead to the O(n²) recompute path. An
+approximation of vLLM's kernel is not enough — only the ACTUAL kernel matches.
+
+**Verdict + default.** `VT_KIMI_DEVICE_MLA` STAYS **OFF**, kept as a documented-MEASURED-NEGATIVE A/B
+knob (parity-lever precedent: §14's `ISLAND_F32ACC`/output-bf16). device-KDA (122/128, 4.24 tok/s)
+remains the best config, itself default OFF (122 ≠ STRICT). Row STAYS `ACTIVE`.
+
+**STRICT residual, sharpened by this measurement.** p7 (and now the coin-flip class generally) needs
+vLLM's ACTUAL kernels, NOT a device approximation: (c) the **chunk_kda** prefill kernel family
+(`chunk_kda_scaled_dot_kkt` + `recompute_w_u` + `chunk_gla_fwd_o_gk` + `fused_kda_gate_chunk_cumsum`,
+FLA `ops/kda.py`) — mirror-first via a Triton-AOT regen for sm_121a (`scripts/regen-triton-aot.sh` +
+new `triton_kernels/*.py`), the spec's named prime suspect; (d) the paged
+`mla::ForwardMlaAttentionBlock` (FA2) for the 7 NoPE-MLA layers — NOT the `vt::Attention` approximation
+tried here; (e) **paged-incremental decode** — coupled with (d) because it needs a decode/paged
+attention op (`query_len ≠ key_len`), which `vt::Attention` cannot express; it kills the O(n²)
+full-recompute (the current 4.24 tok/s is the recompute rate). Each is a substantial multi-kernel
+brick, not a one-shot; recorded as the named follow-on.
+
+---
+
 ## Structured contract (machine-readable — mirrors deepseek-v4-flash.md)
 
 ## Scope

diff --git a/.agents/state.md b/.agents/state.md
@@ -41023,3 +41023,36 @@ is required") is now IMPLEMENTED as the additive device op `vt::KdaGatedDeltaRul
   (`chunk_kda`; regen a Triton-AOT cubin for sm_121a via `scripts/regen-triton-aot.sh`, or native port)
   + paged `mla::ForwardMlaAttentionBlock` (7 NoPE-MLA layers) + paged-incremental decode. Row `ACTIVE`.
   Box left clean (artifacts removed, memory restored, worker parked, no reboot).
+
+## 2026-08-07T12:35 — Kimi-Linear: device NoPE-MLA attention lever `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE (122→109/128, slower); STRICT owes the ACTUAL FA2/chunk_kda kernels (row/KIMI-STRICT-CLOSE, #107)
+- The §15 residual (d) — the 7 NoPE-MLA layers still run a host f64 softmax island; closing p7 needs
+  paged `mla::ForwardMlaAttentionBlock` — was attempted in its DEVICE-COMPUTE form (the §15 device-KDA
+  pattern applied to the MLA half) and MEASURED NEGATIVE on GB10. Additive, default-OFF knob
+  `VT_KIMI_DEVICE_MLA` + `MlaAttnCoreDevice` (`kimi_linear_device.cpp`): the NoPE causal softmax runs
+  through the shared device op `vt::Attention` (f32 online max-subtracted softmax = vLLM's FA2 regime)
+  instead of the f64 host `MlaSoftmaxIsland`. `vt::Attention` carries one head-dim; MLA is asymmetric
+  (qk=qk_nope+qk_rope=192, v=128), so value is zero-PADDED to qk (weighted-sum over the 0 tail = 0, so
+  `out[:,:,:v]` is byte-exact). MLA dims VERIFIED from the real 48.9B config (nah=32, qk_nope=128,
+  qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA) — not the K3 numbers.
+- Unit gate (RED-first, CPU) GREEN: `test_kimi_linear_forward` 14/14·825 (was 13/13·656) — NEW case
+  (g2) `KimiMlaAttnCoreDevice` (pad-V + `vt::Attention`) == a from-first-principles f64 causal-softmax
+  reference (rtol 3e-3); a perturbed scale fails 108 assertions. Env-gated whole-forward green
+  (`VT_KIMI_DEVICE_MLA=1` alone and with `DEVICE_KDA=1`). Same on the GB10 CUDA binary (210 GDN + 23
+  KDA syms linked, CUTLASS-NVFP4 + FA2 + Triton-AOT).
+- Full 48.9B GB10 gate vs the §12 STRICT golden (single-load per config, flock $HOME/gpu.lock,
+  drop_caches, min-avail 21 GiB, no reboot): control device-KDA reproduces 122/128 + 4.24 tok/s EXACTLY;
+  **+device-MLA REGRESSES to 109/128 AND 3.89 tok/s**. WHY (the §14 razor, re-proven): device-KDA works
+  because its recurrence is the SAME algorithm as vLLM's decode kernel (f32-on-bf16); but vLLM's MLA
+  prefill uses FA2 (a specific reduction ORDER), and `vt::Attention`'s plain online-softmax is the right
+  MATH in a DIFFERENT order → it COIN-FLIPS near-ties (breaks p3 16→3 into the §14 `163586×` repeat)
+  while p7 stays diverged, and the per-(t,h) build + 192-dim pad-V waste slow the O(n²) recompute. An
+  approximation of vLLM's kernel is NOT enough — only the actual kernel matches.
+- VERDICT: `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (§14
+  `ISLAND_F32ACC` precedent). device-KDA (122/128, 4.24) remains the best, itself default OFF. The
+  one-brick STRICT-close did NOT land. STRICT residual SHARPENED — needs vLLM's ACTUAL kernels:
+  (c) chunk_kda prefill family (FLA `ops/kda.py`; Triton-AOT regen for sm_121a via
+  `scripts/regen-triton-aot.sh` + new `triton_kernels/*.py`), the prime suspect; (d) paged FA2
+  `mla::ForwardMlaAttentionBlock` (NOT the vt::Attention approximation); (e) paged-incremental decode
+  (needs a decode/paged-attn op, query_len≠key_len, which `vt::Attention` cannot express; kills O(n²)).
+  Each a substantial multi-kernel brick. Row STAYS ACTIVE. Box left clean (build tree/markers removed,
+  memory restored to 112 GiB avail, worker restarted --restart=always, golden preserved, no reboot).
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -309,7 +309,7 @@ built on it rather than keeping the flattering one.
 | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids |
 | `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (request aliases, the MP4 content route, and reference conditioning wiring), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below |
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
-| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | e2e RUNS (bf16-resident §13); per-channel-decay KDA device kernel `vt::KdaGatedDeltaRule` LANDED+GB10: **106→122/128 + 4.24 tok/s (3.1×)**, beats §14's 120 on both axes, NOT STRICT; default OFF | device-KDA runs vLLM's actual GPU recurrence; bf16 knobs now hurt (122→90). p7 near-tie; residual = KDA chunked-prefill + paged FA2 MLA + incremental. §15 |
+| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | e2e RUNS (bf16-resident §13); KDA device op `vt::KdaGatedDeltaRule` GB10 **106→122/128 + 4.24 tok/s (3.1×)**, NOT STRICT, default OFF; device NoPE-MLA `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE 122→109 (§16) | device-KDA = vLLM's actual GPU recurrence; bf16 knobs hurt (122→90); MLA vt::Attention f32-softmax ≠ FA2 order, coin-flips (breaks p3). p7 near-tie; residual = chunk_kda + paged FA2 MLA + incremental §15/§16 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
 | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B UNBLOCKED (the 13-shard bf16 DiT is now indexable) but NOT MEASURED: no bf16 render exists. Detail: benchmark-record + spec §8 |
 | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) |