mudler · mudler · Aug 9, 2026 · Aug 9, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -1,6 +1,6 @@
 # NOW — the one-Read resume surface
 
-<!-- now-updated: 2026-08-08 -->
+<!-- now-updated: 2026-08-09 -->
 
 Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live,
 the gate being chased, what to do next. Never a log — evidence lives in the
@@ -26,7 +26,7 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`.
 | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist |
 | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — |
 | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning |
-| Vulkan 27B lm_head | **column-blocked -1.07 ms/tok, 6/6**; roof is 230 not 273 | 20x bimodality REAL on an IDLE box; reclaim REFUTED |
+| Vulkan 27B decode | **4.285 vs llama.cpp 4.35 (98.5%)** BINDING, 8/8 clean legs @`81ea01f0`; GPU 240.3->227.7 ms/tok; GEMV lever CLOSED | 3.1 ms short. Open: 20x lm_head bimodality REAL on an IDLE box (reclaim REFUTED, mechanism unknown) |
 | `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed (contributor, authorship-preserved); W0 green 4 archs** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) |
 | TP spike #287 (PR #143) | **TP-W1 LANDED**: rank-group table + TP handle (6/6); DSR leak FIXED (unblocks #127/#154/#155) | TP-W2 (linears + loader) |
 | Release | SPIKE; 30/30 | #129 |

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -16542,3 +16542,65 @@ column blocking suppresses it. It is NOT "a lever bigger than every other Vulkan
 optimization" on the evidence available -- at 1 leg in 16 it costs roughly 6% of
 mean decode time, against the 8.5% that column blocking takes off this kernel
 deterministically.
+
+## BACKEND-VULKAN combined: the three levers measured TOGETHER on merged main (2026-08-09, GB10, `81ea01f0`)
+
+Each of the three Vulkan levers was measured against `93852c28` in isolation, so
+none of their numbers described a tree carrying all three. `docs/STATUS.md` was
+deliberately left at the CONSERVATIVE 4.24 (the largest single lever) rather than
+a sum. This entry is the binding combined measurement that replaces it.
+
+**METHOD.** `git archive` of the merged tree to dgx, `vt_matmul.comp` md5 verified
+identical on both sides. Fresh Release configure. 8 wall-clock legs with the page
+cache dropped before each and `flock ~/gpu.lock` held; then a two-length
+GPU-timestamp diff (output-len 36 minus 4, over 32 decode tokens) so prefill and
+one-time costs cancel. Qwen3.6-27B bf16, 1 prompt, 32-in, c1.
+
+**A FIRST ATTEMPT WAS INVALID AND IS RECORDED BECAUSE THE FAILURE IS REUSABLE.**
+It read 0.75-1.27 tok/s. `VLLM_CPP_VULKAN` defaults to `AUTO`, which resolves to
+`OFF` (Vulkan is opt-in so it cannot register into gate builds), and the fresh
+configure omitted `-DVLLM_CPP_VULKAN=ON`. The options had been copied from the
+reference build's `CMakeCache.txt` through `grep | head -15`, and the
+alphabetically-sorted list ended at `TRITON_TARGET`, exactly one line before
+`VULKAN`. Three independent tells were present in the output and are the cheap
+check: NO `[vt vulkan]` lines despite `VT_VULKAN_DISPATCH_STATS=1`, no
+`[vt reference-tier]` lines, and "Asynchronous scheduling is ENABLED" where every
+valid Vulkan run reports it disabled. Same family as this campaign's stale-binary
+false greens, in the opposite direction: a false catastrophe rather than a false
+pass. Re-run after confirming `VLLM_CPP_VULKAN:STRING=ON` in the cache AND that
+the run emits Vulkan lines BEFORE trusting it.
+
+**MEASURED, 8 legs, ALL CLEAN.** TPOT 232.18, 232.42, 232.50, 233.34, 233.42,
+233.47, 233.80, 234.07 ms; decode 4.27-4.31 tok/s. **Median 4.285 tok/s**, spread
+0.8%. Zero bimodal collapses in 8 legs (the ~1.8x pathology hit 1 leg in 16 in the
+lm_head row's block, so its absence here is consistent, not evidence against it).
+
+**MEASURED, per decode token, two-length diff:**
+
+| shader | before (`93852c28`) | merged | delta |
+|---|---:|---:|---:|
+| `vt_matmul_vec` | 214.1 | **210.1** | -4.0 |
+| `vt_matmul` (lm_head) | 12.43 | **11.57** | -0.86 |
+| `vt_rms_norm` -> `vt_rms_norm_wide` | 7.97 | **1.57** | -6.40 |
+| `vt_paged_attn` | 2.02 | 2.00 | - |
+| `vt_gdn_decode` | 1.40 | 1.40 | - |
+| **total GPU** | **240.3** | **227.7** | **-12.6** |
+
+Wall 233.0 ms, so host is **5.3 ms/token**. Each lever reproduced its own claim on
+the merged tree: `vt_rms_norm_wide` at 0.0123 ms/call is exactly the 0.0123 its row
+reported, and the GEMV and lm_head deltas match theirs to within the leg spread.
+
+**WHERE THE REMAINING 3.1 ms IS.** llama.cpp Vulkan is 4.35 tok/s = 229.9 ms/token
+on the same 50.89 GiB weights on this box. `vt_matmul_vec` is now 92% of our GPU
+time at 248.0 GB/s over 52.1 GB/token. The two named residuals are both small:
+lm_head at 219.8 GB/s is 95.4% of the 230.3 GB/s ceiling MEASURED for its own shape
+(worth ~1.3 ms if it could reach the layer GEMVs' 248), and host at 5.3 ms.
+
+**A CORRECTION TO THIS CAMPAIGN'S ROOF ARITHMETIC.** The lm_head row established
+that GB10 does not reach its theoretical 273 GB/s, by running a known-good
+streaming kernel on the identical byte count: 230.3 GB/s. That correctly retires
+the 9.3 ms lm_head floor. It does NOT invalidate the layer-GEMV percentages, which
+MEASURE 243-248 GB/s -- above 230.3 -- so 230.3 is a ceiling for THAT SHAPE
+(k=5120, n=248320, one 2.54 GB buffer), not a device ceiling. Why one shape's
+ceiling sits ~7% below the same kernel's on layer weights is UNEXPLAINED, and is
+the same lone buffer the 20x bimodal collapse attaches to.
diff --git a/.agents/state.md b/.agents/state.md
@@ -43483,3 +43483,61 @@ Settling it needs ~60 more legs. It is the named next experiment, not a claim.
 The coordinator's "bigger than every other Vulkan lever" framing is NOT supported:
 at 1 leg in 16 the collapse costs roughly 6% of mean decode time, against the
 8.5% column blocking takes off this kernel deterministically.
+
+<!-- state: 2026-08-09T13:00 -->
+
+## Vulkan 27B decode: three levers landed and MEASURED TOGETHER; 4.285 vs llama.cpp 4.35
+
+IMMUTABLE HEAD: `81ea01f0` on `main` (PRs #183, #184, #185, #186 all merged).
+
+WHAT LANDED, each verified by the operator re-running the gate rather than taking
+the row's report:
+- #183 `kAttnQkNormRopeGate` native: reference-tier drains during decode 16 -> 0.
+- #184 GEMV wide loads: 90.0 -> 91.1% of roof. llama.cpp's own rows-per-workgroup
+  trick MEASURED 0.966x, a LOSS here, and ships off.
+- #185 `vt_rms_norm` was dispatching ONE WORKGROUP PER ROW, so batch-1 decode ran
+  128 lanes on a single SM: 7.85 -> 1.59 ms/token.
+- #186 lm_head column blocking: -1.07 ms/token, and the 9.3 ms floor the
+  coordinator briefed does NOT exist.
+
+BINDING COMBINED MEASUREMENT (the combination no row tested), GB10, 8 legs, page
+cache dropped before each, `flock $HOME/gpu.lock`: median decode **4.285 tok/s**,
+TPOT 232.18-234.07 ms, all 8 legs clean. GPU 240.3 -> 227.7 ms/token; host 5.3 ms.
+Full detail and the per-shader table are in the benchmark record.
+
+GATES RUN: on the merged tree, `test_vulkan_backend` 32/32 (1905 assertions),
+`test_backend_cross_device` 11/11, `test_opt_paged_engine` on Vulkan 6/6
+token-exact (96/96) with 0 declines, `gen-vulkan-spirv.py --check` byte-clean under
+pinned glslang 16.5.0. Per-branch GB10 gates are in each row's own entry.
+
+BLOCKER / WHY NOT DONE: the goal is llama.cpp Vulkan's 4.35 tok/s = 229.9
+ms/token. We are 3.1 ms short. `vt_matmul_vec` is now 92% of GPU time at 248.0
+GB/s and its lever is CLOSED with the binding constraint identified as DRAM
+bandwidth on a ~50 GB working set streamed once per token. The named residuals are
+small and both are open questions rather than known work: lm_head at 219.8 GB/s is
+95.4% of the 230.3 GB/s ceiling measured for ITS OWN SHAPE, and why that shape's
+ceiling sits ~7% below the same kernel's on layer weights is UNEXPLAINED.
+
+PROHIBITIONS: do not run two 27B processes on dgx at once; that box has
+OOM-rebooted. Take `flock $HOME/gpu.lock`. `local-ai-worker` must be stopped
+before GPU work and restored to `--restart=always` after.
+
+OPEN, NOT CLAIMS:
+- The ~20x `vt_matmul` bimodal collapse is REAL on an IDLE box with 119 GB
+  MemAvailable before every leg; co-residency, page reclaim and load effects are
+  all REFUTED. Mechanism UNKNOWN. The one surviving correlate is near-ceiling
+  OCCUPANCY (106 GB RSS of 119); the ballast test that would settle it was not run
+  because that has OOM-rebooted this box. Zero collapses in the 8 combined legs.
+- Suggestive only (p about 0.07, needs ~60 more legs): collapses landed only on
+  the unblocked lm_head arm.
+
+THREE DOC GATES BLOCKED FACTUALLY-CORRECT CHANGES and were NOT gamed: `check-pr-size`
+counts generated `vulkan_spirv.cpp` as reviewable product code (every Vulkan shader
+PR trips it); `docs/STATUS.md` sits over a shrink-only ratchet that only a
+33,211-char foreign table cell can clear; and `README.md:310` still says "24 native
+ops" (now 25) because the README gate demands an unrelated landing-page edit. These
+are checker gaps, and they are for the developer to rule on.
+
+FIRST RESUME COMMAND:
+    git log --oneline -1 origin/main   # expect 81ea01f0 or later
+    ssh dgx.casa 'grep -E "^VLLM_CPP_VULKAN" ~/vkcomb/build-vk/CMakeCache.txt'  # must be ON
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -361,7 +361,7 @@ built on it rather than keeping the flattering one.
 | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config |
 | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box |
 | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs |
-| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN); 62 host-tier. **27B prefill 21.5x**, decode 4.13->**4.24** vs 4.35. **GEMV CLOSED at 91.1% of roof**; RMSNorm 1 wg/row; lm_head **-1.07ms/tok**. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
+| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN); 62 host-tier. **27B prefill 21.5x**, decode **4.28**/4.35 BINDING (8/8 clean legs, merged tree). GPU 240.3->227.7 ms/tok. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
 | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist |

diff --git a/docs/STATUS.md b/docs/STATUS.md
@@ -422,7 +422,7 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact
 LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU,
 Vulkan (opt-125m exact; 25 native +8 GDN, both
 recurrences + fused attn preamble; 27B prefill 21.5x, decode
-4.24/4.35 llama.cpp, RMSNorm+GEMV+lm_head; #125 VERIFIED
+4.28/4.35 llama.cpp, 3 levers combined; #125 VERIFIED
 [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green
 on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in
 (**blind-written, unverified**); M2 needs verification; gfx1201 hipBLAS +