perf(vulkan): ragged M reaches coopmat — 27B prefill 21.5x

.agents/benchmark-record.md

@@ -15982,3 +15982,72 @@ loads, or caching the K/V slice across the query heads that share a KV head —
 Recorded because the hypothesis was specific and the refutation is reusable: this
 is the second kernel this session where a barrier-count argument looked compelling
 and measured flat (the subgroup GEMV was the first).
+
+### ★ RAGGED M: a one-line predicate was costing 21.5x on prefill (2026-08-08, GB10)
+
+The largest win of this campaign, and it was self-inflicted.
+
+**How it was found.** Two GDN increments moved coverage 11 -> 3 and speed not at
+all, so a prefill-dominated run was profiled with GPU timestamps:
+
+    vt_matmul   433 calls   409,611.9 ms   99.9% of GPU   945.99 ms/call
+
+The UNTILED SCALAR kernel — the portable correctness tier — was carrying every
+prefill GEMM at ~96 GFLOP/s, roughly **1% of what GB10 can do**. GPU busy was 95%
+of wall, so prefill was never host-bound, and the remaining reference-tier ops
+accounted for ~0.1%. **Three prior structural attributions were wrong; the
+timestamp profile settled it in one run.**
+
+**Why coopmat declined, measured not reasoned.** Both obvious candidates were
+excluded by reading the source: every dimension is a whole tile (5120, 17408,
+256) and activations are bf16 (`DBuf dx(d, DType::kBF16, ...)`). The predicate had
+to be made to report itself, gated behind `VT_VULKAN_DISPATCH_STATS`:
+
+    coopmat DECLINED: M is not a multiple of 16  (a.dtype=2 b.dtype=2 m=17 k=5120 n=10240)
+
+**`m = tokens + 1`.** A 512-token prompt gives m=513, and `513 % 16 == 1`. Every
+prefill GEMM, at every prompt length, fell to the scalar kernel.
+
+**The cause was my own hang fix from the previous day.** `coopMatLoad` reads a
+full 16x16 tile unmasked, so at M=1 it read ~30 KB past the activation buffer and
+the fence never signalled. That was fixed by requiring `m % 16 == 0 && n % 16 ==
+0` — correct, and it is why the 27B runs at all instead of hanging. The note left
+at the time read *"a masked/padded load is the better long-term answer, not
+attempted here."* That deferred work WAS the prefill bottleneck. It went unnoticed
+because the only model then running on Vulkan was opt-125m, whose small GEMMs made
+the scalar path cheap.
+
+**The fix: shift the trailing tile back, do not mask.** A tile that would overrun
+M slides down to start at `M-16`, so it reads only REAL rows. Exact, not
+approximate: a result row depends solely on that row of A and on B, never on which
+tile computed it, so the rows shared with the previous tile recompute to
+bit-identical values and the duplicate stores write the same bytes. It needs
+`M >= 16`, which the predicate now requires in place of `M % 16 == 0`; below that
+there is no in-bounds 16-row window, and decode (M=1) is served by the GEMV
+tactic.
+
+**MEASURED, GB10, Qwen3.6-27B, 512-token prefill:**
+
+| | before | after | |
+|---|---:|---:|---:|
+| GEMM ms/call | 945.99 | 30.40 | **31.1x** |
+| prefill tok/s | 1.18 | 25.41 | **21.5x** |
+| E2E ms | 433,321 | 20,192 | **21.5x** |
+
+`vt_matmul` is gone from the profile; `vt_matmul_coopmat` carries 432 calls at
+30.4 ms. 21.5x is far outside this box's 2.1x noise band, so it is callable at
+n=1. A 512-token prefill is now **20.1 s against llama.cpp's ~1.1 s — 18x behind,
+down from 244x.**
+
+**Gate: 26/26, 1563 assertions on GB10.** The new ragged-M case asserts the TACTIC
+(`PipelineExistsFor("vt_matmul_coopmat")`) and exactness at M=17, and it
+short-circuits on llvmpipe, which has no coopmat — the local run shows 1020
+assertions, the GB10 run 1563. A gate that cannot run on the CI device has to be
+run on the device that has the feature, or it proves nothing.
+
+**The durable lesson: a correctness fix can silently become a performance cliff,
+and a selection predicate that can route an entire model onto the correctness tier
+should be able to SAY SO.** Reading the source excluded the two candidates a human
+would guess and pointed at neither; the predicate's own report named it
+immediately. That diagnostic now ships behind the stats flag.
+

docs/BENCHMARKS.md

@@ -350,7 +350,7 @@ built on it rather than keeping the flattering one.
 | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config |
 | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box |
 | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs |
-| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 24 NATIVE (+8 GDN, BOTH recurrences; oracle-gated, no speed); 63 host-tier. opt-125m e2e token-exact on llvmpipe. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
+| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 24 NATIVE (+8 GDN, BOTH recurrences); 63 host-tier. **27B prefill 21.5x on GB10** (ragged-M reached coopmat). opt-125m e2e token-exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
 | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist |

docs/STATUS.md

@@ -411,7 +411,7 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact
 
 LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU,
 Vulkan (opt-125m exact, GEMV 1.8x; 24 native, +8 GDN incl. BOTH
-recurrences, oracle-gated, no speed; qwen3_5 #125 VERIFIED on 27B; CUDA build repaired
+recurrences; 27B prefill 21.5x on GB10; qwen3_5 #125 VERIFIED; CUDA build repaired
 [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green
 on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in —
 **blind-written, unverified** — M2 unblocks on verification; gfx1201 hipBLAS +

src/vt/vulkan/shaders/vt_matmul_coopmat.comp

@@ -85,8 +85,8 @@ const uint VT_CM = 16u;  // the reported M = N = K tile extent
 shared float vt_cm_spill[VT_CM * VT_CM];
 
 void main() {
-  // Output tile coordinates. THE HOST GUARANTEES m % 16 == 0 AND n % 16 == 0, so
-  // every tile is whole.
+  // Output tile coordinates. THE HOST GUARANTEES n % 16 == 0 and m >= 16; a
+  // ragged M is handled by the tile shift below.
   //
   // That guarantee is load-bearing and was learned the hard way: `coopMatLoad`
   // below reads a FULL 16x16 tile with no masking, so a partial tile reads past
@@ -100,6 +100,21 @@ void main() {
   uint tile_m = (tile / tiles_n) * VT_CM;
   uint tile_n = (tile % tiles_n) * VT_CM;
 
+  // RAGGED M, HANDLED BY SHIFTING THE LAST TILE BACK rather than masking the
+  // load. A trailing tile would read rows past M -- the fault described above --
+  // so it is slid down to start at M-16 and therefore reads only REAL rows.
+  //
+  // This is exact, not an approximation. A result row depends solely on that row
+  // of A and on B, never on which tile computed it, so the rows this tile shares
+  // with its predecessor are recomputed to bit-identical values. The duplicate
+  // stores write the same bytes to the same addresses; the values are equal, so
+  // the interleaving cannot matter.
+  //
+  // It requires M >= 16, which the host predicate enforces. Below that there is
+  // no in-bounds 16-row window at all, and the decode path (M=1) is served by the
+  // GEMV tactic instead.
+  if (tile_m + VT_CM > p.m) { tile_m = p.m - VT_CM; }
+
   coopmat<float, gl_ScopeSubgroup, VT_CM, VT_CM, gl_MatrixUseAccumulator> acc =
       coopmat<float, gl_ScopeSubgroup, VT_CM, VT_CM, gl_MatrixUseAccumulator>(0.0);
 

src/vt/vulkan/vulkan_ops.cpp

@@ -32,6 +32,8 @@
 #include <cstdint>
 #include <cstdlib>
 #include <cstring>
+#include <mutex>
+#include <set>
 #include <string>
 #include <vector>
 
@@ -42,6 +44,12 @@
 namespace vt::vulkan {
 namespace {
 
+// Gated on the same flag as the dispatch profile; costs nothing when unset.
+const bool kCoopMatWhy = [] {
+  const char* v = std::getenv("VT_VULKAN_DISPATCH_STATS");
+  return v != nullptr && std::strcmp(v, "0") != 0;
+}();
+
 // Storage dtype -> the shader-side code (vt_common.glsl VT_DT_*).
 uint32_t DtypeCode(DType d) {
   switch (d) {
@@ -478,6 +486,41 @@ bool CoopMatMatmulUsable(const Tensor& a, const Tensor& b, int64_t k, int64_t m,
   if (kDisabled) return false;
 
   const VulkanContext& ctx = VulkanContext::Get();
+
+  // WHY IT DECLINED, reported once per distinct reason under
+  // VT_VULKAN_DISPATCH_STATS. A 27B prefill measured 99.9% of GPU time in the
+  // UNTILED SCALAR kernel at ~96 GFLOP/s -- roughly 1% of what this device can do
+  // -- because this predicate was returning false for every GEMM, and reading the
+  // source did not reveal which clause. Shapes were whole tiles and activations
+  // were bf16, so the obvious two candidates were both excluded by inspection and
+  // the answer still had to be measured. A selection predicate that can silently
+  // route an entire model onto the correctness tier should be able to say so.
+  if (kCoopMatWhy) {
+    const char* why = nullptr;
+    if (!ctx.coopmat_bf16_f32()) why = "device reports no bf16->f32 16x16x16 SUBGROUP config";
+    else if (ctx.subgroup_size() != 32) why = "subgroup size is not 32";
+    else if (a.dtype != DType::kBF16) why = "operand a is not bf16";
+    else if (b.dtype != DType::kBF16) why = "operand b is not bf16";
+    else if (k % 16 != 0) why = "K is not a multiple of 16";
+    else if (m < 16) why = "M is below one 16-row tile";
+    else if (n % 16 != 0) why = "N is not a multiple of 16";
+    if (why != nullptr) {
+      static std::mutex seen_mu;
+      static std::set<std::string> seen;
+      std::string key = std::string(why) + "|" + std::to_string(static_cast<int>(a.dtype)) +
+                        "," + std::to_string(static_cast<int>(b.dtype));
+      std::lock_guard<std::mutex> g(seen_mu);
+      if (seen.insert(key).second) {
+        std::fprintf(stderr,
+                     "[vt vulkan] coopmat DECLINED: %s  (a.dtype=%d b.dtype=%d "
+                     "m=%lld k=%lld n=%lld)\n",
+                     why, static_cast<int>(a.dtype), static_cast<int>(b.dtype),
+                     (long long)m, (long long)k, (long long)n);
+        std::fflush(stderr);
+      }
+    }
+  }
+
   return ctx.coopmat_bf16_f32() && ctx.subgroup_size() == 32 &&
          a.dtype == DType::kBF16 && b.dtype == DType::kBF16 && k % 16 == 0 &&
          // M AND N MUST ALSO BE WHOLE TILES. `coopMatLoad` reads a FULL 16x16
@@ -493,7 +536,18 @@ bool CoopMatMatmulUsable(const Tensor& a, const Tensor& b, int64_t k, int64_t m,
          // inside the allocation and its garbage rows were discarded by the
          // bounds-checked store. Raggedness alone was not enough; the read has to
          // leave the allocation to fault.
-         m % 16 == 0 && n % 16 == 0;
+         //
+         // M NEED ONLY BE AT LEAST ONE WHOLE TILE, not a multiple of one. The
+         // shader slides a trailing tile back to start at M-16, so every read
+         // stays in bounds and the shared rows recompute to identical values.
+         //
+         // Requiring m % 16 == 0 here is what fixed the original hang, and it
+         // MEASURED as the entire prefill bottleneck: prompt length gives
+         // m = tokens + 1, so 513 % 16 == 1 sent every 27B prefill GEMM to the
+         // untiled scalar kernel -- 99.9% of GPU time at ~96 GFLOP/s, about 1% of
+         // this device. N stays whole because a ragged N would need the same
+         // treatment on the B operand and no shape in play needs it.
+         m >= 16 && n % 16 == 0;
 }
 
 // GEMV TACTIC SELECTION (VK-F). Same shape of contract as the coopmat predicate