feat(vulkan): six native GDN/conv1d kernels, and the sub-word buffer bug they found

.agents/feature-matrix.md

@@ -277,7 +277,7 @@ evidence.
 | `BACKEND-CPU` | production CPU | `PARTIAL` | persistent threadpool + chunked GEMM/row dispatch is 1/3/20-thread bit-identical and TSAN-clean; idle-host performance/RSS gate and compute-in-quant remain open | [backend matrix](backend-matrix.md) |
 | `BACKEND-ROCM` | ROCm | `INVENTORIED` | source/dispatch spike required; no "one flag" support claim | [backend matrix](backend-matrix.md) |
 | `BACKEND-MLX` | Apple Metal through MLX | `ACTIVE` | Metal/MLX skeleton ACTIVE: two models (OPT-125m, Qwen3-0.6B) run e2e + pass correctness, native-MSL GEMM, batched command buffers 1.50x (compute-bound at 98%+); optional MLX GEMM provider | [backend matrix](backend-matrix.md) |
-| `BACKEND-VULKAN` | Vulkan | `ACTIVE` | **16 NATIVE kernels**; **opt-125m RUNS END TO END, STRICT token-exact 6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle, 0 provider declines** of the CPU backend's 87 registered ops (GEMM both orientations, embedding, greedy argmax, block-paged attention + KV write, QKV split, rotary apply, elementwise/norm/fusion); the other 71 served by the portable reference tier (CPU kernel, unified memory). No model run e2e, no speed number owed | [backend matrix](backend-matrix.md), [campaign spec](specs/vulkan-full-support.md) |
+| `BACKEND-VULKAN` | Vulkan | `ACTIVE` | **22 NATIVE kernels**; **opt-125m RUNS END TO END, STRICT token-exact 6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle, 0 provider declines** of the CPU backend's 87 registered ops (GEMM both orientations, embedding, greedy argmax, block-paged attention + KV write, QKV split, rotary apply, elementwise/norm/fusion, and the GDN/conv1d glue: sigmoid gate, gated RMSNorm, state gather/scatter, decode conv1d update, fused post-conv); the other 65 served by the portable reference tier (CPU kernel, unified memory). `kGdnPrefill`/`kGdnDecode`/`kCausalConv1dFwd` deliberately still host-tier; `kRopeCosSinCache` stays host by design (double-precision table, mirrors vLLM). No speed number owed | [backend matrix](backend-matrix.md), [campaign spec](specs/vulkan-full-support.md) |
 | `BACKEND-XPU` | Intel XPU | `INVENTORIED` | loyal upstream-platform port, runtime absent | [backend matrix](backend-matrix.md) |
 | `BACKEND-ANE` | encoder/pooling accelerator | `INVENTORIED` | specialized CoreML route only | [backend matrix](backend-matrix.md) |
 

docs/BENCHMARKS.md

@@ -335,7 +335,7 @@ built on it rather than keeping the flattering one.
 | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config |
 | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box |
 | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs |
-| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 16 NATIVE kernels; 71 on the host tier. opt-125m runs e2e token-exact on llvmpipe; no speed measured. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
+| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | **NOT APPLICABLE: nothing measured, claimed or owed.** 22 NATIVE kernels (+6 GDN glue, CPU-oracle gated, no speed); 65 host-tier. opt-125m e2e token-exact on llvmpipe. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
 | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** The W0 skeleton registers 1 of 106 ops and its HIP sources have never been compiled by anyone; no AMD hardware here | A contributor's first `-DVLLM_CPP_HIP=ON` build ([#41](https://github.com/mudler/vllm.cpp/issues/41)). Only once a model runs does a same-box vLLM-ROCm oracle become the gate; the floor is vLLM, quant-matched |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | Parakeet/FastConformer ASR (P1-P4 + ONE-SURFACE fold ROW 1) | **NO number measured, claimed or owed.** Correctness-gated only, CPU f32; the 2026-08-07 surface fold (`vllm_transcribe`, `/v1/audio/transcriptions`) is transcript-byte-identical plumbing, no speed claim. | Floor is `parakeet.cpp`, same clip and box; needs a CUDA provider and a pretrained checkpoint |

docs/FEATURES.md

@@ -212,8 +212,10 @@ kernels for it.
 Vulkan **runs a model end to end**: `opt-125m` greedy is STRICT token-exact, 6/6
 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle, all nine of that model's ops
 dispatched natively with **zero provider declines**, on llvmpipe (no Vulkan GPU is
-reachable here). Still partial: **16 native kernels**, the other 71 ops fall back
-to the portable CPU tier, and quant/MoE/MLA/linear-attention have none at all (MLA
+reachable here). Still partial: **22 native kernels**, the other 65 ops fall back
+to the portable CPU tier. Six of the 22 are the **GDN / conv1d glue** a GDN hybrid
+(Qwen3.6) hits every step, gated against the CPU oracle; the GDN recurrences
+themselves are still on the host tier, and quant/MoE/MLA have none at all (MLA
 is refused at the platform seam, not mis-routed). **No speed number is owed.**
 Build with `-DVLLM_CPP_VULKAN=ON`; off by default.
 

docs/STATUS.md

@@ -410,8 +410,8 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact
 ## Not supported yet
 
 LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU,
-Vulkan (opt-125m exact, GEMV 1.8x,
-decode 8.59->91.7 t/s (10.7x), 1.75x off llama.cpp; qwen3_5 host-ptr bug fixed (#125), unrun off-CUDA
+Vulkan (opt-125m exact, GEMV 1.8x; 22 native, +6 GDN glue gated,
+recurrences host-tier; qwen3_5 host-ptr fixed (#125), unrun off-CUDA
 [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 skeleton:
 backend, platform, 1 of 106 ops; **its HIP sources have never been compiled by
 anyone** — weaker than "build-supported", since no AMD board is here;

src/vt/vulkan/shaders/vt_causal_conv1d_update.comp

@@ -0,0 +1,94 @@
+#version 450
+// vt::CausalConv1dUpdate — the GDN decode-step depthwise causal conv
+// (gdn-semantics.md §3, `causal_conv1d_update` at seqlen == 1): read the OLD
+// K-1 taps, emit one output token, then roll the window left and append the RAW
+// x sample.
+// Ported 1:1 from src/vt/cpu/cpu_ops.cpp:1081-1127 CausalConv1dUpdateKernel.
+//
+// ONE INVOCATION PER (token, channel), which is exactly the CPU kernel's
+// row-chunking unit (cpu_ops.cpp:1096 `ForRows(batch * c_dim, ...)`) and is what
+// makes the read-old-then-roll safe WITHOUT any barrier: each (bt, c) pair owns
+// its own conv_state row slice and its own output element, so the roll at the end
+// races with nothing. Batch rows map to DISTINCT cache slots by contract
+// (src/vt/ops.cpp:1687-1696), so two invocations never share a row.
+//
+// THE STATE ROW IS f32 AND BOUND THROUGH THE 32-BIT VIEW ONLY. On this backend
+// conv_state is f32 by contract (src/vt/ops.cpp: bf16 conv_state is CUDA-only),
+// so a dtype-erased load would be a branch that can never take its other side.
+//
+// THE ROW MAY BE PHYSICALLY WIDER THAN THE WINDOW. For spec-decode rollback the
+// row is widened to (K-1) + num_spec taps; the non-spec update operates on the
+// LEADING K-1 taps at the row's PHYSICAL stride `state_len`
+// (cpu_ops.cpp:1088-1092, mirroring vLLM causal_conv1d.py:66-69). At num_spec == 0
+// state_len == width and the addressing is byte-identical.
+//
+// A NEGATIVE conv_state_index IS NOT AN ERROR: it is upstream's NULL block and
+// the token is SKIPPED (cpu_ops.cpp:1103), leaving both the output element and
+// the cache row untouched.
+//
+// ACCUMULATION ORDER IS THE CPU'S: bias, then taps j = 0..K-2 in order, then the
+// x term. Kept sequential on one invocation rather than tree-reduced across
+// lanes precisely so this kernel stays in the bit-comparable tier with its
+// reference instead of only the NMSE tier.
+#extension GL_GOOGLE_include_directive : require
+#include "vt_common.glsl"
+
+layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in;
+
+layout(binding = 0) buffer Ob32 { uint v[]; } O32;
+layout(binding = 1) buffer Ob16 { uint16_t v[]; } O16;
+layout(binding = 2) readonly buffer Xb32 { uint v[]; } X32;
+layout(binding = 3) readonly buffer Xb16 { uint16_t v[]; } X16;
+layout(binding = 4) readonly buffer Wb32 { uint v[]; } W32;
+layout(binding = 5) readonly buffer Wb16 { uint16_t v[]; } W16;
+// bias; aliases the weight when the caller passed none (p.has_bias == 0).
+layout(binding = 6) readonly buffer Bb32 { uint v[]; } B32;
+layout(binding = 7) readonly buffer Bb16 { uint16_t v[]; } B16;
+layout(binding = 8) buffer Sb { uint v[]; } ST;    // conv_state, f32
+// conv_state_indices, i32; aliases conv_state when absent (p.has_idx == 0).
+layout(binding = 9) readonly buffer Ib { uint v[]; } IDX;
+
+layout(push_constant) uniform Params {
+  uint batch;        // x.shape[0]
+  uint c_dim;        // x.shape[1]
+  uint k;            // weight.shape[1]
+  uint width;        // k - 1
+  uint state_len;    // conv_state.shape[2] — the PHYSICAL row width
+  uint x_rs;         // x.stride[0], in elements (x may be a padded-row view)
+  uint n_state_rows; // conv_state.shape[0], for the bounds guard
+  uint has_bias, has_idx, silu;
+  uint out_dt, x_dt, w_dt, bias_dt;
+  uint out_off, x_off, w_off, bias_off, st_off, idx_off;
+} p;
+
+void main() {
+  uint gid = gl_GlobalInvocationID.x;
+  if (gid >= p.batch * p.c_dim) { return; }
+  uint bt = gid / p.c_dim;
+  uint c = gid % p.c_dim;
+
+  uint srow_row = bt;
+  if (p.has_idx != 0u) {
+    int ci = int(IDX.v[(p.idx_off >> 2) + bt]);
+    if (ci < 0) { return; }                          // NULL block, skip
+    if (uint(ci) >= p.n_state_rows) { return; }      // device analogue of the
+    srow_row = uint(ci);                             // host-side range VT_CHECK
+  }
+  // cpu_ops.cpp:1109-1111 — raw pointer arithmetic over the f32 state, so the
+  // row base is srow_row * c_dim * state_len + c * state_len.
+  uint srow = (p.st_off >> 2) + (srow_row * p.c_dim + c) * p.state_len;
+
+  float xt = VT_LOAD(X32, X16, p.x_dt, p.x_off, bt * p.x_rs + c);
+  float acc = p.has_bias != 0u ? VT_LOAD(B32, B16, p.bias_dt, p.bias_off, c) : 0.0;
+  for (uint j = 0u; j < p.width; ++j) {
+    acc += VT_LOAD(W32, W16, p.w_dt, p.w_off, c * p.k + j) * uintBitsToFloat(ST.v[srow + j]);
+  }
+  acc += VT_LOAD(W32, W16, p.w_dt, p.w_off, c * p.k + p.width) * xt;
+  // Silu spelled `acc / (1 + exp(-acc))`, the CPU reference's spelling
+  // (cpu_ops.cpp:916) — `acc * sigmoid(acc)` is a different rounding.
+  VT_STORE(O32, O16, p.out_dt, p.out_off, bt * p.c_dim + c,
+           p.silu != 0u ? (acc / (1.0 + exp(-acc))) : acc);
+
+  for (uint j = 0u; j + 1u < p.width; ++j) { ST.v[srow + j] = ST.v[srow + j + 1u]; }
+  if (p.width > 0u) { ST.v[srow + p.width - 1u] = floatBitsToUint(xt); }
+}

src/vt/vulkan/shaders/vt_gdn_post_conv.comp

@@ -0,0 +1,136 @@
+#version 450
+// vt::GdnPostConv — the fused GDN post-convolution preamble: split the mixed
+// q|k|v conv output, L2-normalize q and k per head, copy v, and derive the decay
+// g and the gate beta from the raw a/b projections. ONE launch in place of
+// GdnConvSplit + L2Norm(q) + L2Norm(k) + GdnGBeta.
+// Ported 1:1 from src/vt/cpu/cpu_ops.cpp:2337-2417 GdnPostConvKernel; the fusion
+// itself mirrors upstream
+// vllm/model_executor/layers/fla/ops/fused_gdn_prefill_post_conv.py
+// `_fused_post_conv_kernel`, whose grid is (L, H + HV).
+//
+// THE GRID IS UPSTREAM'S, and that is what makes one dispatch cover four
+// different shapes of work: workgroup (i, j) with j < Hk does the q/k L2-norm for
+// head j of token i, and j >= Hk does the v copy AND the g/beta gating for head
+// j - Hk. `j` is derived from gl_WorkGroupID, so it is UNIFORM across the
+// workgroup — which is required, because both branches contain barriers
+// (vt_tg_sum) and a barrier reached by only part of a workgroup is undefined.
+//
+// FOUR OPERANDS ARE BOUND THROUGH THE 32-BIT VIEW ONLY. g_out, beta_out, a_log
+// and dt_bias are f32 BY CONTRACT (src/vt/ops.cpp:3459-3463), so a dtype-erased
+// pair of bindings would be a branch that can never take its other side; the
+// three axes that genuinely vary — conv, the q/k/v triple, and the a/b pair —
+// are specialization constants.
+//
+// SOFTPLUS IS COMPUTED WITH A REAL log1p. GLSL has no log1p and
+// `log(1.0 + exp(x))` loses the whole value once exp(x) drops below 2^-24: the
+// CPU reference is `std::log1p(std::exp(x))` (cpu_ops.cpp:2410) and a shader that
+// returned 0 where the oracle returns 9e-14 would be wrong in the only regime
+// this branch exists for. The identity below is the standard cancellation-free
+// form (Goldberg): scale log(u) by y/(u-1), which is exactly 1 in the regime
+// where u-1 == y and corrects the rounding of `1 + y` otherwise.
+//
+// The reduction is a workgroup tree, so the L2 sum-of-squares does NOT share the
+// CPU's accumulation order: NMSE tier, like every other reducing kernel here.
+// The scale is `1.0 / sqrt(ss + eps)` — a PLAIN sum of squares, not a mean
+// (gdn-semantics.md §4) — and not `inversesqrt`
+// (vt_common.glsl § DELIBERATE DIVERGENCES).
+#extension GL_GOOGLE_include_directive : require
+#include "vt_common.glsl"
+
+layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in;
+
+layout(binding = 0) readonly buffer Cb32 { uint v[]; } C32;   // conv
+layout(binding = 1) readonly buffer Cb16 { uint16_t v[]; } C16;
+layout(binding = 2) buffer Qb32 { uint v[]; } Q32;
+layout(binding = 3) buffer Qb16 { uint16_t v[]; } Q16;
+layout(binding = 4) buffer Kb32 { uint v[]; } K32;
+layout(binding = 5) buffer Kb16 { uint16_t v[]; } K16;
+layout(binding = 6) buffer Vb32 { uint v[]; } V32;
+layout(binding = 7) buffer Vb16 { uint16_t v[]; } V16;
+layout(binding = 8) readonly buffer Ab32 { uint v[]; } A32;   // araw
+layout(binding = 9) readonly buffer Ab16 { uint16_t v[]; } A16;
+layout(binding = 10) readonly buffer Bb32 { uint v[]; } B32;  // braw
+layout(binding = 11) readonly buffer Bb16 { uint16_t v[]; } B16;
+layout(binding = 12) buffer Gb { uint v[]; } G;               // g_out, f32
+layout(binding = 13) buffer Eb { uint v[]; } BETA;            // beta_out, f32
+layout(binding = 14) readonly buffer Lb { uint v[]; } ALOG;   // a_log, f32
+layout(binding = 15) readonly buffer Tb { uint v[]; } DTB;    // dt_bias, f32
+
+layout(constant_id = 0) const uint VT_PC_CONV_DT = VT_DT_F32;
+layout(constant_id = 1) const uint VT_PC_QKV_DT = VT_DT_F32;
+layout(constant_id = 2) const uint VT_PC_AB_DT = VT_DT_F32;
+
+layout(push_constant) uniform Params {
+  uint t;
+  uint hk, dk;
+  uint hv, dv;
+  uint key_dim;    // hk * dk
+  uint value_dim;  // hv * dv
+  uint conv_dim;   // 2 * key_dim + value_dim
+  uint a_rs, b_rs; // araw/braw stride[0], in elements (may be padded)
+  uint conv_off, q_off, k_off, v_off, a_off, b_off;
+  uint g_off, beta_off, alog_off, dtb_off;
+  float eps;
+} p;
+
+// log1p(y) for y >= 0, cancellation-free. See the header note.
+float vt_log1p(float y) {
+  float u = 1.0 + y;
+  return u == 1.0 ? y : log(u) * (y / (u - 1.0));
+}
+
+void main() {
+  uint tid = gl_LocalInvocationID.x;
+  uint lanes = p.hk + p.hv;
+  uint i = gl_WorkGroupID.x / lanes;  // token
+  uint j = gl_WorkGroupID.x % lanes;  // head slot, UNIFORM per workgroup
+  uint row = i * p.conv_dim;
+
+  if (j < p.hk) {
+    // q/k: split then L2-norm over Dk, per head (cpu_ops.cpp:2350-2366).
+    uint h = j;
+    uint qbase = row + h * p.dk;
+    uint kbase = row + p.key_dim + h * p.dk;
+    float qp = 0.0;
+    float kp = 0.0;
+    for (uint e = tid; e < p.dk; e += VT_TG) {
+      float qv = VT_LOAD(C32, C16, VT_PC_CONV_DT, p.conv_off, qbase + e);
+      float kv = VT_LOAD(C32, C16, VT_PC_CONV_DT, p.conv_off, kbase + e);
+      qp += qv * qv;
+      kp += kv * kv;
+    }
+    float qss = vt_tg_sum(tid, qp);
+    float kss = vt_tg_sum(tid, kp);
+    float qinv = 1.0 / sqrt(qss + p.eps);
+    float kinv = 1.0 / sqrt(kss + p.eps);
+    for (uint e = tid; e < p.dk; e += VT_TG) {
+      uint o = (i * p.hk + h) * p.dk + e;
+      VT_STORE(Q32, Q16, VT_PC_QKV_DT, p.q_off, o,
+               VT_LOAD(C32, C16, VT_PC_CONV_DT, p.conv_off, qbase + e) * qinv);
+      VT_STORE(K32, K16, VT_PC_QKV_DT, p.k_off, o,
+               VT_LOAD(C32, C16, VT_PC_CONV_DT, p.conv_off, kbase + e) * kinv);
+    }
+    return;
+  }
+
+  uint h = j - p.hk;
+  // v: plain copy of this head's slice (cpu_ops.cpp:2368-2370).
+  for (uint e = tid; e < p.dv; e += VT_TG) {
+    VT_STORE(V32, V16, VT_PC_QKV_DT, p.v_off, i * p.value_dim + h * p.dv + e,
+             VT_LOAD(C32, C16, VT_PC_CONV_DT, p.conv_off,
+                     row + 2u * p.key_dim + h * p.dv + e));
+  }
+  // g/beta from a/b + A_log/dt_bias (gdn-semantics.md §6, cpu_ops.cpp:2372-2381).
+  // One element per head, so one lane does it; the rest of the workgroup is
+  // already busy with the v copy above.
+  if (tid == 0u) {
+    uint idx = i * p.hv + h;
+    float x = VT_LOAD(A32, A16, VT_PC_AB_DT, p.a_off, i * p.a_rs + h) +
+              uintBitsToFloat(DTB.v[(p.dtb_off >> 2) + h]);
+    float sp = x > 20.0 ? x : vt_log1p(exp(x));  // softplus, threshold 20
+    float alog = uintBitsToFloat(ALOG.v[(p.alog_off >> 2) + h]);
+    G.v[(p.g_off >> 2) + idx] = floatBitsToUint(-exp(alog) * sp);
+    float braw = VT_LOAD(B32, B16, VT_PC_AB_DT, p.b_off, i * p.b_rs + h);
+    BETA.v[(p.beta_off >> 2) + idx] = floatBitsToUint(vt_sigmoid(braw));
+  }
+}