perf(vulkan): fused attn preamble native -- 16 host round trips/token gone

.agents/benchmark-record.md

@@ -16177,3 +16177,16 @@ the ~30 flushes/token. The declines name exactly three ops: `kCausalConv1dFwd`
 (op 5, prefill only), `kRopeCosSinCache` (op 66) and `kAttnQkNormRopeGate`
 (op 67). The last two run per full-attention layer, 16 per token, and are the
 next lever; they are kernel work, not plumbing.
+
+## Rolled out of the scoreboard on 2026-08-09
+
+Moved verbatim from `docs/BENCHMARKS.md` by `scripts/roll-benchmark-record.py`. Nothing edited or deleted.
+
+## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700)
+
+| Path | Warm tok/s | Notes |
+|------|------------|--------|
+| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill |
+| server `/v1/completions` | ~38 | exclusive |
+| server `/v1/chat` thinking off | ~32 | after expert cache |
+| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack |

.agents/specs/vulkan-full-support.md

@@ -327,11 +327,84 @@ the umbrella, not a substitute for them.
 | **VK-H** | **Attention variants + samplers** (16 ops) | B (samplers), G (attn variants) | **83/83 — closes the op surface** |
 | **VK-I** | **AMD/RDNA (or Arc) bring-up** | hardware acquisition | The staging path for non-host-visible memory, and the gate re-run where Vulkan actually matters |
 
+### 6.0a `VK-G` partial: the FUSED ATTN PREAMBLE landed — 2026-08-09
+
+`row/BACKEND-VULKAN-QKNORM`. `kAttnQkNormRopeGate` — gemma-RMSNorm(q) +
+gemma-RMSNorm(k) + partial NeoX RoPE from the precomputed cos/sin cache + the
+gate passthrough — is native. Module count 24 -> 25.
+
+**Why this op, and only this op.** After the GDN rows there was no kernel-speed
+lever left in decode: `vt_matmul_vec` runs at 90% of the GB10 bandwidth roof and
+`lm_head` at 74%. What remained was the reference tier's ARCHITECTURAL cost —
+`src/vt/op_provider.cpp` drains the recorded command batch (submit + blocking
+fence) before it can hand a host kernel device memory, so a reference-tier op
+costs a full GPU round trip no matter how little arithmetic it does. Three ops
+declined; `kCausalConv1dFwd` is prefill-only and `kRopeCosSinCache` is
+deliberately host-side (the double-precision table, vLLM's own split), which left
+this one. The 27B has 64 layers of which 48 are linear-attention, so it fires 16
+times per decoded token.
+
+**Ported from.** Per-element math 1:1 from our own CPU reference
+`src/vt/cpu/cpu_ops.cpp:956-1010 AttnQkNormRopeGateKernel` (itself the transcription of
+vLLM's `QKNormRoPEFusionPass` -> `_C.fused_qk_norm_rope`); dispatch shape from our
+CUDA kernel `src/vt/cuda/cuda_ops.cu:1316-1394` — one workgroup per (token, head)
+over Hq + Hkv slots, CUDA's `dim3(t, hq+hkv)` grid flattened, with the two paired
+normed elements RECOMPUTED per output rather than staged in shared memory. The
+reduction idiom is `vt_rms_norm_gated.comp`'s and the cos/sin indexing is
+`vt_rope_from_cache.comp`'s, minus the positions indirection: this cache is the
+per-step `[T,rot]` fill, so the row is `tok * rot`.
+
+**MEASURED on GB10 (Qwen3.6-27B bf16, Vulkan, 1 prompt, 32-in, c1, page cache
+dropped before every run).** By the two-length flush diff (output-len 4 vs 12,
+both arms measured, neither inferred):
+
+| per decoded token | main | this row |
+|---|---:|---:|
+| reference-tier drains | 16 | **0** |
+| command-buffer flushes, all reasons | 18 | **4** |
+| GPU dispatches | 884 | 900 |
+| GPU-active | 236.3 ms | 234.6 ms |
+| wall (median TPOT) | 253.5 ms | 246.2 ms |
+| host time, wall minus GPU-active | 17.2 ms | **11.6 ms** |
+
+That is the whole mechanism in one table. The GPU does the SAME work to within
+noise and takes four more dispatches per token; what changes is that 16 submit +
+blocking-fence round trips per token stop happening. The four remaining flushes
+are the batcher's own copy-src and descriptor-ring-full drains, not provider
+declines, and the reference-tier decline list is now exactly `kRopeCosSinCache`
+and `kCausalConv1dFwd`, both by design.
+
+**PAIRED DECODE**, 6 order-alternated AB/BA pairs at 32-in/32-out, page cache
+dropped before each of the 12 runs: **this row wins 5 of 6**. Median TPOT over
+the non-outlier legs 249.74 ms -> 242.34 ms, i.e. decode **4.00 -> 4.13 tok/s**
+against llama.cpp Vulkan's 4.35. Two legs (one per arm, pairs 2 and 3) came in at
+~446-448 ms, 1.8x the block median; discarding the pairs that contain them leaves
+**4 of 4**, and it is the same ~3% either way.
+
+**HONESTY ON THE SPEED NUMBER.** The effect is ~3% and this box swings ~2x run to
+run, which is why the readout is a per-pair win rate and not a mean. It is also
+why the first paired block of this row was thrown away: the host had rebooted and
+brought `local-ai-worker` back up under `--restart=always`, and a contended block
+reads ~450-505 ms against a quiet box's ~250 ms, roughly 14x the effect being
+measured. The mechanism table above is the load-bearing evidence — the host-time
+column measures the thing this row removes directly and does not depend on the
+wall clock being reproducible.
+
+**Correctness.** NMSE vs the CPU oracle in the same binary on the GB10 NVIDIA
+driver: q `5.69e-15`, k `1.59e-14` (gemma=1); `6.5e-15` / `1.43e-14` (gemma=0);
+the bf16-q/k + f32-gate arm is BIT-EXACT. The f32 arm is NOT bit-exact and is not
+claimed to be — the workgroup tree reduction changes the mean square's
+accumulation order, the same tier `vt_rms_norm` and `vt_rms_norm_gated` sit in.
+The gate passthrough IS held bit-exact, against the SOURCE rather than against the
+oracle's output. `test_vulkan_backend` 29/29 (2329 assertions) on GB10;
+`test_opt_paged_engine` still 6/6 token-exact (96/96), 0 declines.
+
 ### 6.0 `VK-G` partial: the two GATED-DELTA RECURRENCES landed — 2026-08-08
 
 `row/BACKEND-VULKAN-GDN-CORE`. `kGdnPrefill` and `kGdnDecode` are native; the
 Vulkan module count goes 22 -> 24 and the GDN family 6 -> 8. `kCausalConv1dFwd`,
-`kRopeCosSinCache` and `kAttnQkNormRopeGate` stay on the reference tier.
+`kRopeCosSinCache` and `kAttnQkNormRopeGate` stay on the reference tier (the last
+of the three is closed by §6.0a).
 
 **What was ported, and from where.** Per-step arithmetic 1:1 from
 `src/vt/cpu/cpu_ops.cpp:1280-1311` `GdnHeadTokenStep`; dispatch shape and state

docs/BENCHMARKS.md

@@ -361,7 +361,7 @@ built on it rather than keeping the flattering one.
 | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config |
 | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box |
 | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs |
-| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 24 NATIVE (+8 GDN); 63 host-tier. **27B prefill 21.5x**, decode 2.99->**3.93** (GDN state in place, 98->2 copy flushes/tok, 8/8 paired; target 4.35). opt-125m exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
+| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN); 62 host-tier. **27B prefill 21.5x**, decode 4.00->**4.13** (fused attn preamble native, flushes 18->4/tok, 5/6 pairs; target 4.35). opt-125m exact. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns |
 | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist |
@@ -383,12 +383,3 @@ built on it rather than keeping the flattering one.
 
 Build flags, environment variables, and the full gate list are in
 [BUILD.md](BUILD.md) and [ENVIRONMENT.md](ENVIRONMENT.md).
-
-## 2026-08-08 — Gemma4 FP8 stream lab (gfx1201 R9700)
-
-| Path | Warm tok/s | Notes |
-|------|------------|--------|
-| vllm-cli Paris HIP=0 stream experts | ~38 | `--repeat` after cold expert fill |
-| server `/v1/completions` | ~38 | exclusive |
-| server `/v1/chat` thinking off | ~32 | after expert cache |
-| llama.cpp Vulkan Q8 tg128 (bar) | ~98 | separate stack |

docs/FEATURES.md

@@ -215,11 +215,11 @@ kernels for it.
 Vulkan **runs a model end to end**: `opt-125m` greedy is STRICT token-exact, 6/6
 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle, all nine of that model's ops
 dispatched natively with **zero provider declines**. Qwen3.6-27B runs too, both
-GDN recurrences native and the GDN state cache read and written IN PLACE rather
-than gathered to a host copy per layer per token: **decode 3.93 tok/s against
-llama.cpp Vulkan's 4.35, prefill 21.5x** (GB10). Still partial at 24 native
-kernels plus 8 GDN, the rest on the portable CPU tier (decode still pays for
-`kRopeCosSinCache` and `kAttnQkNormRopeGate`); quant/MoE/MLA have none at all.
+GDN recurrences and the fused attention preamble native, its GDN state cache read
+and written IN PLACE: **decode 4.13 tok/s against llama.cpp Vulkan's 4.35,
+prefill 21.5x** (GB10). Still partial at 25 native kernels plus 8 GDN, the rest
+on the portable CPU tier (decode's one remaining host op, `kRopeCosSinCache`,
+mirrors vLLM's own split and stays there); quant/MoE/MLA have none at all.
 Build with `-DVLLM_CPP_VULKAN=ON`; off by default.
 
 ## Serving, API and operations

docs/STATUS.md

@@ -420,9 +420,9 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact
 ## Not supported yet
 
 LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU,
-Vulkan (opt-125m exact; 24 native +8 GDN, BOTH
-recurrences; 27B prefill 21.5x, decode 3.93/4.35 llama.cpp (GDN state
-in place, 98->2 copy flushes/tok); #125 VERIFIED
+Vulkan (opt-125m exact; 25 native +8 GDN, both
+recurrences + fused attn preamble; 27B prefill 21.5x, decode
+4.13/4.35 llama.cpp, 18->4 flushes/tok; #125 VERIFIED
 [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green
 on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in
 (**blind-written, unverified**); M2 needs verification; gfx1201 hipBLAS +