perf(vulkan): decode GEMV reads four elements per load, and the row lever is dead

[localai-bot]

vt_matmul_vec is 89-90% of 27B decode GPU time and moves ~52 GB of bf16
weights per token, so it was the largest single line item left. Two structural
differences against llama.cpp's mul_mat_vec (pin 237ad9b96) were the
candidates. One is worth about 1%, the other is a measured LOSS on this
device, and the win is 7x smaller than the isolated sweep first said. All
three of those are the result.

What shipped

VT_MM_PACK, a specialization constant selecting the LOAD WIDTH for a 16-bit
operand: one element per load (2 B, the previous behaviour), two through the
buffer's existing 32-bit view, or four through a new 64-bit view of the same
VkBuffer. Ported in intent from llama.cpp's data_b_v4 /
dequantize4_2aligned. Same bytes, same coalescing, so this cannot reduce
DRAM traffic; it reduces load instructions, four to one. Default 2, degrading a
width at a time when K or an operand byte offset is not aligned, and declining
for an f32 operand.

What did NOT work, with its number

VT_MM_ROWS, llama.cpp's NUM_ROWS (it sets this per vendor via rm_stdq),
computing 2 or 4 output elements per workgroup. Measured 0.966x at 2 rows and
0.968x at 4 — a loss in every pass at every load width. It ships OFF. What it
was supposed to buy was never traffic (weights are read exactly once per token at
any row count) and the activation re-reads it saves were L2 hits; halving the
workgroup count halves the independent sequential read streams the memory
controller sees, which is the likely cost. The axis is kept, off by default,
because llama.cpp raises it exactly on the AMD GCN and Intel parts this backend's
portability case is aimed at. It is bit-identical to rows=1 at any value,
verified by full-output bit-hash rather than by argument.

Measured

Isolated sweep (benchmarks/vulkan_gemv_ab.cpp, 7 real 27B shapes, 9 arms x 4
rotated passes, each arm paired against the baseline in the same pass because
the box drifted 15.5% peak-to-peak between passes):

	pack 0	pack 1	pack 2
rows 1	1.000x	1.025x	1.086x
rows 2	0.966x	1.017x	1.047x
rows 4	0.968x	1.014x	1.039x

Real 27B decode, the number of record — two-length GPU-timestamp diff (len 36
minus len 4, over 32 tokens) so prefill and one-time costs cancel, run twice with
arm order reversed:

per decoded token	blk1 pack0	blk1 pack2	blk2 pack0	blk2 pack2
vt_matmul_vec ms	211.99	209.52	214.59	210.38
GB/s (52.1 GB/token)	245.8	248.7	242.8	247.6
% of 273 GB/s roof	90.0%	91.1%	88.9%	90.7%

1.012x and 1.020x, not 1.086x. The sweep was in the wrong regime: it re-reads
one 356 MB buffer 320 times, so its DRAM rows and TLB stay hot and instruction
issue becomes visible; decode streams 50 GB of distinct weights once per token,
where DRAM is the whole story. The campaign already has "a negative result is
regime-dependent" on record; this is the same lesson running the other way — a
win can be a regime artefact too.

e2e AB/BA: 14 pairs across two blocks; 6 pairs had both legs clean, pack=2 won
5 of 6, median paired 1.0064x, clean-leg medians 242.38 -> 240.54 ms =
4.126 -> 4.157 tok/s. Measured noise floor on clean legs 0.88% (baseline arm)
and 1.92% (pack arm), so 0.7% is at the edge of e2e resolvability and it is the
GPU-timestamp diff that resolves it. Stated plainly rather than dressed up.

The GEMV lever is essentially empty and now we know why. The binding
constraint is DRAM bandwidth on a ~50 GB working set streamed once per token —
not instruction issue, not latency, not activation re-reads.

The finding worth more than the 1%

The ~1.8x bimodal decode legs this campaign has repeatedly discarded as
environmental are vt_matmul — the lm_head — collapsing 19.5x: 12.35 ->
242.12 ms/token, i.e. 205 GB/s -> 10.5 GB/s on 2.54 GB of weights.
vt_matmul_vec beside it moved 1.2% in the same legs; TTFT is unchanged across
the modes (6.21-6.55 s over all 28 legs), so it is neither load nor prefill. It
hit 8 of 28 legs, arm-symmetric. This is one kernel entering a slow state, not
the environment, and it is a bigger lever than every remaining Vulkan kernel
optimisation combined. Being chased separately.

Gates, verified by the operator rather than taken on report

Re-run independently on a clean llvmpipe build:

• test_vulkan_backend 30/30, 1823 assertions (was 29/1786; the new case and
  37 assertions are the variant-selection test)
• test_backend_cross_device 11/11, 132 assertions
• VLLM_CPP_DEVICE=vulkan test_opt_paged_engine 6/6 prompts token-exact
  (96/96 tokens), 0 declines
• gen-vulkan-spirv.py --check under pinned glslang 16.5.0: committed SPIR-V is
  up to date

Agent-reported and not re-run by me: GB10 test_vulkan_backend 30/30 with 2366
assertions; all 9 (rows, pack) arms verified numerically; scratch mutation
(reverting kGemvPackDefault to 0) turns the selection test red 6/33 and green
on restore.

Known-red gates, named rather than papered over

check-pr-size counts the regenerated vulkan_spirv.cpp as reviewable product
code — the same checker gap every Vulkan shader PR hits. check-public-doc-tables
still fails on the docs/STATUS.md ratchet, which main already violates and this
branch leaves byte-neutral. The other preflight failures (undocumented
VT_GEMMA4_*/VT_ROCM_* vars, trailer commit 39da53c7) are pre-existing on the
base.

FOLLOWING_AGENTS_PROTOCOL