perf(vulkan): lm_head column blocking -1.07ms/tok; the 9.3ms floor I briefed does not exist

[localai-bot]

The brief's premise was wrong, and that is the biggest finding

I briefed this row with a 9.3 ms floor for the lm_head GEMM: 2.543 GB at
GB10's 273 GB/s. That floor does not exist, because nothing on this box
reaches 273 GB/s. Running vt_matmul_vec — a genuinely coalesced streaming
kernel — on the identical byte count measures 11.04 ms = 230.3 GB/s.
That is the practical ceiling. Real headroom was 1.4 ms, not the 3.1 I
claimed.

Every "% of roof" figure this campaign has quoted against 273 GB/s is optimistic
by the same ~16%.

Measured

lever	result
K-unroll (the lever I briefed as highest-confidence)	ZERO. 12.430 rolled vs 12.532 unrolled ms/call, 3 replicates each — half a percent the wrong way. Not shipped.
Column blocking (the row's own idea)	12.485 → 11.541 ms/call, 6 of 6 interleaved pairs. Two-length diff −1.066 ms/token. 203.7 → 220.4 GB/s = 95.7% of the measured ceiling.

VT_VULKAN_MATMUL_NCOLS gives a workgroup 128×NCOLS consecutive output
columns, with lane t owning columns t + c*128 (not t*NCOLS + c) so every
load stays coalesced. Blocking is a trade, not monotone: NCOLS 8 is slower
than not blocking at all (243 workgroups starves latency hiding) and NCOLS 2
ties. The optimum is interior, at 4.

Byte-exactness of the general matmul path is preserved, and known by memcmp
rather than by argument: each accumulator owns one output element and sums the
whole K sequentially. The gate compares both arms bitwise in one process. Two
scratch mutations bite — and notably an earlier version of the fixture
survived the reassociation mutation because its operands were exact in f32,
so the fixture now spans decades.

Prefill: zero impact. vt_matmul runs exactly once per output token; prefill
takes coopmat. Blocking is host-gated to bt == 0 regardless.

e2e: 8 AB/BA pairs, wins 7/8 and 3/4 clean pairs, 242.67 → 241.90 ms TPOT.
0.3% of wall, at the noise edge, so the GPU-timestamp number is the binding one.

The bimodal collapse is REAL — this refutes what I told two agents

I twice relayed that the ~1.8x bimodal legs were self-inflicted contamination
from concurrent 27B benchmarking. That is now refuted by direct measurement.

Reproduced with the box exclusively this row's, flock ~/gpu.lock held on every
27B run: 16 legs, MemAvailable 119,024–119,276 MB of 119 GB before every leg.
One leg hit 259.62 ms/call and is indistinguishable from its healthy neighbours
on every memory metric — RSS 105,994,272 KB vs 105,995,280, major faults 1098 vs
1090, TTFT 6346 ms vs 6273 — and it started with more free memory.

Refuted: co-residency, page reclaim, load and prefill effects. Separately,
four legs with 46k–304k major faults (40–270× normal) read only 12.46–13.92 ms, so
heavy paging costs ~10%, not 20×.

A cheap reproducer (nn orientation + realloc cycles in vulkan_gemm_ab.cpp)
reaches the same kernel on the same 2.543 GB in seconds instead of a 3-minute /
106 GB leg: 144 measurements, all 12.39–13.96 ms, zero collapses on an idle
box.

Inferred, not established: the one variable left is that the 27B holds 106 GB
RSS of 119 GB while the reproducer holds 2.5 GB — near-ceiling occupancy, not
contention. The ballast test that would settle it was not run because that has
OOM-rebooted this box.

Suggestive, explicitly not a claim: every collapse landed on the flat arm
(NCOLS 1: 3/23 legs; NCOLS 4/8: 0/31), Fisher one-sided p ≈ 0.07. Needs ~60 more
legs.

And it is not "bigger than every other lever", which is also my
overstatement: at 1 leg in 16 it costs ~6% of mean decode time, against the 8.5%
column blocking removes deterministically.

Gates, verified by the operator on the MERGED tree

The three-lever combination is something no row tested. On a clean llvmpipe build
of the merge: test_vulkan_backend 32/32, 1905 assertions (all three rows'
mechanism tests present), test_backend_cross_device 11/11,
VLLM_CPP_DEVICE=vulkan test_opt_paged_engine 6/6 token-exact (96/96), 0
declines, and gen-vulkan-spirv.py --check reproduces the committed SPIR-V
byte-for-byte under pinned glslang 16.5.0.

Row-reported on GB10, not re-run here: test_vulkan_backend 30/30 (2369),
test_backend_cross_device 11/11, test_opt_paged_engine 6/6 token-exact.

Conflicts, resolved by kind

Eight, none by automatic three-way combine. Union for the append-only record, the
spec narrative, ENVIRONMENT.md rows and the test cases; merge-by-key for
STATUS.md and the BENCHMARKS.md row (trimmed to the 220-char cell bound). In
vulkan_context.{h,cpp} both rows had added a near-identical pipeline-key
accessor under different names — kept the general PipelineKeys() as the one
taking the mutex and reimplemented PipelineKeysFor() as a filter over it, rather
than shipping two walks of the same cache. A resolution script asserted anchor
uniqueness, which caught that ======= appears nine times as a comment rule in
the test file.

Known red, named rather than papered over

check-pr-size counts the regenerated vulkan_spirv.cpp as reviewable product
code; docs/STATUS.md remains over its shrink-only ratchet, pre-existing on main.

FOLLOWING_AGENTS_PROTOCOL