perf(vulkan): ragged M reaches coopmat — 27B prefill 21.5x - #162
Conversation
The largest win of this campaign, and it was self-inflicted. A prefill-dominated 27B run profiled with GPU timestamps put vt_matmul -- the UNTILED SCALAR kernel -- at 433 calls, 409,611.9 ms, 99.9% of GPU time, 945.99 ms/call. That is ~96 GFLOP/s, about 1% of this device. GPU busy was 95% of wall, so prefill was never host-bound and the remaining reference-tier ops were ~0.1%. Three prior structural attributions were wrong; one timestamp profile settled it. WHY COOPMAT DECLINED HAD TO BE MEASURED. Both candidates a reader would guess were excluded by inspection: every dimension is a whole tile (5120, 17408, 256) and activations are bf16. So the predicate was made to report itself under VT_VULKAN_DISPATCH_STATS, and it named the clause immediately: coopmat DECLINED: M is not a multiple of 16 (a.dtype=2 b.dtype=2 m=17 k=5120 n=10240) m = tokens + 1, so a 512-token prompt gives 513 and 513 % 16 == 1. EVERY prefill GEMM at every prompt length fell to the scalar kernel. THE CAUSE WAS THE EARLIER HANG FIX. coopMatLoad reads a full 16x16 tile unmasked, so at M=1 it read ~30 KB past the activation buffer and the fence never signalled. That was fixed by requiring m % 16 == 0 && n % 16 == 0 -- correct, and it is why the 27B runs instead of hanging. The note left at the time said a masked or padded load was the better long-term answer and was not attempted. That deferred work WAS the prefill bottleneck, hidden because the only model then running on Vulkan was opt-125m, whose small GEMMs made the scalar path cheap. THE FIX SHIFTS THE TRAILING TILE BACK rather than masking. A tile that would overrun M slides down to start at M-16 and therefore reads only real rows. Exact, not approximate: a result row depends solely on that row of A and on B, never on which tile computed it, so rows shared with the previous tile recompute to bit-identical values and the duplicate stores write the same bytes. It needs M >= 16, which the predicate now requires in place of M % 16 == 0; below that there is no in-bounds window, and decode (M=1) is served by the GEMV tactic. MEASURED, GB10, Qwen3.6-27B, 512-token prefill: GEMM ms/call 945.99 -> 30.40 31.1x prefill tok/s 1.18 -> 25.41 21.5x E2E ms 433,321 -> 20,192 21.5x vt_matmul is gone from the profile. 21.5x is far outside this box's 2.1x noise band, so it is callable at n=1. A 512-token prefill is now 20.1 s against llama.cpp's ~1.1 s -- 18x behind, down from 244x. Gate 26/26, 1563 assertions ON GB10; the new ragged-M case asserts the TACTIC and exactness at M=17 and short-circuits on llvmpipe, which has no coopmat (local run shows 1020). opt-125m STRICT 6/6 token-exact. Durable lesson: a correctness fix can silently become a performance cliff, and a selection predicate able to route an entire model onto the correctness tier should be able to say so. That diagnostic now ships behind the stats flag. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
1047901 to
d827b33
Compare
|
LANDED on Landing required repairing two gates that were red on main and blocking every push ( Two README rows corrected in passing were false, not merely long: Vulkan claimed "Skeleton: 8 ops… No model runs yet" while News two screens above said a model runs end-to-end, and ROCm claimed HIP sources were "never compiled" — untrue since #140. Gates on the merge: trailer contract OK, Closing per the PR-disposition protocol. |
2 participants
27B prefill 21.5×. The largest win of this campaign, and it was self-inflicted.
How it was found
Two GDN increments moved coverage 11→3 and speed not at all, so a prefill-dominated run was profiled with GPU timestamps:
The untiled scalar kernel — the portable correctness tier — was carrying every prefill GEMM at ~96 GFLOP/s, about 1% of what GB10 can do. GPU busy was 95% of wall, so prefill was never host-bound and the remaining reference-tier ops were ~0.1%. Three prior structural attributions were wrong; one profile settled it.
Why coopmat declined had to be measured, not reasoned
Both candidates a reader would guess were excluded by inspection: every dimension is a whole tile (5120, 17408, 256) and activations are bf16 (
DBuf dx(d, DType::kBF16, ...)).So the predicate was made to report itself, behind
VT_VULKAN_DISPATCH_STATS:m = tokens + 1. A 512-token prompt gives 513, and513 % 16 == 1. Every prefill GEMM, at every prompt length, fell to the scalar kernel.The cause was the earlier hang fix
coopMatLoadreads a full 16×16 tile unmasked, so at M=1 it read ~30 KB past the activation buffer and the fence never signalled. That was fixed by requiringm % 16 == 0 && n % 16 == 0— correct, and it is why the 27B runs instead of hanging.The note left at the time: "a masked/padded load is the better long-term answer, not attempted here." That deferred work was the prefill bottleneck, hidden because the only model then running on Vulkan was opt-125m, whose small GEMMs made the scalar path cheap.
The fix shifts the trailing tile back rather than masking
A tile that would overrun M slides down to start at
M-16, so it reads only real rows.Exact, not approximate: a result row depends solely on that row of A and on B, never on which tile computed it — so rows shared with the previous tile recompute to bit-identical values and the duplicate stores write the same bytes. Needs
M >= 16, which the predicate now requires instead ofM % 16 == 0; below that there is no in-bounds window, and decode (M=1) is served by the GEMV tactic.Measured — GB10, Qwen3.6-27B, 512-token prefill
vt_matmulis gone from the profile. 21.5× is far outside this box's 2.1× noise band, so it is callable at n=1. A 512-token prefill is now 20.1 s against llama.cpp's ~1.1 s — 18× behind, down from 244×.Gates
26/26, 1563 assertions on GB10. The new ragged-M case asserts the tactic (
PipelineExistsFor("vt_matmul_coopmat")) and exactness at M=17 — and it short-circuits on llvmpipe, which has no coopmat, so the local run shows 1020 assertions and the GB10 run 1563. A gate for a feature CI's device lacks has to be run where the feature exists, or it proves nothing.opt-125mSTRICT 6/6 token-exact.🤖 Generated with Claude Code