VK-C bench: coopmat GEMM is 11.1x-32.9x over our scalar kernel on Thor

[localai-bot]

The VK-C speed A/B and the harness that produced it. Stacked on #97 (the coopmat tactic itself) — merge that first.

Split out of #97 because check-pr-size flagged 969 lines against a 900 cap; the seam is the tactic vs measuring it.

Result: 11.1x–32.9x on NVIDIA Thor

M × K × N	COOPMAT	scalar	speedup	coopmat GFLOP/s
128 × 1024 × 1024	0.495 ms	5.470 ms	11.1×	542
512 × 2048 × 2048	2.596 ms	81.997 ms	31.6×	1,655
1024 × 4096 × 4096	19.896 ms	654.181 ms	32.9×	1,727

worst rel err = 0.000e+00 in every arm.

How the arms are kept honest

Same binary, one variable. Both arms run examples/vulkan-gemm-ab; the only difference is VT_VULKAN_COOPMAT, which forces the scalar tactic when 0. Comparing two builds would confound the kernel with everything else that differs between them.

Each arm proves which tactic it ran. PipelineExistsFor is checked and printed. Without that, a silent fallback posts a perfectly plausible pair of timings that mean nothing — the scalar kernel is equally correct, so numbers alone cannot tell the arms apart. This is the same failure mode the op-provider decline counters exist for, and it already nearly slipped through this row's correctness gate.

Each arm verifies against a host oracle, so a tactic cannot be fast by being wrong.

The lever is default-ON

VT_VULKAN_COOPMAT unset means the capability probe decides — nobody has to set anything to get the fast path. It only disables, and it lives in the documented "rollback and bisect switches" section: a default-on fast path with an off switch, so a suspect coopmat result can be bisected without a rebuild.

Thor is a second coopmat device

Not a re-run of the first. Thor exposes the same 11 configurations as GB10, including 16x16x16 bf16/bf16/f32/f32 SUBGROUP at subgroup size 32 — so the capability probe and both selection preconditions are now confirmed on a differently-architected board (Jetson aarch64, sm_110) as well as the gate box.

What this number is NOT — and it matters more than the number

The baseline is our own vt_matmul.comp, which is deliberately naive and untiled: one invocation per output element, sequential f32 accumulation, no shared-memory blocking, written to match the CPU's accumulation order and serve as the portable correctness tier.

So this measures tensor cores against a naive scalar loop, not our coopmat against a competent scalar GEMM. A tiled scalar kernel would close much of this gap by itself. The honest reading is "the coopmat tactic is worth having and is correctly selected", not "we are 32× faster than anyone".

It is also not a competitor comparison. llama.cpp's Vulkan backend is untouched here; that is BENCH-VK-LLAMA / VK-E, still owed.

Speedup growing with size (11× → 33×) is what a tensor-core path should do: fixed per-dispatch cost amortises and arithmetic intensity rises.

Thor access, recorded because it cost time

• Kairos is immutable — no cmake, no apt. A user-local cmake tarball in ~/tools is the way in.
• GPU access needs root: the NVIDIA memory manager returns permission denied for the normal user (NvRmMemInitNvmap failed).
• The documented docker route does not work for Vulkan there — the ICD is not injected into the container even with NVIDIA_DRIVER_CAPABILITIES=all, so instance creation fails. sudo on the host is the working path.
• No container held the GPU (all exited), so nothing was stopped and nothing needed restoring.

🤖 Generated with Claude Code