Two named levers in the CPU backend, both already located by profile and now
corroborated by the first quant-matched measurement (#333, 6dbedf9f).
What the numbers say
Against llama.cpp on the SAME GGUF, idle GB10, 20 ARM cores:
| Workload |
Axis |
ours |
llama.cpp |
ratio |
| in128 t=10 |
prefill |
9.94 |
9.97 |
0.997x TIE |
| in128 t=10 |
decode |
1.31 |
6.41 |
0.204x |
| in512 t=20 |
prefill |
2.23 |
13.13 |
0.170x |
| in512 t=20 |
decode |
0.29 |
5.00 |
0.058x |
llama.cpp is flat from 128 to 512 tokens; our prefill falls 81% and decode
75%. That is the signature of fixed per-op overhead, not slow kernels.
We are not idle. A two-length diff cancelling the load phase puts our decode at
1987% CPU against llama.cpp's 1810% — every core busy, burning 5.6x the
CPU-seconds per token. We are spinning, not starved.
Lever 1 — decode is synchronisation-bound, not kernel-bound
The 2026-08-06 CPU profile (benchmark-record.md):
Threadpool synchronisation is 47.15% of decode (ThreadReady +
PollForWork + Barrier), rising to ~58% on the secondary-thread view.
At M=1 the per-op work is too small to amortise the barrier.
Batch-1 decode issues many tiny ops and pays a full threadpool barrier on each.
src/vt/cpu/cpu_threadpool.{cpp,h}.
Explicitly NOT the defect, per that same profile: the elementwise GEMM
(BtM4Neon/Bt16Neon, 21.5% prefill / 24.9% decode) is already on its optimised
NEON tier and is what remains once avoidable work is gone. No GEMM kernel
change can reach this.
Lever 2 — paged attention branches on dtype per element
~39% of prefill is the paged-attention inner loop, ~21% of it inside a
per-element dtype switch. It is visible at src/vt/cpu/cpu_paged_attn.cpp:29:
float LoadF32(const Tensor& t, int64_t elem_offset) {
switch (t.dtype) {
case DType::kF32: return t.Ptr<float>()[elem_offset];
case DType::kF16: return F16ToF32(t.Ptr<uint16_t>()[elem_offset]);
case DType::kBF16: return BF16ToF32(t.Ptr<uint16_t>()[elem_offset]);A branch per element, in the hot loop, on a value that is constant for the whole
tensor. The profile calls this "a known, already-solved defect class" — hoist the
dispatch out of the loop.
Scope and bar
Both are CPU-backend levers; the CUDA path is untouched. The bar is
llama.cpp on the same GGUF, same harness, bands calibrated before any delta is
read, prefill and decode reported separately. vLLM remains an open gap for
Muse Glimmer and is not the bar here.
Correctness first: every existing CPU gate must stay green, and any change that
alters numerics must show it.
No ceiling is to be declared. The 0.997x prefill tie at 128 tokens shows the
kernels are competitive when overhead is amortised; the gap is what we add per
op, and that is addressable.
Two named levers in the CPU backend, both already located by profile and now
corroborated by the first quant-matched measurement (#333,
6dbedf9f).What the numbers say
Against llama.cpp on the SAME GGUF, idle GB10, 20 ARM cores:
llama.cpp is flat from 128 to 512 tokens; our prefill falls 81% and decode
75%. That is the signature of fixed per-op overhead, not slow kernels.
We are not idle. A two-length diff cancelling the load phase puts our decode at
1987% CPU against llama.cpp's 1810% — every core busy, burning 5.6x the
CPU-seconds per token. We are spinning, not starved.
Lever 1 — decode is synchronisation-bound, not kernel-bound
The 2026-08-06 CPU profile (
benchmark-record.md):Batch-1 decode issues many tiny ops and pays a full threadpool barrier on each.
src/vt/cpu/cpu_threadpool.{cpp,h}.Explicitly NOT the defect, per that same profile: the elementwise GEMM
(
BtM4Neon/Bt16Neon, 21.5% prefill / 24.9% decode) is already on its optimisedNEON tier and is what remains once avoidable work is gone. No GEMM kernel
change can reach this.
Lever 2 — paged attention branches on dtype per element
~39% of prefill is the paged-attention inner loop, ~21% of it inside a
per-element dtype switch. It is visible at
src/vt/cpu/cpu_paged_attn.cpp:29:A branch per element, in the hot loop, on a value that is constant for the whole
tensor. The profile calls this "a known, already-solved defect class" — hoist the
dispatch out of the loop.
Scope and bar
Both are CPU-backend levers; the CUDA path is untouched. The bar is
llama.cpp on the same GGUF, same harness, bands calibrated before any delta is
read, prefill and decode reported separately. vLLM remains an open gap for
Muse Glimmer and is not the bar here.
Correctness first: every existing CPU gate must stay green, and any change that
alters numerics must show it.
No ceiling is to be declared. The 0.997x prefill tie at 128 tokens shows the
kernels are competitive when overhead is amortised; the gap is what we add per
op, and that is addressable.