perf(vulkan): pipelined submission, and llama.cpp's fence spin REJECTED for wrong numbers - #191
Merged
Merged
Conversation
…n as wrong on GB10 The 27B Vulkan decode gap to llama.cpp is host-side, and the campaign's "host = wall - GPU" figure of 5.3 ms/token was an INFERENCE across two different runs. This row measures our own side of the boundary directly. `VT_VULKAN_HOST_PROFILE=1` attributes every nanosecond spent inside `Dispatch` to a phase and separates the submit from the fence wait inside the flush. Two-length diff (output-len 36 minus 4, over 32 decode tokens), 27B on GB10, so prefill and one-time costs cancel: 900 dispatches and 4 flushes per decoded token flush-submit 1.50 ms/token (vkEndCommandBuffer + vkQueueSubmit) record 0.75 (barrier + bind + push + vkCmdDispatch) descriptor 0.29 (vkUpdateDescriptorSets) bookkeeping 0.28 pipeline-lookup 0.10 HOST TOTAL 3.04 ms/token flush-wait 234.67 ms/token (blocked on the GPU) Host plus fence-wait is 237.7 against that run's 237.2 ms TPOT, so the two account for the whole step. **Host is ~3 ms, not 5.3.** The rest of the old subtraction is GPU-timeline time the per-dispatch timestamps never covered -- ~900 inter-dispatch barrier drains and the command-buffer boundaries -- and no amount of host work reaches it. The lever was smaller than it was sized. And every nanosecond of the 3.0 ms was SERIALISED against the GPU: each flush submitted and then immediately waited, leaving the device idle for the whole of the host's recording of the next batch. The fix is llama.cpp's, whose own comment states the mechanism -- "Submit after enough work has accumulated, to overlap CPU cmdbuffer generation with GPU execution" (`ggml_backend_vk_graph_compute`, ggml-vulkan.cpp:16192-16195 and :16417-16423 @ pin `237ad9b96`); it submits several times per graph and waits exactly once, in `ggml_vk_wait_for_fence` (:2298). A flush here now submits and returns. Each in-flight slot owns a command buffer, its own command POOL (a pool resets as a whole, so one shared pool cannot serve an in-flight buffer), its own fence, its own query range, and a DISJOINT SLICE of every pipeline's descriptor ring -- that last one is the correctness argument, since a descriptor set is read at execution time and rewriting one under a submitted command buffer is silent corruption. A set can only be rewritten when its slot is re-entered, and a slot is re-entered only through `RetireSlotLocked`, which waits on that slot's fence first. The three host read paths (Copy, Memset, Synchronize/FlushPending) now DRAIN -- submit plus wait for every in-flight batch -- because "no batch is open" no longer means "the GPU is finished". A barrier is emitted before every dispatch INCLUDING the first in a command buffer, which is what carries a dependency across the command-buffer boundary now that the previous batch may still be running. The descriptor ring goes 128 -> 256 so a 2-deep pipeline keeps the same 128 sets per batch the un-pipelined ring had. Also off the per-dispatch path: the dispatch histogram was a `map<string,uint64_t>` lookup 900 times per token for a counter only a diagnostic reads (now a field on the pipeline, aggregated on demand); the bound-buffer set was a `std::set<void*>` allocating a node per new buffer (now a growable open-addressed table); and the descriptor writes were two heap vectors per dispatch (now stack arrays). MEASURED, same binary, `VT_VULKAN_INFLIGHT` 1 vs 2, cache dropped before every leg, one `flock`, order alternated: 5 of 5 paired legs favour pipelining, by 1.15 / 1.39 / 1.41 / 1.61 / 1.76 ms/token, median **-1.41 ms/token**. The box drifted 229.12-235.94 ms across legs of the IDENTICAL control configuration, so only the paired delta is quotable and no absolute tok/s claim is made. **Against the 3.1 ms owed to reach llama.cpp Vulkan's 4.35 tok/s, the goal is NOT met.** A FENCE SPIN WAS TRIED, WAS THE LARGER HALF OF THE APPARENT WIN, AND IS REJECTED. Porting llama.cpp's `vkGetFenceStatus` spin (:2306-2326) read 229.43 against 232.07 for pipelining alone -- and it COMPUTES THE WRONG NUMBERS on GB10. Same binary, one variable: `VT_VULKAN_FENCE_SPIN=0` gives `test_vulkan_backend` 33/33 and opt-125m 6/6 token-exact at INFLIGHT 1 AND 2; `=1` gives 16/33 and 17/33 and a diverging opt-125m at both. Failing identically with pipelining on and off is what identifies the spin rather than the pipelining. The failures are host reads returning pre-dispatch contents, and opt-125m degenerates to "the the capital capital of of the the world world". It passed 33/33 on llvmpipe, so only the real device found it -- and a full 8-leg 27B A/B had already been run against that arm, every number of which is VOID because the model was emitting garbage while being timed. The spin is deleted and the negative is recorded beside the retire path. MECHANISM GATE. Pipelining records the same kernels in the same order, so a slot count that fell back to 1, a rotation that never happened or a wait that crept back into the flush would leave every numeric assertion passing. The new case asserts in-flight depth, the submit/blocked-wait counters, the ring base rotating, the slice partitioning the ring, and one barrier per dispatch. Three scratch mutations: degrading the flush to submit-and-wait is caught; dropping the head-of-command-buffer barrier is caught; and collapsing every slot onto descriptor slice 0 -- the exact silent-corruption hazard -- passed the ENTIRE file on llvmpipe until the structural ring-base assertion existed. GATES, GB10, `VLLM_CPP_VULKAN:STRING=ON` verified in the cache and `[vt vulkan]` lines in every run: `test_vulkan_backend` 33/33 (2479 assertions) pipelined and 33/33 (2476) at INFLIGHT=1, `test_backend_cross_device` 11/11 (132), `test_opt_paged_engine` 6/6 prompts token-exact (96/96 tokens) with 0 declines in BOTH arms, and a 128-token x 2-prompt decode gave BYTE-IDENTICAL token ids across arms (md5 9a9fd41a97ca56a3beeebe420d874ae1). llvmpipe: the same, plus VT_VULKAN_BATCH=0. Committed SPIR-V unchanged and up to date. Two record gates fail identically before and after this change and are NOT gamed: `docs/STATUS.md` sits over a shrink-only ratchet it already exceeded by 2 kB on main (this change leaves it 3 chars smaller), and `.agents/state.md` carries one prior entry whose anchor is misplaced. `.agents/NOW.md` was over its 6,000-char budget on main and is now inside it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude:claude-opus-5 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 9, 2026
perf(vulkan): barrier only on a PROVEN dependency, and the llama.cpp gate is already MET THE HEADLINE IS NOT THE LEVER. No absolute number existed for main after #191 landed pipelined submission, so this row measured one first: 7 clean legs, page cache dropped per leg, GPU serialized, 27B bf16 32/32 c1, TPOT 228.66 to 230.25 ms = MEDIAN 4.36 tok/s against llama.cpp Vulkan's 4.35. One further leg was discarded for a NAMED cause -- an overlapping compile job the row had started itself -- not the unexplained 1-in-16 collapse. The 4.285 of record predates #191. Clean-leg spread is 0.69%, which IS the noise floor, so this is a NARROW pass and is written down as one. OPERATOR VERIFICATION, at its real strength. I re-measured independently on a different main SHA with its own fresh configure and read 4.58 tok/s -- but n=1, on a freshly rebooted box with clean memory, so I treat it as OPTIMISTIC and keep the 7-leg median as the binding figure. It corroborates the direction, not the value. I abandoned further legs rather than reboot the box a third time (below). THE BRIEFED PREMISE WAS WRONG, AGAIN MINE. I sized this lever at "~2.3 ms of GPU time outside any timestamped dispatch". It measures 0.56 ms/token and does NOT move when 178 barriers/token are removed. TOP_OF_PIPE is stamped when a command reaches top of pipe, which a preceding barrier does not hold back, so the drain is billed INSIDE the following dispatch's interval. Span minus sum understates barrier cost; the per-command-buffer span is the honest probe. READ/WRITE SETS: NEITHER PLUMBED NOR GUESSED. I offered two options and the row took a third and better one -- the GLSL already states it and glslang ENFORCES it, since writing a `readonly buffer` is a compile error and the promise survives as a NonWritable decoration in the SPIR-V. gen-vulkan-spirv.py now reflects each module's binding_count and writable_mask out of the compiled blob (the same technique as the existing spec_ids reflection) and GetPipeline hard-checks the declared binding count against what the host binds. Zero op-layer edits, and the source of truth is machine-checked. Also verified: vkCmdDispatch is the ONLY device command in this backend that touches a storage buffer, and AllocBuffer gives each VkBuffer its own dedicated VkDeviceMemory, so distinct buffers cannot alias. Access sets clear ONLY when a barrier is recorded, never at a flush, which is what carries the invariant across the pipelined command-buffer boundary. MEASURED: barriers 900.0 -> 722.0 per token (-19.8%); GPU span 231.38 -> 230.29 ms (-1.09); span-minus-sum 0.556 -> 0.563 (unmoved, as above). e2e 12 order-alternated pairs, 8 to smart, mean paired delta -0.94 ms, sign test p = 0.19 -- NOT ESTABLISHED against a box that drifts 8.6 ms across four pairs of its own base arm. DEFAULT OFF. Gate-clean everywhere tried and the mechanism is directly measured, but the e2e effect is not resolved past this box's drift, the goal is already met without it, and this is the exact hazard class that shipped garbage two days ago. One env var away with all evidence attached. THE MUTATION RESULTS ARE THE MOST VALUABLE PART. - Always-report-independent -> 32/35, 330 assertions failed, and they are VALUE failures (0 == -150: a dispatch reading memory before its producer wrote it). The unconditional barrier is load-bearing, proven on the real driver. - Remove ONLY the write-after-read half -> 34/35, exactly 2 assertions failed, BOTH the new structural counters, and EVERY value assertion passed. A partial implementation that handles RAW and WAW but forgets WAR computes correct numbers on this hardware today. That is the whole argument for asserting on counters rather than on tolerances, demonstrated instead of asserted. OPERATOR-VERIFIED GATES, on GB10, all four combinations of the barrier arm and the pipelining depth, because llvmpipe is largely serial and hides exactly this class of bug -- this campaign already has a change that passed 33/33 on llvmpipe while computing garbage on the real driver: smart=0 inflight=1 test_vulkan_backend 2647/2647 opt-125m 6/6 token-exact smart=0 inflight=2 test_vulkan_backend 2650/2650 opt-125m 6/6 token-exact smart=1 inflight=1 test_vulkan_backend 2647/2647 opt-125m 6/6 token-exact smart=1 inflight=2 test_vulkan_backend 2650/2650 opt-125m 6/6 token-exact plus test_backend_cross_device 132/132 and, on llvmpipe from a clean build, test_vulkan_backend 35/35 (2107) and opt-125m 6/6 with 0 declines. gen-vulkan-spirv.py --check reproduces the committed SPIR-V byte-for-byte. The row additionally reports a 256-token decode byte-identical across all four arms. INCIDENTS, DISCLOSED. The row hit a stale-binary FALSE GREEN first: `cp -a` of a build directory silently rebuilds the ORIGINAL sources because CMake caches absolute paths, and produced a byte-identical binary. Redone in place with source and binary md5s verified. Separately the box OOM-rebooted twice today (NVRM NV_ERR_NO_MEMORY, mem_desc.c:1359) -- once here, once during my own verification, on the FIRST leg with nothing else running. Root cause is now understood and is NOT this change: the 27B holds ~50 GB of safetensors in page cache AND ~50 GB of Vulkan buffers, both host RAM on a 119 GB unified box, and after a leg exits that memory is not promptly reclaimed, so leg N+1 allocates while leg N is still outstanding. The CUDA NVFP4 path already solves this by releasing the mmap'd shards after upload (114 -> 67 GiB RSS); the Vulkan loader does not. That is a real robustness bug for any 27B user on a Spark and is left NAMED, not fixed here. Also updates the headline benchmark source and the two README claims it licenses, both of which still said 98.5% and understated the current state. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
localai-bot
pushed a commit
that referenced
this pull request
Aug 10, 2026
docs(FEATURES): Vulkan is at parity with llama.cpp, not behind it (#310) `docs/FEATURES.md` said Qwen3.6-27B decodes **4.24 tok/s vs llama.cpp's 4.35**. That is the pre-#191 number: it claimed we were BEHIND when the binding measurement is parity MET, and it contradicted the three other public surfaces, all of which were reviewed and landed saying 4.36: docs/STATUS.md decode 4.36/4.35 MET docs/BENCHMARKS.md decode 4.36 vs 4.35, MET (7 clean legs) README.md decodes 4.36 vs llama.cpp Vulkan 4.35 tok/s Corrected, and it keeps saying the pass is NARROW: the binding figure is the median of 7 clean legs at TPOT 228.66-230.25 ms, and the 0.69% leg spread IS the noise floor. The detail stays in BENCHMARKS and the record rather than being restated here. Also adds the load-phase result the paragraph was missing beside its memory one (`ENG-LOAD-DIRECT-UPLOAD`, #150): a load is 1.54x faster warm, alongside the already-recorded 100.8 -> 53.4 GiB peak RSS. WHY THE PARAGRAPH IS SHORTER THAN THE SUM OF ITS PARTS. The page budgets prose at 20 paragraphs and 700 chars each, and my first correction failed both. The paragraph-count failure was the interesting one: `_prose_paragraphs` treats any line starting with `*` as a list item, and on main the wrap happened to put `**prefill 21.5x**` at the start of a line -- so the checker had been reading ONE paragraph as TWO, and the page was inside its 700-char budget only by accident. Rewrapping honestly merged them and exposed 891 real chars. Rather than restore the accidental split or spend the count, the paragraph was tightened to 677 (`6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle` -> `6/6 prompts vs the vLLM 0.25.0 oracle`; `all nine of that model's ops` -> `every op of that model`), which is what the checker asks for: the numbers belong in BENCHMARKS, and this page explains what works. DELIBERATELY NOT DONE: moving Vulkan out of STATUS's "Not supported yet" section. Running a 27B at llama.cpp parity is a real milestone, but 25 native ops of 112 with quant/MoE/MLA at zero is not "supported", and the entry already states precisely what does and does not work. Gates: check-public-doc-tables, check-doc-checkpoint and check-agent-record all green. No source file changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
bakon11
pushed a commit
to bakon11/vllm.cpp
that referenced
this pull request
Aug 10, 2026
…dler#310) `docs/FEATURES.md` said Qwen3.6-27B decodes **4.24 tok/s vs llama.cpp's 4.35**. That is the pre-mudler#191 number: it claimed we were BEHIND when the binding measurement is parity MET, and it contradicted the three other public surfaces, all of which were reviewed and landed saying 4.36: docs/STATUS.md decode 4.36/4.35 MET docs/BENCHMARKS.md decode 4.36 vs 4.35, MET (7 clean legs) README.md decodes 4.36 vs llama.cpp Vulkan 4.35 tok/s Corrected, and it keeps saying the pass is NARROW: the binding figure is the median of 7 clean legs at TPOT 228.66-230.25 ms, and the 0.69% leg spread IS the noise floor. The detail stays in BENCHMARKS and the record rather than being restated here. Also adds the load-phase result the paragraph was missing beside its memory one (`ENG-LOAD-DIRECT-UPLOAD`, mudler#150): a load is 1.54x faster warm, alongside the already-recorded 100.8 -> 53.4 GiB peak RSS. WHY THE PARAGRAPH IS SHORTER THAN THE SUM OF ITS PARTS. The page budgets prose at 20 paragraphs and 700 chars each, and my first correction failed both. The paragraph-count failure was the interesting one: `_prose_paragraphs` treats any line starting with `*` as a list item, and on main the wrap happened to put `**prefill 21.5x**` at the start of a line -- so the checker had been reading ONE paragraph as TWO, and the page was inside its 700-char budget only by accident. Rewrapping honestly merged them and exposed 891 real chars. Rather than restore the accidental split or spend the count, the paragraph was tightened to 677 (`6/6 prompts / 96/96 tokens vs the vLLM 0.25.0 oracle` -> `6/6 prompts vs the vLLM 0.25.0 oracle`; `all nine of that model's ops` -> `every op of that model`), which is what the checker asks for: the numbers belong in BENCHMARKS, and this page explains what works. DELIBERATELY NOT DONE: moving Vulkan out of STATUS's "Not supported yet" section. Running a 27B at llama.cpp parity is a real milestone, but 25 native ops of 112 with quant/MoE/MLA at zero is not "supported", and the entry already states precisely what does and does not work. Gates: check-public-doc-tables, check-doc-checkpoint and check-agent-record all green. No source file changes. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
perf(vulkan): pipelined submission, and llama.cpp's fence spin REJECTED for wrong numbers
THE GOAL IS NOT MET, and the lever was smaller than I sized it.
THE PREMISE I GAVE THIS ROW WAS WRONG. I briefed host time as 5.3 ms/token, and
that figure was an INFERENCE across two different runs (wall from one, GPU-active
from another). An env-gated host phase profiler (VT_VULKAN_HOST_PROFILE=1) and a
two-length diff (output-len 36 minus 4, over 32 decode tokens) measure it directly:
per decoded token ms
flush-submit (vkEndCommandBuffer + vkQueueSubmit) 1.50
record (barrier/bind/push/vkCmdDispatch) 0.75
descriptor (vkUpdateDescriptorSets) 0.29
bookkeeping 0.28
pipeline-lookup 0.10
HOST TOTAL 3.04
flush-wait (blocked on the GPU) 234.67
900 dispatches and 4 flushes per token, matching the prior row. Host + wait is
237.7 against that run's 237.2 ms TPOT, so the instrument closes. Host is ~3 ms,
NOT 5.3. The other ~2.3 ms is GPU-timeline time outside any timestamped dispatch
(~900 inter-dispatch barrier drains) and no host work can reach it.
WHAT SHIPPED. llama.cpp's pipelined submission (ggml-vulkan.cpp:16192-16195,
:16417-16423 @ 237ad9b96): a flush submits and returns instead of blocking, with a
per-slot command buffer, pool, fence, query range and a DISJOINT descriptor-ring
slice. All 3.0 ms of host work was previously serialized against an idle GPU.
Same binary, VT_VULKAN_INFLIGHT 1 vs 2, cache dropped, order-alternated:
5/5 paired legs favour it -- 1.15 / 1.39 / 1.41 / 1.61 / 1.76 ms, median
-1.41 ms/token. Against the 3.1 ms owed, that is not enough. The box drifted
229.12-235.94 ms across legs of the IDENTICAL control config, so no absolute
tok/s claim is made from this row.
WHAT DID NOT WORK, AND IT NEARLY GOT SHIPPED. llama.cpp's vkGetFenceStatus spin
(:2306-2326) read 229.43 vs 232.07 -- the larger half of the apparent win, and
enough to reach 4.36-4.39 tok/s, i.e. to MEET THE GOAL. It computes WRONG NUMBERS
on GB10. Same binary, one variable:
FENCE_SPIN=0, INFLIGHT 1/2 33/33 33/33 6/6 exact both
FENCE_SPIN=1, INFLIGHT 1/2 16/33 17/33 DIVERGES both
Host reads return pre-dispatch buffer contents; opt-125m degenerates to "the the
capital capital of of the the world world". Failing identically with pipelining ON
and OFF is what acquits the pipelining itself. It passed 33/33 on llvmpipe, so
only the real driver exposed it. A full 8-leg order-alternated 27B A/B had already
been run against that arm and EVERY number in it is void, because the model was
emitting garbage while being timed. Spin deleted, negative recorded in-source.
This is the whole reason a speed lever is gated on token-exactness and not on a
tolerance: the fastest arm here was the broken one.
CORRECTNESS. GB10: test_vulkan_backend 33/33 (2479 assertions) pipelined and 33/33
(2476) at INFLIGHT=1; test_backend_cross_device 11/11; test_opt_paged_engine 6/6
token-exact (96/96) with 0 declines in BOTH arms; a 128-token x 2-prompt decode
gives byte-identical token ids across arms. No shader changed.
MECHANISM GATE. Asserts in-flight depth, submit/blocked-wait counters, ring-base
rotation, slice partitioning, and one barrier per dispatch. Of three scratch
mutations, collapsing every slot onto descriptor slice 0 -- the exact silent
corruption this change makes possible -- PASSED the entire file on llvmpipe until
the structural ring-base assertion existed. The same blind spot the fence spin
exploited.
OPERATOR VERIFICATION on the merged tree, which is the four-lever combination no
row tested: test_vulkan_backend 33/33 (1936 assertions), test_backend_cross_device
11/11 (132), test_opt_paged_engine on Vulkan 6/6 token-exact (96/96) with 0
declines, gen-vulkan-spirv.py --check byte-clean under pinned glslang 16.5.0.
THE NEXT LEVER, NAMED. Host is now ~1-2 ms and largely overlapped. The largest
identified residual is the ~2.3 ms of inter-dispatch GPU time: we emit an
unconditional full memory barrier before all ~900 dispatches per token, where
llama.cpp emits one only on a real dependency (ggml_vk_sync_buffers, :3193). That
is correctness-risky and belongs in its own row.
CAVEAT: one leg stalled 23 s inside a single token with median ITL normal at
233 ms. The signature does not match the recorded 20x vt_matmul bimodality;
mechanism unestablished, discarded, and NOT attributed to this change.
Conflicts against main were resolved by kind: NOW.md and the BENCHMARKS keyed row
merged BY KEY (taking the corrected ~3 ms host figure over my inferred 5.3 and
keeping the REJECTED spin, which is a negative that must not be lost), state.md
unioned chronologically.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude-Code:claude-opus-5 [Claude Code]