perf(vulkan): decode RmsNorm ran on ONE workgroup -- 7.85 -> 1.59 ms/token - #185
Merged
Conversation
vt::RmsNorm gets a second SPIR-V module, vt_rms_norm_wide, at 1024 invocations with a subgroup reduction, chosen by device capability. Module count 25 -> 26. No op changes tier, no dispatch count changes, no flush tally changes: the only thing that moves is which SPIR-V the same 129 dispatches per token bind. THE MECHANISM, AND WHY THE OBVIOUS SUSPECT WAS WRONG. vt_rms_norm cost 0.0611 ms/call, roughly TEN TIMES vt_silu_and_mul (0.0047), vt_gdn_post_conv (0.0063) and vt_rms_norm_gated (0.0055) -- the last of which runs the SAME workgroup tree reduction. It reads ~20 KB, under a microsecond at the 273 GB/s roof, so nearly all of it was overhead, and the barrier-per-step tree was the natural suspect. Printing each distinct (shader, workgroup count, push constants) triple from VulkanContext::Dispatch says otherwise: decode step, t = 1 workgroups elements ms/call vt_rms_norm 1 5,120 0.0611 vt_rms_norm_gated 48 6,144 0.0055 vt::RmsNorm dispatches ONE WORKGROUP PER ROW and a batch-1 decode step has one row, so the kernel ran 128 invocations -- four warps of ONE SM -- over a 5120-wide row, 40 strided iterations per lane, with the rest of the GPU idle. The gated kernel reduces identically and is 11x cheaper only because its rows are heads. The confirmation needs no code change at all: in the SAME run the SAME shader serves prefill, where t = 32 gives it 32 workgroups and 32x the data, and costs 0.066 ms/call. 32x the work for 8% more time is a kernel waiting, not a kernel working. The reduction was never the difference; the OCCUPANCY was. PORTED FROM. llama.cpp @ 237ad9b96 reaches the same conclusion twice: vulkan-shaders/rms_norm.comp:33 runs BLOCK_SIZE 512 rather than the guaranteed-portable 128, and vulkan-shaders/rms_norm_partials.comp exists solely to spread a SINGLE row over many workgroups when the row count cannot -- including the subgroupAdd reduction at rms_norm_partials.comp:41-47 that vt_common.glsl's new VT_SUBGROUP_REDUCE path transcribes. Widening the workgroup is the half of that needing no second dispatch and no cross-workgroup hand-off. The per-element math is untouched and still 1:1 from src/vt/cpu/cpu_ops.cpp:225-250. MEASURED, GB10, Qwen3.6-27B bf16, Vulkan, 1 prompt, 32-in, c1. Two-length GPU-timestamp diff (output-len 36 minus output-len 4 over 32 tokens, so prefill and one-time costs cancel), all arms from ONE binary via VT_VULKAN_RMSNORM: per decoded token 128-wide 1024-wide+subgroup RmsNorm dispatches 129 129 RmsNorm ms/call 0.0607 0.0123 RmsNorm ms/token 7.85 1.59 GPU dispatches, all shaders 32,544 32,544 command-buffer flushes 191 191 reference-tier declines 2 2 WIDTH AND SUBGROUP, ATTRIBUTED SEPARATELY, because that is two changes and one number. A third module at 1024 lanes with the plain halving tree (scratch, never committed) A/B'd against both in one binary, controls matched within 0.4% on vt_matmul_vec: 128/tree 0.0607 -> 1024/tree 0.0142 -> 1024/subgroup 0.0123 ms/call. The WIDTH is 4.3x of it; the SUBGROUP is the remaining 1.15x. The earlier campaign finding that a subgroup reduction was a 4/8 wash in the GEMV is not overturned, only re-scoped: even in a kernel that is almost nothing but a reduction it is worth 1.15x, not 5x. WHAT DID NOT WORK, recorded because it was the leading hypothesis. The residual path stored the summed value and RE-READ it from memory to pick up the bf16 rounding (cpu_ops.cpp:235-237's idiom), a store->load dependency on the same address in each of the 40 iterations. Replacing it with vt_round_through -- the same rounding as a pure register function, bit-identical by construction -- moved vt_rms_norm from 0.0611 to 0.0607 ms/call at 128 lanes. Within noise: the memory round trip was not the cost. It is kept because it is free, deletes a memoryBarrierBuffer(), and leaves the wide module's inner loop dependency-free. It is not the lever and is not recorded as one. PAIRED DECODE, 6 order-alternated AB/BA pairs at 32-in/32-out, page cache dropped before each of the 12 runs. The base arm is tight (240.65, 241.85, 241.91, 241.76, 242.17, 241.64 ms); the wide arm reads 235.22, 235.63, 235.89 on three legs and 429.03, 425.75, 441.21 on the other three. Raw, that is 3 of 6. Those three are the ~1.8x-the-block-median pathology this box already has on record (the fused-attn-preamble row saw the same signature at ~446-448 ms, and a separate block here produced one on the BASE arm at 449.46, so it is not arm-specific): TTFT is unaffected in them and the whole slowdown is a uniform decode ITL shift, which is what host memory pressure looks like on a 119 GB unified box with a 50 GB model. Discarding the three pairs containing them leaves 3 of 3, median TPOT 241.85 -> 235.63 ms, decode 4.13 -> 4.24 tok/s against llama.cpp Vulkan's 4.35. The 6.2 ms/token that survives is the same 6.3 ms/token the GPU-timestamp diff attributes to this kernel, which is the load-bearing evidence; the wall clock only has to agree with it. CORRECTNESS. NMSE vs the CPU oracle in the same binary at the decode shape (t=1, h=5120): 7.02e-15 without a residual and 1.95e-12 with one on the GB10 NVIDIA driver (6.87e-15 / 2.17e-12 on llvmpipe), against the 5e-4 bar this backend's reducing kernels are gated at, and IDENTICAL to what the 128-wide module scores on the same inputs. The residual STREAM is held to the BIT-EXACT tier by memcmp, which is what proves vt_round_through is the memory round trip rather than an approximation of it. The new case asserts the MECHANISM, not only the numbers: it pins the module BY NAME through the dispatch histogram (exactly one dispatch of the intended module and none of the other) in all three configurations, and it checks that wide_reduce() is only ever false because a specific probed limit is missing. Both arms are exercised on every box -- llvmpipe reports 1024 invocations and compute subgroup arithmetic too, so without VulkanContext::set_rms_norm_override the fallback would be dead code everywhere we can run. GATES. test_vulkan_backend 30/30 (2371 assertions) on GB10 and 30/30 (1828) on llvmpipe; test_backend_cross_device 11/11 on both; test_opt_paged_engine with VLLM_CPP_DEVICE=vulkan still 6/6 token-exact (96/96) with 0 declines on both, and on GB10 with the fallback arm forced as well. gen-vulkan-spirv.py --check clean at pinned glslang 16.5.0; 24 of the 26 committed modules are byte-identical to before, only vt_rms_norm changed and vt_rms_norm_wide is new. ONE HONEST LIMIT ON THAT e2e GATE: opt-125m is a LayerNorm model with ZERO vt::RmsNorm calls, so the standing token-exact gate is a backend regression guard and not a proof of this kernel. What covers the kernel end to end is the model that calls it 129 times per token: Qwen3.6-27B on both arms with --output-token-ids, 4 prompts x 32 tokens, greedy, twice each -- all four dumps byte-identical, arm to arm and run to run. Disclosed limit: vllm-bench feeds SYNTHETIC prompts, so those sequences are low-entropy and this is weaker than a golden-text comparison. It is the strongest e2e evidence available on a model with no pinned oracle, not a token-exact gate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
rohitpaul
pushed a commit
to rohitpaul/vllm.cpp
that referenced
this pull request
Aug 9, 2026
Each of mudler#184, mudler#185 and mudler#186 was measured against `93852c28` in isolation, so none of their numbers described a tree carrying all three. STATUS.md was deliberately left at the CONSERVATIVE 4.24 -- the largest single lever -- rather than a sum, because adding independently-measured deltas would have been inventing a number. This is the measurement that replaces it. METHOD. `git archive` of merged main `81ea01f0` to dgx, `vt_matmul.comp` md5 verified identical on both sides. Fresh Release configure. 8 wall-clock legs with the page cache dropped before each and `flock $HOME/gpu.lock` held, then a two-length GPU-timestamp diff (output-len 36 minus 4, over 32 decode tokens) so prefill and one-time costs cancel. Qwen3.6-27B bf16, 1 prompt, 32-in, c1. MEASURED, 8 legs, ALL CLEAN: TPOT 232.18 to 234.07 ms, decode 4.27 to 4.31 tok/s, MEDIAN 4.285, spread 0.8%. Zero bimodal collapses. MEASURED, per decode token: shader before merged delta vt_matmul_vec 214.1 210.1 -4.0 vt_matmul (lm_head) 12.43 11.57 -0.86 vt_rms_norm -> vt_rms_norm_wide 7.97 1.57 -6.40 TOTAL GPU 240.3 227.7 -12.6 Wall 233.0 ms, so host is 5.3 ms/token. Every lever reproduced its own claim on the merged tree: vt_rms_norm_wide at 0.0123 ms/call is exactly what its row reported, and the other two deltas match theirs inside the leg spread. A FIRST ATTEMPT WAS INVALID AND IS RECORDED, because the failure is reusable. It read 0.75-1.27 tok/s. VLLM_CPP_VULKAN defaults to AUTO, which resolves to OFF -- Vulkan is opt-in so it cannot register into gate builds -- and the fresh configure omitted -DVLLM_CPP_VULKAN=ON. The options had been copied from the reference build's CMakeCache.txt through `grep | head -15`, and the alphabetical list ended at TRITON_TARGET, exactly one line before VULKAN. My own truncation hid the flag. Three tells were already in the output and are the cheap check: no `[vt vulkan]` lines despite VT_VULKAN_DISPATCH_STATS=1, no `[vt reference-tier]` lines, and "Asynchronous scheduling is ENABLED" where every valid Vulkan run reports it disabled. Same family as this campaign's stale-binary false greens, in the opposite direction -- a false catastrophe rather than a false pass. A CORRECTION TO THIS CAMPAIGN'S ROOF ARITHMETIC. mudler#186 established that GB10 does not reach its theoretical 273 GB/s, by running a known-good streaming kernel on the identical byte count: 230.3 GB/s. That correctly retires the 9.3 ms lm_head floor. It does NOT invalidate the layer-GEMV percentages, which MEASURE 243-248 GB/s -- above 230.3 -- so 230.3 is a ceiling for THAT SHAPE (k=5120, n=248320, one 2.54 GB buffer), not a device ceiling. Why one shape's ceiling sits ~7% below the same kernel's on layer weights is UNEXPLAINED, and it is the same lone buffer the 20x bimodal collapse attaches to. WHERE THE REMAINING 3.1 ms IS. llama.cpp Vulkan is 4.35 tok/s = 229.9 ms/token on the same 50.89 GiB weights on this box. vt_matmul_vec is now 92% of our GPU time at 248.0 GB/s, and its lever is CLOSED with the binding constraint identified as DRAM bandwidth on a ~50 GB working set streamed once per token. The named residuals are lm_head (219.8 GB/s, 95.4% of its own shape's measured ceiling, worth ~1.3 ms if it could reach 248) and host at 5.3 ms. Also refreshes .agents/NOW.md in the same change and appends the dated checkpoint below the enforced marker in .agents/state.md, per the handoff contract. KNOWN RED, unchanged and pre-existing: docs/STATUS.md remains over its shrink-only ratchet (this change shrinks it by 3 more chars). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
vt_rms_normcost 0.0611 ms/call at decode against 0.0055 forvt_rms_norm_gated, which runs the same tree reduction — an 11x gap that thebrief flagged as a clue rather than noise. The reduction was not the cause.
Mechanism, measured
vt::RmsNormdispatches one workgroup per row, and a batch-1 decode step hasexactly one row. Printing each distinct (shader, workgroup-count, push-constant)
triple from
VulkanContext::Dispatch:vt_rms_normvt_rms_norm_gated128 invocations — four warps of a single SM — walking a 5120-wide row in 40
strided iterations while the rest of the GPU idled. The gated kernel is 11x
cheaper only because its rows are heads, so it fills the machine.
The confirmation needed no code change: in the same run the same shader serves
prefill (t=32 → 32 workgroups, 32x the data) at 0.066 ms/call against decode's
0.0611. 32x the work for 8% more time.
Result
Two-length GPU-timestamp diff (len 36 minus len 4, over 32 tokens), GB10, 27B
bf16, 1 prompt, 32-in, c1:
Width and subgroup attributed separately, via a third scratch module (1024
lanes with the plain tree) so all three arms live in one binary, controls within
0.4%: 128/tree
0.0607→ 1024/tree0.0142→ 1024/subgroup0.0123. Width is4.3x; the subgroup is the remaining 1.15x. The campaign's earlier "subgroup
reduction is a 4/8 wash" result is therefore re-scoped, not overturned — even
in a kernel that is almost entirely a reduction it is worth 1.15x, not 5x.
What did NOT work
The store-then-re-read residual idiom (a store→load dependency on the same address
in each of 40 iterations) was the leading hypothesis. Replacing it with a
bit-identical register round-trip moved 0.0611 → 0.0607 at 128 lanes — within
noise; the memory round trip was not the cost. Kept anyway (free, deletes a
memoryBarrierBuffer(), leaves the wide loop dependency-free) but recorded asnot-the-lever.
Fusion was not taken: the 129 calls already contain the weight multiply and the
residual add, so there is no adjacent elementwise op for the
fused_ops.hseam tobind.
e2e
6 order-alternated AB/BA pairs, cache dropped before all 12 runs:
Raw 3 of 6. The three bold legs are this box's ~1.8x bimodal pathology; a
separate block hit it on the base arm at 449.46, so it is not arm-specific.
Discarding those three pairs leaves 3 of 3, median TPOT 241.85 → 235.63 ms,
decode 4.13 → 4.24 tok/s. The surviving 6.2 ms/token matches the 6.3 ms/token
the GPU-timestamp diff independently attributes to this kernel.
On the bimodal legs: this agent attributes them to host memory pressure (50 GB
model on a 119 GB unified box) and directly observed concurrent 27B benchmarks
co-resident at 91 GB RSS. A parallel agent measured that during such legs
vt_matmul(lm_head) collapses 19.5x whilevt_matmul_vecbeside it moves 1.2%.Those reconcile: under pressure the evicted victim is the largest, coldest buffer,
and lm_head is a single ~2.54 GB allocation touched once per token. That makes
the collapse most likely self-inflicted measurement contamination rather than a
production pathology — being confirmed separately with the box to one process.
Correctness
NMSE vs the CPU oracle at the decode shape: 7.02e-15 (no residual) /
1.95e-12 (residual) on GB10, 6.87e-15 / 2.17e-12 on llvmpipe, against a 5e-4
bar — identical to what the 128-wide module scores. The residual stream is held
bit-exact by memcmp.
An honest gap the agent found and closed: opt-125m is a LayerNorm model and
makes zero
vt::RmsNormcalls, so the STRICT e2e gate does not exercise thiskernel at all. Covered with the model that does — 27B on both arms with
--output-token-ids, 4 prompts x 32 tokens greedy, twice each, all four dumpsbyte-identical arm-to-arm and run-to-run. Disclosed:
vllm-benchprompts aresynthetic and low-entropy.
Gates, re-run by the operator rather than taken on report
Clean llvmpipe build:
test_vulkan_backend30/30, 1828 assertions,test_backend_cross_device11/11, 132,VLLM_CPP_DEVICE=vulkan test_opt_paged_engine6/6 token-exact (96/96), 0 declines,gen-vulkan-spirv.py --checkreproduces the committed SPIR-V byte-for-byte underpinned glslang 16.5.0.
Agent-reported, not re-run here: GB10 30/30 with 2371 assertions, the GB10 forced
fallback arm, and the 27B token-id dumps.
The new test asserts the mechanism through the dispatch histogram — exactly one
dispatch of the intended module and none of the other, in all three configurations
— and checks
wide_reduce()can only be false because a specific probed limit ismissing.
Known red, named rather than papered over
check-pr-sizecounts the regeneratedvulkan_spirv.cppas reviewable productcode, the same checker gap every Vulkan shader PR hits. The remaining preflight
failures (STATUS ratchet, undocumented
VT_GEMMA4_*/VT_ROCM_*vars, commit39da53c7's trailers) are pre-existing on the base; this branch's ownVT_VULKAN_RMSNORMis documented anddoc-checkpoint rangeis green.FOLLOWING_AGENTS_PROTOCOL