fix(cuda-arch): arch-gate the WMMA selectors — pre-Ampere guards were a live trap - #56
Merged
Conversation
… a live trap W1/W1b compiled the five bf16-WMMA prefill kernels under `#if __CUDA_ARCH__ >= 800` with an `#else __trap()`. The predicate that SELECTS them (cuda_paged_attn.cu:2611) is entirely host-side — shape, dtype and env only — and never consulted the device capability. So on a Turing/Volta/Pascal board a d=256 bf16 prefill would still select a kernel whose body is a trap: the guard made the TU compile and left a runtime crash behind it. Gate the predicate on the cached DeviceCaps::sm_major >= 8. The whole WMMA ladder (gqa / flash2 / flash2vec) derives from `wmma`, so one term covers all four launchers, and <sm_80 now falls through to LaunchPrefillFlash — the portable CUDA-core register-tiled flash, which is correctness-grade and uses no tensor cores. GB10 is unaffected by construction: sm_121a gives sm_major 12, so the term is always true and the gate models' selection is unchanged. Build-verified on dgx (nvcc 13.0.88), cuda_paged_attn.cu -Werror=all-warnings: sm_75 rc=0 0 warnings, sm_121a rc=0 0 warnings. No board ran it. NOT fixed here, and recorded for a follow-up decision: the same arch-blind selection exists for the kernels guarded in the other two TUs. cuda_gdn.cu launches GdnChunkWUWmma/DeltaHWmma/OWmma (:5026, :5194, :5229) with no arch term, and cuda_matmul_nvfp4.cu's bf16 MoE grouped GEMM (:1397, :1406) terminates in a WMMA kernel with no non-WMMA fallback in that launcher. Neither has a portable path to fall through to the way attention does, so the fix is a design call — a portable body or an explicit host-side error — not a mechanical edit. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Completes the class the prefill-selector fix opened. W1/W1b made the WMMA bodies COMPILE on <sm_80 behind `__trap()`; every predicate that SELECTS them was host-side only (shape, dtype, env), so a pre-Ampere board would still pick a trapping kernel. A compile guard without a selector guard is not a fix — it converts a build error into a runtime crash on exactly the boards it enables. Two remaining chokepoints, each now requiring DeviceCaps::sm_major >= 8 and each falling through to a portable path that already exists: - cuda_gdn.cu:5359 (GdnPrefillKernelCuda) -> GdnScanCuda, the sequential scan that already serves the arbitrary-dim corners. This is the single chokepoint: all 7 guarded GDN launches live in LaunchChunkedPrefill, itself reached only from here, and the TSc=float instantiations use the TF32 WmmaCfg, so f32 is not a way around the guard. - cuda_matmul_nvfp4.cu WmmaEnabled():77 -> the naive / tiled / split-K CUDA-core kernels. Folded into the predicate rather than its six call sites because all six mean the same thing and each already has a CUDA-core fallthrough. Queried LIVE rather than latched in the static env cache: the device context need not exist at static-init time, and a wrong value cached there is unrecoverable. Both fail safe — caps invalid selects the portable path. Verified on dgx (nvcc 13.0.88). All three TUs -Werror=all-warnings rc=0, 0 warnings, at BOTH sm_75 and sm_121a. GB10 inert by measurement: sm_121a SASS IDENTICAL for all three against the pre-change build (933,178 + 825,294 + 137,542 lines, anon-namespace hash normalized) — the change is host-side and no device code moved. W2 is rescoped in the spec from a correctness brick to a SPEED brick. The bf16-WMMA path is only `is_prefill && d == 256 && bf16 q+KV`; all decode and every other prefill shape already run portable kernels, and these gates route d=256 to the CUDA-core flash on <sm_80. Pre-Ampere correctness is therefore covered by paths that exist today, and the llama.cpp fattn-tile/fattn-vec port buys prefill throughput only — against a floor (llama.cpp on-card) that needs hardware we do not have. docs/STATUS.md: the superseded 18/20 narrative is collapsed into the binding result, net -1537 chars, and the size ratchet is lowered 257201 -> 256224 in the same change per the gate's own instruction. Detail moved to .agents/benchmark-record.md. No number is claimed or owed: nothing executes on these arches, no library link exists, and sm_70 stays uncompilable until a CUDA 12.x toolkit is wired. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this fixes
W1/W1b made the bf16-WMMA kernel bodies compile on
<sm_80by wrapping them in#if __CUDA_ARCH__ >= 800with an#else __trap(). But every predicate that selects those kernels was host-side only — shape, dtype, env var — and never consulted the device.So on a Turing/Volta/Pascal board the host would still pick a kernel whose body is a trap. A compile guard without a selector guard is not a fix; it converts a build error into a runtime crash on exactly the boards it is meant to enable.
The three chokepoints
Each now requires
DeviceCaps::sm_major >= 8, and each falls through to a portable path that already existed — no new kernel is introduced:cuda_paged_attn.cu:2611LaunchPrefillFlash— CUDA-core register-tiled flashcuda_gdn.cu:5359(GdnPrefillKernelCuda)GdnScanCuda— the sequential scancuda_matmul_nvfp4.cuWmmaEnabled():77Two details worth review attention:
LaunchChunkedPrefill, itself reached only fromGdnPrefillKernelCuda. TheTSc=floatinstantiations use the TF32WmmaCfg, so f32 is not a way around the guard — the whole chunked path had to be gated, not just the bf16 branch.WmmaEnabled()rather than its six call sites, because all six mean the same thing and each already has a CUDA-core fallthrough. It is queried live rather than latched in the static env cache: the device context need not exist at static-init time, and a wrong value cached there would be unrecoverable.All three fail safe — an unreadable device capability selects the portable path.
Verification (dgx, nvcc 13.0.88)
sm_75: 20/20 unconditional CUDA TUs compile,-Werror=all-warnings, 0 errors 0 warnings.sm_121a(GB10) inert by measurement, not by argument. All three TUs compile 0-warn, and their SASS is identical to the pre-change build — 933,178 + 825,294 + 137,542 lines (anonymous-namespace hash normalized; that hash shifts on any edit, which is why a raw diff showsFunction :headers only). The change is host-side; no device code moved.This is a COMPILE result, not a link and not execution. No Turing, Volta or Pascal board exists here and none ran this.
sm_70remains uncompilable until a CUDA 12.x toolkit is wired — nvcc 13 rejects it outright.W2 rescoped
The spec is updated: the bf16-WMMA path is only
is_prefill && d == 256 && bf16 q+KV. All decode and every other prefill shape already run portable kernels, and these gates route d=256 to the CUDA-core flash on<sm_80. Pre-Ampere correctness is therefore covered by paths that exist today, and the llama.cppfattn-tile/fattn-vecport becomes a speed brick — ~2,000 lines adapting contiguous KV to our paged block-table layout, against a floor (llama.cpp on-card) that needs hardware we do not have. Worth doing only for a genuine speed claim once a T4/V100 is reachable.Docs
docs/STATUS.mdcollapses the superseded 18/20 narrative into the binding result and lowers the size ratchet in the same change, percheck-public-doc-tables.py's own instruction. Detail moved to.agents/benchmark-record.md. Both doc gates pass.No number is claimed or owed: nothing executes on these arches.
🤖 Generated with Claude Code