fix(ROCM): AttnQkNormRopeGate output-dtype dispatch -- the 0.8B GDN forward divergence root-caused and fixed (RED anchor turns GREEN) - #570
Draft
VikashLoomba wants to merge 2 commits into
Conversation
This was referenced Aug 13, 2026
This was referenced Aug 13, 2026
…lden lane in the Qwen3-0.6B near-tie gate Issue mudler#41 M4 milestone. The oracle is upstream vLLM AT THE PIN (555967922, runtime 0.23.1rc1.dev1511+g555967922 -- verified == the parity-pin block) built from source as a ROCm wheel for gfx1100 in rocm/vllm-dev:base and committed as the image vllm-rocm-oracle:555967922-gfx1100 (recipe: .agents/specs/rocm-m4-oracle.md). Gate result on this box (4x RX 7900 XTX, ROCm 7.14, enforce_eager, batch=1, flock ~/gpu.lock): oracle K=10 per-prompt greedy deterministic in every cell; our ROCm engine 16/16 prompts PASS -- 11/16 strict token-exact vs base greedy, 5/16 near-tie band, max teacher-forced gap 0.125 nats, 0 forward-divergent, 125/125 assertions, backend proof all 8 Qwen3-dense ops selections>0 and declines==0 on device 5. The p0 France/Italy flip is a literal 0.0000-nat tie in the oracle's own logits (the Metal M3b signature, reproduced on ROCm). Change shape: the parity gate's Metal/Tenstorrent device-golden lane gains the kROCM arm (identical anchor+band logic, ROCm's own oracle-backed golden pair); dgx base goldens byte-untouched; the *_rocm goldens are additive. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
…he 0.8B GDN forward divergence, root-caused and fixed Issue mudler#41. The ROCm dispatcher for the fused full-attention preamble keyed its template on the SOURCE dtype (qgate.dtype) instead of the OUTPUT dtype like the CUDA lane (LaunchAttnPreambleOut switches on q_out.dtype). The 0.8B bf16 model runs bf16 QKV-projection output + f32 q/k/gate outs (the f32-attention path; no FA-2 on ROCm), which the src-keyed dispatch mis-launched as all-bf16 -- writing bf16 bits through the f32 out pointers. Evidence chain (gfx1100, flock): CPU==pinned-oracle token-exact / ROCm diverges at tok0; VT_DUMP_ACT per-layer residual dumps: layers 0-2 clean (<=9.6e-3), layer 3 (first full-attn) 2.1e-1; sub-stage dumps: block_out 4.4e-1 with post_input_norm clean; VT_DUMP_ATTN op dumps: fa0_q rms-rel 1.196 with qkv/gate clean; VT_ROCM_ATTN_CPU_REF=1 unchanged. Fix: dispatch on q_out.dtype; bf16-out requires bf16-src (VT_CHECK). Teeth: the cross-device AttnQkNormRopeGate case gains the bf16-src -> f32-out combo at the real 0.8B dims (Hq=8, Hkv=2, Dh=256, rot=64); mutation-proven (fails with the fix reverted, passes with it). Gates on this branch (gfx1100, flock ~/gpu.lock): test_backend_cross_device 19/19 (346 assertions, incl. the new combo); test_qwen3_paged_engine 16/16 unchanged; e2e 0.8B 'The capital of France is' -> ' Paris.\nThe capital of France is' == pinned oracle == our CPU. The M4 0.8B gate with the re-derived GREEN goldens lands in the stacked sibling on row/ROCM-M4-GDN-GATE; full HIP ctest zero-delta vs base measured there. Also: three pre-existing hipcc/clang-only build breaks that block ANY full HIP build on this toolchain: same-line double CAPTURE in test_qwen3_5_gdn_spec_routing.cpp and test_ops_quant_repack.cpp, and -Wunused-const-variable on kGdnLayers (uses compile out on HIP). Debug instrumentation env-gated and off by default (VT_DUMP_ACT / VT_DUMP_ACT_SUB / VT_DUMP_ATTN; deepseek_v4 VT_DUMP_ACT precedent; env-doc-allowlist updated). FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: pi:kimi-k3 [pi]
VikashLoomba
force-pushed
the
row/ROCM-GDN-08B-FIX
branch
from
August 14, 2026 07:38
90b49ee to
fe606b6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(ROCM): AttnQkNormRopeGate dispatch keyed on the OUTPUT dtype — the 0.8B GDN forward divergence, root-caused and fixed (issue #41)
Restructured after review: this PR now carries ONLY the kernel fix + its teeth + the debug instrumentation, and lands BELOW the 0.8B gate (#559) so every commit is individually green on the gate host.
Root cause
The ROCm dispatcher for the fused full-attention preamble templated the kernel on the source dtype (
qgate.dtype) where the CUDA lane (LaunchAttnPreambleOut) switches on the output dtype (q_out.dtype). The 0.8B bf16 model runs bf16 QKV-projection output + f32 q/k/gate outs (the f32-attention path; no FA-2 on ROCm) — silently mis-launched as all-bf16, writing bf16 bits through f32 out pointers.Evidence chain (gfx1100, GPU-locked)
VT_DUMP_ACT): layers 0–2 clean (≤9.6e-3 rms-rel), layer 3 (first full-attn) ignites to 2.1e-1VT_DUMP_ACT_SUB):block_out4.4e-1 withpost_input_normclean → inside FullAttnBlockPagedVT_DUMP_ATTN):fa0_qrms-rel 1.196 withfa0_qkv/fa0_gateclean → the preamble's q/k pathVT_ROCM_ATTN_CPU_REF=1unchanged → attention core exoneratedFix + teeth
Dispatch on
q_out.dtype;VT_CHECKbf16-out requires bf16-src. The cross-device case gains the bf16-src → f32-out combo at the real 0.8B dims (Hq=8, Hkv=2, Dh=256, rot=64) — mutation-proven: fails with the fix reverted, passes with it.Gates (gfx1100, flock)
test_backend_cross_device19/19 (346 assertions)test_qwen3_paged_engine16/16 unchangedThe capital of France is→Paris.\nThe capital of France is== pinned oracle == our CPUAlso in this PR: three pre-existing hipcc/clang-only build breaks that blocked any full HIP build (same-line double
CAPTURE×2;-Wunused-const-variableon a CUDA-gated constant), and the env-gated dump instrumentation that found the bug (VT_DUMP_ACT/VT_DUMP_ACT_SUB/VT_DUMP_ATTN, off by default, allowlisted).