ROCm gfx1200: Qwen3-0.6B produces wrong greedy output despite all-native execution

[joral]

ROCm gfx1200 (RX 9060 XT): Qwen3-0.6B produces wrong greedy output despite all-native execution — embedding gather cleared

Board: AMD Radeon RX 9060 XT, gfx1200 (Navi 44, RDNA4, discrete). ROCm
7.2.3, hipClang/Clang 22.0.0. M0/M1 independently verified on this board (see
my earlier comment).

Repro:

./build-hip/examples/vllm-cli --model <Qwen3-0.6B, bf16 safetensors> \
  --prompt 'The capital of france is' --max-tokens 8 --temperature 0

• CPU (--device cpu): Paris. The capital of the United States — correct.
• ROCm (default auto, picks kROCM): 1000000 — garbage.

No crash, no missing-kernel throw, no reference-tier fallback (discrete board,
none available). VT_OP_PROVIDER_STATS=1 shows every exercised op resolving
selected=vt-native: kEmbedding, kRopeCosSinCache, kCastBf16, kRmsNorm,
kMatmulBT, kQkvSplit, kRopeFromCache, kReshapeAndCache,
kPagedAttention, kSiluAndMul, kGreedyArgmax. All of these have
cross-device coverage in test_backend_cross_device.cpp and pass there against
synthetic data — so this is either a real-shape/real-value edge case none of
that synthetic coverage hits, or a bug outside the op kernels themselves.

Isolated to one token. Capitalizing the prompt ('...of France is') gets
the correct answer ( Paris. The capital of Italy is Rome). Both prompts
tokenize to 5 tokens, identical except position 3: 47587 ("france") vs
9625 ("France"). Same shape, same op sequence — one input token id differs
and the output goes from coherent to garbage.

Ruled out:

• The is_cuda()-vs-is_cpu() host-pointer-aliasing defect at
  dense_attn_block.h:181 — already fixed generically, confirmed correct for
  kROCM in current source.
• The embedding gather itself. Pulled model.embed_tokens.weight directly
  from the safetensors file and ran vt::Embedding on CPU and ROCm at the
  real row indices 47587 and 9625 (not a repacked small table — same offset
  arithmetic as production). Both rows are bit-exact against the CPU
  oracle. The table bytes and the gather kernel are correct for this row; the
  bug is downstream, in the per-layer forward.

Not yet root-caused. Candidates: kRmsNorm (cross-lane reduction),
merged-QKV kMatmulBT (hipBLASLt GEMM), RoPE, kPagedAttention (hand-written,
least test coverage), or kGreedyArgmax (its ROCm ArgmaxK NaN handling looks
suspect by inspection — a NaN never satisfies v > best, so a fully-NaN row
would leave arg at a sentinel 0x7fffffff rather than behaving like the CPU
path; unconfirmed as the cause here, but cheap to check first since it's the
last op before the wrong token is picked).

Full writeup, evidence, and next steps: .agents/specs/rocm-gfx1200-m2-correctness.md
(this PR/branch — will link once posted).

Distinct from #201 and #132 — neither reproduced on this board/toolchain.
Does not touch the in-flight Qwen3.5 GDN kernel work (disjoint op set, disjoint
board).

[joral]

Root-caused: this is a near-tie flip, not a kernel defect

Dug through the whole pipeline. Short version: not a bug to fix — the
backend computes the right thing to within ordinary cross-hardware bf16
tolerance, and lost an unusually close near-tie on this specific prompt.

The logits. Final-position top-5, same prompt, same greedy/temp=0:

	#1	#2	#3	#4	#5
CPU	ĠParis 15.1216	Ġlocated 15.0541	Ġ 15.0483	Ġthe 14.9142	Ġin 14.5216
ROCm	Ġ 15.1009	ĠParis 15.0907	Ġlocated 15.0620	Ġthe 14.9496	Ġin 14.5309

Same five candidates, same magnitudes, clustered within ~0.07 logit units of
each other (max logit ≈15) on both backends. CPU resolves toward Paris by
a margin of 0.0675; ROCm resolves toward Ġ (a bare leading-space token —
content-free) by just 0.0102. Picking a content-free token derails the rest of
greedy decode into out-of-distribution territory, which is why the completion
reads as total garbage rather than a merely-different word — the mechanism is
an ordinary near-tie flip, not a broken computation.

Per-layer drift (L2 norm of the last token's hidden row after every
decoder layer): oscillates in a tight 0.01%–0.6% band across all 28 layers,
present from layer 0 (0.27%), no jump, no blow-up, sometimes CPU higher and
sometimes ROCm higher. Textbook ordinary bf16/reduction-order noise
accumulating uniformly, not a localized op bug — full table in
.agents/specs/rocm-gfx1200-m2-correctness.md.

Ruled out along the way:

• The is_cuda()-vs-is_cpu() host-pointer defect at dense_attn_block.h:181
  — already fixed generically, confirmed correct for kROCM.
• The embedding gather — pulled model.embed_tokens.weight straight from the
  safetensors file, ran the real op at the real row indices (47587/9625) on
  both backends, bit-exact.
• A localized op-level bug — the per-layer table shows uniform drift from
  layer 0, not a break at any specific op.

Not attempted: the project's own teacher-forcing decisive-measurement
method (scripts/qwen3-neartie-gap-transformers.py) would be the rigorous
confirmation that the oracle itself treats this as a genuine near-tie — needs
a real vLLM install, not set up on this box. Flagging as environment-blocked
rather than skipping silently; the evidence already gathered (identical top-5
set, sub-1% uniform drift, zero NaN/Inf anywhere) is strong enough that this
isn't treated as blocking closure.

Full writeup: .agents/specs/rocm-gfx1200-m2-correctness.md (this
branch/PR).

[VikashLoomba]

Cross-lane corroboration for the near-tie conclusion — the same phenomenon on two other boards, reported on #41:

• gfx1100 (4× RX 7900 XTX, discrete, ROCm 7.14), current main, Release: Qwen3-0.6B greedy 'The capital of France is' → Paris. The capital of Italy is Rome (deterministic 3/3), CPU on the same build → Paris. The capital of France is also. Second prompt 'The sky is': 8/8 token parity with CPU. Full table.
• gfx1103 (Radeon 780M, integrated, TheRock nightly), reproduced independently by @arch-btw: identical ROCm and CPU sequences, token-for-token, including the same token-3 flip and the same 3/3 determinism. Report (his is linked from mine above).

So three independent boards (two discrete RDNA, one integrated), two toolchains, produce the identical ROCm-side sequence while both CPU runs agree — exactly the signature your per-layer drift table predicts (uniform bf16/reduction-order noise, no localized break). One addition to the "why garbage" mechanism: on gfx1100 the flipped token was a content word (Italy), so the continuation stayed coherent — which is why the gfx1100/gfx1103 runs read as "plausible but different" while your Ġ flip reads as garbage. Same coin, two faces.

This also bears on the M2 recording question pending on #41: with three boards in the same regime, "strict greedy parity vs the CPU backend" will keep producing attempted-not-met results on near-tie prompts. The distributional gate (ours inside the oracle's K-run set) looks like the honest bar for M2 on ROCm — maintainer's call, flagged in my #41 follow-up.

[joral]

I have only seen this on Qwen3-0.6B so far. The larger qwen3 models that will fit my card (I think 1.3B and 4B? something like that) I have not run into this behavior. I haven't spent a lot of time trying to recreate it, but I wonder if it's just something about that particular model or its size.

[VikashLoomba]

That size-dependence fits the mechanism: smaller models have flatter top-logit gaps (your 0.6B top-5 were all within ~0.07 units), so greedy near-ties come up far more often than at 1.3B/4B where the top token usually separates clearly. Same reason the gate-model record matters more than any single prompt: at this size some prompt in any set will be near-tied. The K-run distributional gate absorbs exactly this class — a single flipped token isn't a failure, being outside the oracle's own run-to-run set is.