cuda: fix routed MMVQ fusion dispatch for quantized mul_mat_id

[rapatel0]

Summary

ggml_cuda_mul_mat_vec_q was lowering routed MUL_MAT_VEC_FUSION into the dedicated mul_mat_vec_q_moe kernel whenever ids != nullptr. That MoE kernel computes only the routed matmul — it does not apply x_bias, gate, gate_bias, or GLU. So FUSION metadata was being silently dropped on every routed dispatch.

This affects every CC; it's a dispatch shortcut bug, not a code-gen issue.

Repro

test-backend-ops -p mxfp4 on MUL_MAT_VEC_FUSION with use_id=1:

Configuration	Error vs CPU ref	Tolerance
with_bias=1, with_gate=0	0.013	0.005
with_bias=0, with_gate=1	7.18	0.005
with_bias=1, with_gate=1	8.15	0.005

Pattern explanation:

• Bias-only: just the bias offset missed → small error
• Gate-enabled: entire GLU multiplication missed → catastrophic
• Non-routed FUSION (use_id=0): passes (different dispatch path)
• Non-fused MUL_MAT_ID: passes (MoE-only kernel is correct for that case)

Fix

Two changes in ggml/src/ggml-cuda/mmvq.cu (8 insertions, 5 deletions):

1. Dispatch shortcut: mul_mat_vec_q_switch_ncols_dst now uses the dedicated MoE kernel only when ids != nullptr && !has_fusion. Routed FUSION falls through to the existing fusion-capable generic MMVQ kernel, which already applies the main path, bias, gate path, gate bias, and GLU activation.

2. Routed ID indexing: the generic mul_mat_vec_q kernel's routed single-column path now reads ids[channel_dst + token_idx*ids_stride] and sets sample_dst = 0 in routed mode. Matches the existing F16/BF16 fusion path in mmvf.cu and the dedicated MMVQ MoE kernel layout, and corrects ID lookup when there is more than one token worth of routing rows.

Validation

Built + tested on V100 (sm_70) inside a CUDA 12.2 dev pod against this branch's HEAD:

./test-backend-ops -p mxfp4 --output csv | grep -cE "FAIL|FAILED"

	Before	After
Total MXFP4 records	1352 (80 failed)	1352 (0 failed)
MUL_MAT_VEC_FUSION records	120 (76 failed)	120 (0 failed)
Routed FUSION (use_id=1)	80 (76 failed)	80 (0 failed)
MUL_MAT, MUL_MAT_ID (non-fused)	PASS	PASS (no regression)

test-backend-ops -p f8_e4m3_b128 still reports 0 failures over 12 records.

Test plan

• Built clean on V100 sm_70 (verified)
• test-backend-ops -p mxfp4 0 failures (verified)
• test-backend-ops -p f8_e4m3_b128 no regression (verified)
• (reviewer) verify on sm_80+ — same bug should reproduce there with the same test pattern, and the fix should resolve it

Notes

Found while bringing up DeepSeek-V4-Flash native FP4/FP8 (MXFP4 experts + F8_E4M3_B128 dense) on 4× V100 32GB hardware against a pruned 8-expert GGUF. Without this fix, MoE decode produces garbage on the routed FUSION path used for the expert FFN dispatch.

🤖 Generated with Claude Code