The FP8 tower's cuBLASLt default rests on a false premise: vLLM selects FlashInferFP8ScaledMMLinearKernel, not nvjet/cuBLASLt (26.7% of the 27B decode step)

[localai-bot]

Our FP8 W8A8 tower defaults to a cuBLASLt GEMM, and the code justifies that
choice by asserting it mirrors vLLM. It does not. vLLM runs FlashInfer here.

The claim

src/vllm/model_executor/models/qwen3_5.cpp:1286-1288:

By DEFAULT the GEMM is cuBLASLt fp8 (vt::MatmulFp8CublasLt — mirrors vLLM's
nvjet_qqtst fp8 dense); VT_DENSE_CUBLASLT_FP8=0 selects the cutlass sm120
fp8 GEMM (vt::MatmulFp8Cutlass).

The measurement

From the pinned oracle's own startup log, captured on dgx.casa while serving
nvidia/Qwen3.6-35B-A3B-NVFP4@491c2f1e in its production graphed configuration
(vLLM 0.25.0, linear_backend='auto', six independent server starts):

6  Selected FlashInferFP8ScaledMMLinearKernel

Not cuBLASLt. Not CUTLASS. nvjet_* is a cuBLASLt kernel family — vLLM does
not call cuBLASLt for this at all, so "mirrors vLLM's nvjet_qqtst fp8 dense"
cannot be true as written.

Why it matters for #213

The FP8 tower is the second-largest mass in the 27B decode step. From the
per-kernel attribution:

kernel	calls/step	ms/step	share
nvjet_sm121_qqsss_mma_64x128x128	48	18.51	16.2%
nvjet_sm121_qqtst_mma_64x64x128	48	7.05	6.2%

25.6 ms/step combined, ~26.7% of the step, reading 7.789 GiB of F8_E4M3
(5.156 linear_attn, 1.562 self_attn, 0.996 mlp). The 27B currently runs
0.843–0.861x against vLLM with a ~14.7 ms/step deficit, and the verified lever set
accounts for only about two thirds of it. A whole kernel family chosen on a false
premise, over a quarter of the step, is the largest unexamined structural
divergence left.

This is also a POL-MIRROR-VLLM matter, not only a performance one: AGENTS.md
says to mirror vLLM and to ground conclusions in the whole executing chain —
"FlashInfer, CUTLASS, cuBLASLt, DeepGEMM, torch/Inductor, generated code, and
local dispatch" — citing the file:line ported from. The comment cites a kernel
family the oracle never enters.

What is owed

1. Correct the comment regardless of what follows. A false upstream claim in a
   dispatch decision is how the next reader inherits the mistake.
2. Establish what FlashInfer's fp8 scaled-MM actually does at decode M on this
   GPU — the tile, the epilogue, the scaling granularity (per-tensor vs
   per-channel vs per-token), and where activation quantisation happens — and
   compare against MatmulFp8CutlassD / MatmulFp8CutlassPreQuantD and the
   cuBLASLt heuristic we currently reach.
3. Note a related gap already recorded: cuda_matmul_fp8_cutlass.cu:125-129
   has never compiled the M<=16 / M<=32 decode tile configs that vLLM's path
   carries. So the earlier A/B that made cuBLASLt the default was run against a
   CUTLASS arm missing exactly the small-M configs decode needs — the comparison
   that chose the current default was not a fair one.
4. Trace both sides with the same tool before any throughput claim, per
   AGENTS.md: source inspection establishes candidates, matching traces establish
   what ran.

Reproducing the evidence

Start the pinned oracle on either NVFP4 gate model in its production
configuration and grep the server log:

grep -o 'Selected [A-Za-z]*FP8[A-Za-z]*Kernel' <server.log>

Logs from the run above are on the gate host under
~/work/oracle-ssm-ab/q35/*server.log.

Refs #213.

[localai-bot]

CORRECTION — my headline claim was too strong

I traced this to the bottom and the framing above is wrong in an important way. Correcting it before anyone acts on it.

What I got right: vLLM does select FlashInferFP8ScaledMMLinearKernel (6/6 server starts, oracle's own log), and the comment at qwen3_5.cpp:1287 does not describe what vLLM does.

What I got wrong: "FlashInfer, not cuBLASLt" is a false dichotomy. FlashInfer is a dispatcher here, not a kernel family. The real chain:

FlashInferFP8ScaledMMLinearKernel.apply_scaled_mm (vllm/model_executor/kernels/linear/scaled_mm/flashinfer.py:72-85)
→ flashinfer_scaled_fp8_mm (vllm/utils/flashinfer.py:864-890)
→ bmm_fp8(a.unsqueeze(0), b.unsqueeze(0), scale_a, scale_b, out_dtype, **"auto"**)

And in the installed flashinfer (flashinfer/gemm/gemm_base.py:6408+):

backend: Literal["cudnn", "cublas", "cutlass", "auto"] = "cublas"
...
if backend == "auto":
    backends = bmm_fp8.suitable_auto_backends

So the backend defaults to cublas, and vLLM explicitly overrides it to auto — selecting among cuDNN / cuBLAS / CUTLASS, with enable_flashinfer_autotune=True in the engine config (confirmed in the same server banner). cuBLAS is one of the candidates. If autotune picks it for these shapes, then vLLM's fp8 GEMM lands on the same cuBLASLt kernels we launch, and there is no kernel-family gap at all.

The finding that actually survives

Restated conservatively, and it is narrower but still real:

1. vLLM autotunes across three backends per shape; we hardcode one. VT_DENSE_CUBLASLT_FP8 selects cuBLASLt unconditionally. Even if cuBLAS wins most shapes, "wins by autotune" and "chosen unconditionally" differ wherever cuDNN or CUTLASS would win — and the fp8 tower spans several distinct shapes (q/k/v, GDN in_proj_qkv/z, o_proj/out_proj).
2. The comment is still unsupported as written. vLLM does not hardcode nvjet_qqtst; it may reach cuBLAS via autotune, which is a different statement with different consequences. "Mirrors vLLM" should not survive unless a paired trace shows the same kernel for the same shape.
3. The CUTLASS small-M gap stands independently — cuda_matmul_fp8_cutlass.cu:125-129 never compiled the M<=16 / M<=32 decode configs, so the A/B that made cuBLASLt our default compared against a CUTLASS arm missing the configs decode needs. That was an unfair comparison regardless of what vLLM does.

What settles it

Only a paired trace. Per AGENTS.md, source inspection establishes candidates; matching traces establish what ran. The decode-only nsys profile now in flight for #213 captures both arms with the same tool, so it will show whether vLLM's fp8 projections land on nvjet_* (same as ours) or on a cuDNN/CUTLASS kernel. Nobody should act on this issue until that lands.

My apologies for the overclaim — I inferred a kernel family from a dispatcher name and filed before following the call chain to its backend selection. That is precisely the error this repo's own rule warns about: ground conclusions in the whole executing chain, including FlashInfer, CUTLASS, cuBLASLt and local dispatch.

[localai-bot]

SETTLED BY PAIRED TRACE — and my premise was wrong twice

A decode-only nsys two-length diff (8 vs 136 tokens, --cuda-graph-trace=node),
both arms in the same tool, on an idle box. Node-level tracing proven: launch
counts are integers to three decimals (192.000, 112.000, 64.000, 48.000, 1.000).

No FlashInfer fp8 kernel executes on either arm. The only flashinfer::
kernels in vLLM's decode trace are the paged-attention pair. So
FlashInferFP8ScaledMMLinearKernel is selected at config time but on sm_121
dispatches into cuDNN.

And "vLLM never enters cuBLASLt for this" — my original claim — is also false.
vLLM enters cuBLASLt for half its fp8 projections.

What each arm actually launches

ours — 208 fp8 GEMMs, all cuBLASLt, 40.487 ms/step

ms/step	launches	kernel
18.120	48	nvjet_sm121_qqsss_mma_64x128x128… — GDN in_proj_qkv
17.064	112	nvjet_sm121_qqtst_mma_64x64x128… — GDN in_proj_z + out_proj + attn o_proj
4.537	16	nvjet_sm121_qqsss_mma_192x24x128… — attn q_proj
0.816	32	nvjet_sm121_qqsss_mma_128x32x128…splitK — attn k/v

vLLM — 128 fp8 GEMMs, 32.880 ms/step, split across TWO libraries

ms/step	launches	kernel
17.555	48	cudnn_generated_fort_native_sm120_matMul… — GDN in_proj_qkvz merged
10.165	64	nvjet_sm121_qqtst_mma_128x72x128… — GDN out_proj + attn o_proj (cuBLASLt)
5.159	16	cudnn_generated_fort_native_sm120_matMul… — attn qkv merged

The real finding: shape and fusion, not library

Bytes/step are identical — both arms read the same 208 F8_E4M3 tensors,
7,214,202,880 B = 6.7188 GiB/step. Effective rate: ours 165.9 GiB/s vs vLLM
204.3 GiB/s, 23.2% faster at identical traffic. So this was never a
kernel-quality or library gap.

Per shape, the deficit is one place:

	ours	vLLM	delta ms
GDN input projections	25.433 ms, 96 GEMMs, qkv at 129.3 GiB/s	17.555 ms, 48 merged qkvz, 213.6 GiB/s	+7.88
GDN out_proj + attn o_proj	9.751 ms, 192.3 GiB/s	10.165 ms, 184.5 GiB/s	−0.41
attn q,k,v	5.352 ms, 3 GEMMs/layer	5.159 ms, merged	+0.19

Our in_proj_qkv (10240×5120) runs at 129.3 GiB/s — 60% of the 213.6 GiB/s the
same tower reaches on the merged shape. Issued at vLLM's rate it would cost
10.97 ms instead of 18.12 ms.

Disposition

The comment at qwen3_5.cpp:1286-1288 is still wrong and still owes a fix, but
for none of the reasons I gave: not FlashInfer-vs-cuBLASLt, not library choice.
Merging the GDN fp8 in_proj_qkv + in_proj_z into one qkvz GEMM is the
lever — worth ~7.9 ms of the 17.33 ms/step deficit and removing 48 launches/step.
That is the fp8-side twin of the existing gdn-merged-input-projections row.

The CUTLASS small-M gap (cuda_matmul_fp8_cutlass.cu:125-129) remains a separate,
still-valid observation, but it is no longer on the critical path.

Retitling this issue to match what was measured would be appropriate.