Qwen3.6 NVFP4 baselines (27B and 35B-A3B) must reach vLLM speed parity

[localai-bot]

Both NVFP4 baselines from nvidia/ trail vLLM on decode throughput on GB10 (sm_121a), and neither is at the gate's floor of >= 1.00x on every axis. This issue tracks closing both to parity.

Where we are

Gate model	Revision	Ratio vs vLLM 0.25.0 (production graphed)
nvidia/Qwen3.6-27B-NVFP4	0893e160	0.847x / 0.861x / 0.853x / 0.843x at c1/c2/c4/c8
nvidia/Qwen3.6-35B-A3B-NVFP4	491c2f1e	c1 0.977x, c2 0.964x, c4 1.025x, c8 0.964x, c16 0.932x, c32 0.971x; mean TTFT 0.93-0.98x everywhere

Correctness is not in question on the 27B: the greedy continuation is byte-identical to vLLM, captured from the same warm process as the numbers.

The 27B deficit is flat across the concurrency sweep, so it is not batching or the
scheduler. A decode-window nsys diff (--cuda-graph-trace=node) shows we are
~100% GPU-busy at c1, so there is no host or launch-overhead lever either: the
whole deficit is kernel efficiency. ~96 ms/token for us against vLLM's ~79 ms,
which is 78% of this box's ~273 GB/s bandwidth roof against vLLM's ~95%.

Root-cause theme

A ten-lens scan of the decode hot path against pinned vLLM 555967922 found that
the deficit is dominated by one recurring defect class: we materialize a wider
dtype than vLLM does, at load or at a GEMM output, and then re-read it every
decode step. This is the same class as the already-fixed LoadAttnDense bug,
where the modelopt_mixed FP8 attention tower fell through to a BF16 branch that
dequantized it — worth 18% when fixed.

Confirmed instances, largest first:

1. The NVFP4 lm_head is dequantized to BF16 at load. LoadLmHeadAnyDtype's
   U8 branch (qwen3_5_dense_weights.cpp:266-306) runs
   DequantCtNvfp4WeightToF32 -> BF16 -> TransposeBf16, yielding a dense BF16
   [in,out] operand. We stream ~2.54 GB/step for logits where vLLM streams
   ~0.72 GB. The code comment at :208-210 names this explicitly as deferred
   work. vLLM keeps the head quantized: ModelOptMixedPrecisionConfig.get_quant_method
   handles ParallelLMHead, and _quantized_layer_prefix_candidates appends the
   bare lm_head key. This is the 11.38 ms/step single-call
   cutlass_80_tensorop_s16816gemm_bf16_128x64_32x6_nn_align2 in the profile —
   and storing the operand transposed to [K,N] is also why an SM80 tile is
   selected on an sm_121a part.
2. FP8 tower GEMMs hardcode an F32 output where vLLM's fp8_linear emits the
   model dtype (BF16). Caller-side invocation-parity, the axis our existing GEMV
   guard does not cover.
3. Dense MLP gate_proj/up_proj are never merged on the ModelOpt W4A16
   Marlin route — 192 Marlin launches/step where vLLM issues 128. The merged
   arm exists and is default-ON but is gated on IsTrueW4A4(), which a W4A16
   checkpoint never satisfies. This is also a shared-seam violation: AGENTS.md
   requires mergeable MLP projections to route through
   layers::MlpGateUpMethodBase / vt::MergedGemmGroup, and the Qwen3.6 dense
   path does not.
4. GDN in_proj_qkv + in_proj_z stay two separate FP8 GEMMs. The merged
   path exists and is default-ON, but only for a BF16 owner, so an FP8 tower can
   never reach it.
5. 35B GDN activations are F32 where vLLM's are BF16, and GdnOutDType() is
   gated on num_experts == 0, so the MoE model never got the BF16 flip. This is
   the documented "41 CastF32 launches per step".

Definition of done

• Both models at >= 1.00x vLLM decode throughput at every measured concurrency,
  plus TTFT and memory axes at or above floor.
• Token-exactness preserved against the pinned oracle. Every change here either
  keeps arithmetic identical or moves it toward vLLM's own dtype flow, so
  correctness should improve rather than be traded away.
• Same-tool nsys traces on both arms before any throughput claim, medians of 3
  on an idle box, same-binary A/B for each lever.

Each lever lands as its own row/<ID> PR with a committed spec under
.agents/specs/, per AGENTS.md.

[localai-bot]

The 27B gap is now FULLY ATTRIBUTED — and it is two levers, both traffic/fusion

First measurement of nvidia/Qwen3.6-27B-NVFP4@0893e160 through a reproducible
recipe, on an idle box, with the noise band measured before any delta was read.

vllm.cpp 9ff2742e (= origin/main@c9924566 + PR #229). Oracle: pip vLLM
0.25.0 + FlashInfer 0.6.13, production graphed config, no --enforce-eager.
Build guards confirmed present: FlashAttention-2 … ENABLED for arch(es) [121a]
and VLLM_CPP_TRITON resolved ON with real sm_121a AOT variants.

Throughput — BINDING

Medians of 3, 1024-in/128-out, greedy, ignore_eos, prefix caching off both arms.

	c1	c2	c4	c8
ours tok/s	9.4201	17.2474	29.5132	46.7061
vLLM tok/s	11.3646	20.3858	34.6041	54.0616
ratio	0.8289	0.8461	0.8529	0.8639
TPOT ratio	1.2245	1.1902	1.2313	1.2250
TTFT ratio	1.0077	1.1111	0.9722	0.9992

This confirms the previously recorded 0.843–0.861x band — the ad-hoc numbers
were not wrong. c16/c32 were not measured (see below).

Noise band, measured two ways before interpreting anything: ±0.03% at c1 for
back-to-back legs on a warm server, and 0.29–1.85% leg-to-leg including server
reload + drop_caches, drifting monotonically downward on both arms so it
largely cancels in the ratio. The gaps below are one to two orders of magnitude
outside both.

Other axes

axis	ours	vLLM	ratio
cold launch → /health	33.38 s	182.41 s	5.46x FASTER
peak host RSS	21.10 GiB	13.09 GiB	1.612x — BELOW FLOOR, open gap
peak GPU memory	—	—	PENDING: nvidia-smi returns [N/A] on this GB10 unified part

The step delta closes to four decimal places

Decode-only nsys two-length diff (8 vs 136 tokens), --cuda-graph-trace=node,
both arms in the same tool. Node-level tracing proven — launch counts are
integers to three decimals. Two independent capture pairs agree to 0.005%.

Ours 98.906 ms/step vs vLLM 81.577 ms/step; GPU-busy 99.2% / 99.3%. The c1
step is GPU-bound on both arms, so the entire gap is kernel time — no host,
scheduler or launch-overhead lever exists here.

+8.6414  lm_head / Marlin
+7.6068  FP8 tower
+0.0532  splitK reduce
+1.0279  everything else
= +17.3292 ms/step   vs measured +17.3292

Lever 1 — keep lm_head NVFP4 (~8.1 ms of 17.33)

The checkpoint ships lm_head.weight as U8/NVFP4 (248320×2560 + FP8 scales =
0.6660 GiB). vLLM keeps it packed and runs it as its 129th Marlin launch. We
dequantize to BF16 and read 2.3683 GiB — +1.702 GiB/step of pure avoidable
traffic, 11.183 ms/step, 11.3% of our step, as a lone
cutlass_80_tensorop_s16816gemm_bf16_128x64_32x6_nn_align2.

Our Marlin is not slower than theirs: combined Marlin+lm_head is ours 54.511 ms
for 11.3330 GiB (207.9 GiB/s) vs vLLM 45.870 ms for 9.6309 GiB (210.0 GiB/s).
Same kernel efficiency — we simply move more bytes.

Already implemented on row/PERF-27B-LMHEAD-FP4; this profile sizes it.

Lever 2 — merge the GDN FP8 in_proj_qkv + in_proj_z (~7.9 ms of 17.33)

Both arms read identical fp8 bytes: 7,214,202,880 B = 6.7188 GiB/step.
Effective rate ours 165.9 GiB/s vs vLLM 204.3 GiB/s. Per shape:

	ours	vLLM	delta ms
GDN input projections	25.433 ms, 96 GEMMs, qkv at 129.3 GiB/s	17.555 ms, 48 merged qkvz at 213.6 GiB/s	+7.88
GDN out_proj + attn o_proj	9.751 ms	10.165 ms	−0.41
attn q,k,v	5.352 ms, 3/layer	5.159 ms, merged	+0.19

in_proj_qkv (10240×5120) runs at 60% of the rate the same tower reaches on the
merged shape. Also removes 48 launches/step. See #252 for the full per-kernel
tables and the correction to the earlier library-choice framing — it is shape and
fusion, not library: vLLM's fp8 splits cuDNN (112 merged) + cuBLASLt (64), and
no FlashInfer fp8 kernel executes on either arm.

Together ≈16 of the 17.33 ms/step, which would put c1 kernel time within ~1.5%
of the oracle.

Tokens — near-tie, not a defect

Greedy 32-token continuations are not byte-identical. Asked the oracle for its
own margins first, per the record's rule: 8 of 32 positions have a top1−top2
margin of exactly 0.000000, 16 of 32 are ≤0.25 nats, and at the first divergence
the oracle's top FIVE tokens are all exactly −2.651325 — our arm picked one of
those five. The oracle's own greedy decode is undetermined at half the positions on
this corpus, so this is the ratified distributional case, not a numerical defect.
An exact position-by-position alignment with our own logprobs is still owed.

Not measured / owed

• c16 and c32. Both canonical --execute attempts died: one aborted when a
  foreign 20-way build contended the denominator arm mid-timing (measured bias:
  contended vllm c8 read 0.918 where idle reads 0.841), the other when
  dgx.casa OOM-rebooted — twice today. Both roots preserved as -VOID-* with
  VOID-REASON.txt; PR fix(gate): the online gate has no key for the nvidia 27B, and a skipped model gate records as passed #229's harness fix worked and got past the old abort.
• Our c1 ratio is host-memory-state sensitive. The same binaries read
  0.7604/0.8408 before a reboot and 0.8289/0.8639 after — our arm 6.1% faster,
  vLLM 3.4% slower across it. Protocols also differed, so the variables are not
  separated. Candidate: the documented GB10 weight-residency/ATS state. Open.
• Full GEMM invocation-parity in the AGENTS.md sense (resolved template dtypes,
  algorithm policy) — entry points, launch counts, per-shape times and byte traffic
  are captured on both arms in one tool; resolved template parameters are not.

[localai-bot]

The GDN fp8 merge predicate FIRES — lever 2 is reachable on both gate models

The implementer of PERF-27B-GDN-FP8-QKVZ could not reach the gate host and flagged this as the open question that decides whether the lever is worth anything: the merge only fires if in_proj_qkv.input_scale and in_proj_z.input_scale are bitwise equal. If they differ by an ulp, the guard correctly keeps the legacy two-GEMM path and the ~7.9 ms is worth zero.

Measured directly from the checkpoints' safetensors, comparing the raw f32 bytes (not an approximate compare — bitwise, the same predicate the guard uses):

checkpoint	GDN layers	bitwise EQUAL	DIFFER	verdict
nvidia/Qwen3.6-27B-NVFP4@0893e160	48	48	0	predicate fires on every layer
nvidia/Qwen3.6-35B-A3B-NVFP4@491c2f1e	30	30	0	predicate fires on every layer

So the lever is real on both gate models, and the 35B gets it for free from the same change — it shares this FP8 tower. That matters because the 35B's own parity gap has never been attributed, and this is the first lever known to apply to it.

This also confirms the implementer's circumstantial reasoning without relying on it: it argued the scales probably match because FuseRmsNormFp8QuantEnabled is default-ON and requires exactly this equality on this tower. That was inference; this is measurement.

Probe: reads the safetensors headers directly, no torch/safetensors dependency, so it runs anywhere the checkpoint does — ~/probe_scales.py <snapshot> on the gate host.

What lever 2 still owes

Every GPU gate is unrun (that agent had no CUDA and no device). Specifically: the CUDA build, the two new CUDA cases (merged == split bitwise, and the pre-capture resident being non-null), test_qwen27_paged_engine 235/235, test_qwen36_paged_engine 315/315, the greedy correctness gate, the VT_GDN_MERGED_QKVZ_FP8=0|1 A/B, and the nsys 96 → 48 structural check.

Two caveats it recorded that a reviewer should not miss:

1. Its launch-count assertion is inert on the model that test gates. The 48-merged/96-split check went into test_qwen27_paged_engine.cpp, which pins unsloth/Qwen3.6-27B-NVFP4 — whose GDN input projections are BF16, so the check skips by construction. The binding 96 → 48 evidence is the nsys structural check on the nvidia checkpoint, not that assertion.
2. Prefill scratch grows ~190 MB transient at T=8192 (merged f32 output is 64 KB/token vs 40+12 KB split). Negligible at decode; worth watching on a unified-memory box that has rebooted repeatedly today.

Both levers now have a measured basis: lm_head is operator-gated on real hardware (PR #263, 4/4 including SACRED), and this one's enabling predicate is confirmed on 78 of 78 GDN layers across the two models.