perf(PERF-27B-LMHEAD-FP4): keep the ModelOpt NVFP4 lm_head packed (#213)

[localai-bot]

Row PERF-27B-LMHEAD-FP4 — issue #213 — spec
.agents/specs/perf-27b-lmhead-nvfp4.md.

The dense Qwen3.6 loader no longer dequantizes a ModelOpt NVFP4 lm_head to
BF16: it keeps the head packed and runs the logits GEMM on the same Marlin
W4A16 family vLLM pins for it. Commits 1-2 are the original spec and
implementation; commit 3 resolves round-1 review findings; commit 4 resolves
round-2, rebased onto origin/main a0fa12c7.

Scope, stated precisely: this changes the dense lm_head and nothing else.
Every other NVFP4 projection — MLP gate/up/down, attention o_proj, GDN
out_proj, MoE shared experts — is byte-unchanged on every backend.

Why this row matters: it is half the parity gap, and it is pure traffic

Operator decode-only nsys two-length diff (8 vs 136 tokens,
--cuda-graph-trace=node, both arms same tool, idle box) attributes the entire
27B deficit:

Component	ms/step
lm_head / Marlin	+8.6414
FP8 tower	+7.6068
splitK reduce	+0.0532
everything else	+1.0279
total	+17.3292 (vs measured +17.3292)

The nvidia checkpoint ships lm_head.weight as U8/NVFP4 (248320x2560 + FP8
scales = 0.6660 GiB) and vLLM runs it as its 129th Marlin launch. Dequantizing it
made us read 2.3683 GiB instead — +1.702 GiB/step of avoidable traffic,
11.183 ms/step, 11.3% of our step, as a lone
cutlass_80_tensorop_s16816gemm_bf16_128x64_32x6_nn_align2.

This refines the earlier framing: our Marlin is not slower than vLLM's.
Combined Marlin+lm_head is ours 54.511 ms for 11.3330 GiB (207.9 GiB/s) vs
vLLM 45.870 ms for 9.6309 GiB (210.0 GiB/s) — same kernel efficiency, we simply
moved more bytes. So this is a pure traffic win, the cleanest kind.

Measured (gate checkpoint nvidia/Qwen3.6-27B-NVFP4@0893e160, GB10)

Axis	Result
Peak host RSS	21.06 → 19.36 GiB (−1.70 GiB) — re-measurement OWED. The reading predates ENG-LOAD-DIRECT-UPLOAD (#150), which rewrote LoadCtNvfp4Raw to borrow mmap'd bytes and so changed what host RSS counts. Marked owed in BENCHMARKS/ENVIRONMENT/FEATURES/STATUS/NOW and the spec rather than carried as accepted.
Greedy continuations	byte-identical between our OWN packed and dequant arms — solid. This is an arm-vs-arm claim, NOT ours-vs-oracle: the oracle's own greedy is undetermined at ~8 of 32 positions on this corpus (top1-top2 margin exactly 0.000000; at one divergence its top five tokens were all exactly -2.651325), so cross-engine token diffs are judged against those margins, the ratified distributional case.
test_qwen27_paged_engine	235/235, unchanged
Throughput	packed 11.197/11.193 vs dequant 9.418/10.163 tok/s — indicative only: packed wins every leg and the packed legs agree to 0.04%, but the dequant legs disagree by 7.9%, so the direction is established and the magnitude is NOT. A binding grid is owed.

These CUDA numbers are the reviewer's and the operator's, quoted as theirs. No
CUDA gate was run in either fix round; a CUDA re-run on the changed head is
owed to the operator.

Round-1 findings resolved

1. MEDIUM-HIGH — a compressed-tensors-named NVFP4 head silently became W4A4
   (alpha = 0.0078125, IsTrueW4A4() == true), contradicting the code's own
   comment and docs/USAGE.md, taking the fp4-activation GEMM vLLM refuses for
   an output head, and making the pre-capture Marlin build early-return. The head
   is now W4A16 under both spellings unless VT_MODELOPT_W4A4=1, mirroring
   ModelOptNvFp4W4A16LinearMethod, which deletes input_scale
   (modelopt.py:1365, registered :1358 — re-verified against the pinned
   oracle 555967922). The masking bug is fixed too: the model loader's head
   probe accepts the CT naming, so a CT head is no longer read as
   tie_word_embeddings.
2. MEDIUM — a backend without an fp4 GEMM dequantized the whole head on
   EVERY forward call (~2.54 GB per step at the gate model, where the old code
   paid it once at load). Nvfp4Weight gains a d_dequant_b resident built once
   like d_packed, and the registry prepare hook builds it up front.
3. MEDIUM — the paged arms and the pre-capture build had no coverage. Both of
   the reviewer's exact mutations now go RED.
4. MEDIUM — the unreachable if (lm_head_fp4.Empty()) guard in
   PrepareBf16Resident is deleted; the RSS win is the loader no longer building
   the f32 + bf16 arrays.
5. LOW-MED — "~2.3 GiB" corrected to the arithmetic and measured 1.70 GiB.
6. Records — Qwen3.6 NVFP4 baselines (27B and 35B-A3B) must reach vLLM speed parity #213 in the roadmap issue table, FEATURES.md cell made
   self-consistent, BENCHMARKS.md entry with the indicative/solid split, spec
   anchors and the IsQwen27QuantizedLinear deviation recorded, and the new test
   moved out of test_qwen27_paged_engine's configuration block.

Round-2 findings resolved

1. BLOCKER — finding 2 above traded a per-call temporary for a lifetime
   resident on every non-CUDA backend, for every NVFP4 projection. The caching
   sat inside MatmulNvfp4{F32,Bf16}D, which also serve DenseMlpBlock
   gate/up/down, attention o_proj, GDN out_proj and the MoE shared experts;
   kMatmulNvfp4 is registered CUDA-only, so CPU, Vulkan, Metal, HIP and
   Tenstorrent all take that fallback. Their steady state went from packed-only
   to packed plus a bf16 expansion of the whole tower — roughly 4x the packed
   bytes, on the backends where Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark #203 already reports the 27B peaking at
   100.8 GiB. CUDA is unaffected, which is why the CUDA gate could not see it.

   Residency is now a property of the weight, not of the GEMM:
   Nvfp4Weight::keep_dequant_b, default OFF, set by LoadDenseLmHead and by
   nothing else. Per NVFP4 weight, steady state, on a backend with no fp4 GEMM:

   	packed bytes	dequantized bf16 2*K*N
   before this row	resident	per-call temporary
   after round 1	resident	resident, for every one
   now	resident	resident for the head alone

   Alternatives weighed and rejected (recorded in the spec): a per-backend
   switch, and dequantizing the head into lm_head at prepare time (the prepare
   hook may hold BORROWED const weights, so only mutable state is writable).

   New coverage: the residency case now populates an NVFP4 tower as well as
   an NVFP4 head and asserts no tower projection holds a d_dequant_b after two
   forwards. RED under the mutation that removes the guard — 6 failing
   assertions, 3 projections × 2 layers, exactly the round-1 behavior.

2. Overstated claim / seam divergence — the round-1 caching landed on
   qwen3_5.cpp's private dispatcher while shared-seam
   dense_nvfp4::MatmulNvfp4W4A16D still rebuilt K*N bf16 per call. That
   parallel dispatcher predates this row: dense_nvfp4_gemm.h was extracted
   from qwen3_5.cpp's anonymous namespace and its own SCOPE comment records
   that the true-W4A4 path stays private there, so the two carry independent
   Dev, MakeTensor, ResidentNvfp4 and DequantNvfp4ToBLayout copies.
   Unifying them is a refactor this row does not do; the exception is now
   recorded explicitly in the spec. The opt-in instead lives on the shared data
   type Nvfp4Weight, which both dispatchers can read, defaulted OFF. They
   can still disagree — dense_nvfp4_gemm.h:626-631 ignores the flag
   outright and rebuilds the per-call temporary for every weight — they merely
   do not, because no weight reachable from MatmulNvfp4W4A16D opts in today,
   and the divergence direction is benign: the shared seam UNDER-caches, so it
   cannot reintroduce the whole-tower expansion. Honoring the flag there belongs
   to the unification refactor. This PR body and the commit messages claim only
   the dense lm_head.

3. LOW — anchor drift in the file this row added: the input_scale delete
   was cited as modelopt.py:1359-1362 at two sites; at the pinned oracle that
   range is a blank line, the def and its docstring. Corrected to :1365.

4. LOW — docs/USAGE.md advertised K*N/2 + K*N/16 unqualified; it now
   states that a backend with no fp4 GEMM pays the packed bytes plus one
   bf16 operand, built at prepare, and that only the head is kept that way.

5. Judgement calls — MakePackedHead had external linkage and built its fp4
   operands through MakeOwned's f32 branch (4 B/element over-allocation); it is
   now internal and exactly sized. IsQwen27QuantizedLinear's "lm_head is never
   quantized" rule is deliberately left alone: deleting it and its checked-in
   expectation costs product review budget this PR does not have, and the
   function still has zero production callers, so it moves no behavior.

Round-3 findings resolved

1. The load-bearing claim had zero coverage. keep_dequant_b is documented
   as "set by LoadDenseLmHead and by nothing else", and the round-2 fix is
   correct as written — but nothing enforced the scope. The reviewer added one
   line to LoadNvfp4AnyNaming (qwen3_5_dense_weights.cpp:377):

   Nvfp4Weight r;
   r.keep_dequant_b = true;   // a second setter

   and every gate stayed green: test_qwen27_dense_lmhead_fp4 6/6 1170/1170,
   test_qwen27_paged_forward 20/20 702/702, test_mtp_speculator 13/13
   171/171. That function is what every dense NVFP4 tower projection flows
   through — MLP gate/up/down via LoadDenseMlp, attention q/k/v/o via
   LoadAttnDense, GDN out_proj via LoadGdnDense — so the mutation reopens
   the round-2 blocker verbatim. It escaped because the residency case built its
   tower with MakeNvfp4Weight (direct struct construction, never a loader), and
   because no test in the repo called LoadDenseMlp, LoadDenseAttn or
   LoadQwen3_5DenseWeights at all. The CUDA gate is blind by construction:
   kMatmulNvfp4 is registered on CUDA, so ResidentNvfp4DequantB is never
   reached there.

   This was the third instance of one pattern on this row — the code right at
   every site it is tested at, its scope unpinned — so the new case pins the
   invariant, not the call site. It loads a fully-NVFP4 dense layer of BOTH
   layer types, under BOTH namings (ModelOpt weight_scale_2 and
   compressed-tensors weight_packed), through the real LoadQwen3_5DenseLayer,
   and sweeps every Nvfp4Weight the layer struct owns — fields, not call
   sites — asserting the flag is false on all of them and true on the head. A
   census assertion (11 routed projections per naming) stops a fixture that
   silently stopped producing NVFP4 weights from reading as a pass.

   RED mutation	failures
   keep_dequant_b = true in LoadNvfp4AnyNaming's ModelOpt arm (the reviewer's)	11
   the same in LoadCtNvfp4Raw (the other naming arm)	11
   a genuinely FOURTH setter, one line after LoadDenseMlp's down_proj load	4

   Each restored byte-for-byte; green after: 7/7, 1314/1314. The third
   mutation is the one that matters for the ask: it is outside
   LoadNvfp4AnyNaming, and the sweep still catches it.

   LoadQwen3_5DenseLayer gains a has-taking overload declaration. The
   resolver-only overload answers has with a constant true, which forces every
   projection down the compressed-tensors spelling and so cannot reach the
   ModelOpt arm at all. The definition already existed; only the declaration is
   new, and it is the sole non-test edit in this round.

2. Overstated claim, corrected. The spec and this body said the shared
   Nvfp4Weight opt-in means "the two dispatchers cannot disagree about a
   weight". They can — dense_nvfp4_gemm.h:626-631 ignores the flag outright.
   They merely don't, because nothing reachable there sets it, and the
   disagreement direction is benign (the shared seam under-caches, never
   over-caches). Reworded in the spec and above.

3. The CPU footprint direction was undisclosed. Every record carried only the
   CUDA -1.70 GiB. On a no-fp4-GEMM backend the head holds 0.666 GiB packed
   plus a 2.368 GiB bf16 operand. Arithmetic from the same K*N, now in
   docs/USAGE.md, docs/BENCHMARKS.md and the spec:

   Backend	before	after	delta
   CUDA	2.368 bf16	0.666 packed	-1.70 GiB
   Vulkan (unified, Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark #203)	2.368 host bf16 + 2.368 device copy	0.666 packed + 2.368 device bf16	-1.70 GiB
   plain CPU	2.368 bf16	0.666 + 2.368	+0.67 GiB

   So Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark #203's backend genuinely improves and plain CPU genuinely regresses by the
   packed head's own bytes — paid once, instead of rebuilding 2.368 GiB on every
   decode step as that backend did before this row.

4. A backend-asymmetric hard abort, now recorded. PrepareLmHeadResident
   (qwen3_5.cpp:6534) aborts for any Qwen3_5DenseWeights whose lm_head_fp4
   is non-empty but did not come from LoadDenseLmHead, and only on a backend
   with no fp4 GEMM (CUDA returns at :6533 first). Unreachable in production, and
   preferred to a silent fall-through because a head that quietly lost its opt-in
   would rebuild 2.54 GB per step with no symptom but throughput.
   tests/vllm/models/test_qwen27_paged_forward.cpp:1377 constructs exactly such
   a weight and passes only because it never calls Prepare.

Merge with origin/main

Merged origin/main @848d4a87 (22 commits) rather than rebasing, so no pushed
commit is rewritten. Keyed records were resolved by taking the target-branch
version wholesale and reapplying this row's scoped edit — never an automatic
three-way combination:

• .agents/NOW.md — main's shortened f32-out / invocation-parity / MiniMax rows
  kept; the 27B key was byte-identical on base and main, so this row's edit
  is reapplied verbatim (compressed by 3 characters to stay inside the
  6,000-char budget, which main now sits exactly at).
• docs/STATUS.md — main's 35B binding-grid rewrite (f500d1a6) kept; the 27B
  key was likewise untouched by main, so this row's edit is reapplied verbatim.
• include/vllm/model_executor/models/qwen3_5_weights.h — both sides ADD members
  to Nvfp4Weight (keep_dequant_b/d_dequant_b here, ResidentSlot resident_marlin{,_pair} from CUDA BF16 MoE resident cache reuses stale pointers after engine teardown #237/fix(moe): the weights own their resident device state, not a static address map (#237) #276); union, neither edited.

A second merge then picked up a4f72f86 (waiver registry deleted) and
f323907e (KV-MOONCAKE-STORE spec). That one overlapped .agents/roadmap_v1.md's
issue table — by KEY, not by line: main added #287, this row adds #213.
Verified by key rather than accepted as an automatic combination.

After both merges the records differ from origin/main by exactly this row's
scoped edit: NOW.md 1 line, roadmap_v1.md 1, STATUS.md 1, BENCHMARKS.md
14, ENVIRONMENT.md 1, FEATURES.md 1, USAGE.md 28. The branch is level with
origin/main.

450a1b69 on main retired the per-class PR line budgets, so the earlier
"product 900/900" claim is moot and has been dropped.

Gates

• Clean-configured CPU Release build (-Werror): 0 warnings, 0 errors.
• Full ctest -j 1: 369/369, 0 failed. Round 2's single failure
  (test_serve_low_tools) was pre-existing on main from The online-serving gate aborts on current main: harness looks for examples/server, the build produces examples/vllm-server #222 and is fixed by the
  merge.
• Focused: test_qwen27_dense_lmhead_fp4 7/7, 1314/1314 (was 6/6, 1170/1170),
  test_qwen27_paged_forward 20/20, 702/702, test_mtp_speculator
  13/13, 171/171.
• scripts/agent-preflight.sh: all gates green (pre-edit, --staged, and
  post-commit).
• NO CUDA GATE WAS RUN in round 3, and none is owed by it — the round
  touches no CUDA behavior. Round 3's own commit changes no file under src/;
  its only non-test, non-record edit is a header declaration. The operator's
  own run on cf9a3ba7 (test_qwen27_paged_engine 235/235 SACRED,
  test_qwen27_paged_forward 702/702, test_qwen27_dense_lmhead_fp4 1170/1170,
  test_qwen27_dense_forward 333/333, 0 skipped, FA2 marker present) stands.
• Still owed to the operator, unchanged from round 2: the binding throughput grid
  and the post-Model load / cold start time: measure it properly, then cut it #150 RSS re-measurement.

[localai-bot]

Operator verification — CUDA gate rerun independently

Per AGENTS.md ("The operator reruns the row's gate itself. An implementer or reviewer report is an input, never a gate result."), I rebuilt and regated this head myself rather than accepting the report.

Provenance. Head fa6ef3fe, transferred by git archive (never rsync), md5 8ed322ffa6baa4c44f52a48362ede4fc verified identical on both ends. Clean configure, sm_121a, Release, -DVLLM_CPP_CUTLASS_DIR=$HOME/cutlass-4.5.0 -DVLLM_CPP_TRITON=ON. Idle box, single flock $HOME/gpu.lock, ctest -j 1.

The build was verified non-degraded before gating — my script hard-aborts if the marker is missing, because a CUTLASS-less build silently omits FA2 and every result drifts:

-- CUTLASS found at /home/mudler/cutlass-4.5.0; enabling sm120a NVFP4 cutlass GEMM
-- FlashAttention-2 prefill/decode: ENABLED for arch(es) [121a]
build_exit=0

Result — 4/4 PASS:

1/4 Test  #32: test_qwen27_dense_forward ........   Passed    3.41 sec
2/4 Test  #75: test_qwen27_paged_forward ........   Passed   15.49 sec
3/4 Test #379: test_qwen27_paged_engine .........   Passed   48.49 sec   <- SACRED
4/4 Test #380: test_qwen27_dense_lmhead_fp4 .....   Passed    1.61 sec   <- the new gate
100% tests passed, 0 tests failed out of 4
focused_exit=0

So the SACRED 27B gate is intact and the new lmhead gate passes on a build I made from this exact head.

What this row is worth, measured

From the decode-only nsys two-length diff (8 vs 136 tokens, --cuda-graph-trace=node, both arms in the same tool, idle box, node-level tracing proven by integral launch counts) recorded on #213:

• The checkpoint ships lm_head.weight as U8/NVFP4 (248320x2560 + FP8 scales = 0.6660 GiB). vLLM keeps it packed and runs it as its 129th Marlin launch. We dequantized it to BF16 and read 2.3683 GiB — +1.702 GiB/step of avoidable traffic, 11.183 ms/step, 11.3% of our step, as a lone cutlass_80_tensorop_s16816gemm_bf16_128x64_32x6_nn_align2.
• Our Marlin is not slower than vLLM's. Combined Marlin+lm_head: ours 54.511 ms for 11.3330 GiB (207.9 GiB/s) vs vLLM 45.870 ms for 9.6309 GiB (210.0 GiB/s). Same kernel efficiency — we simply moved more bytes. This is a pure traffic win, and nobody should go looking for a Marlin tuning problem that does not exist.
• It is 8.6414 ms of the 17.3292 ms/step deficit, which closes to four decimal places as lm_head 8.6414 + FP8 tower 7.6068 + splitK 0.0532 + other 1.0279.

The FP8 half is row/PERF-27B-GDN-FP8-QKVZ (spec pushed); together they are ~16 of the 17.33 ms.

Still owed before merge

A fresh scoped re-review of this head — the round-1 findings were fixed, and AGENTS.md requires the loop to run to PASS, not to stop at the fix. Dispatching that now.

Not claimed

I did not re-run the throughput A/B or the greedy continuation on this head. The throughput figures in the body remain INDICATIVE (packed faster in all four legs, packed legs agreeing to 0.04%, but the dequant legs disagreeing by 7.9%). Note also that cross-engine greedy text is not expected to be byte-identical here: the oracle's own greedy is undetermined at ~8 of 32 positions on this corpus (top1-top2 margin exactly 0.000000; at one divergence its top five tokens were all exactly -2.651325). Judge any token diff against the oracle's margins, not byte equality.