From 9e4c7703c95d0086bbc0482310fb36aaf56d36d7 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 6 Aug 2026 11:45:38 +0000 Subject: [PATCH] =?UTF-8?q?record(serve):=20SERVE-FRONTEND-SLICE=20?= =?UTF-8?q?=E2=80=94=20c2-c8=20frontend=20residual=20MEASURED=20(perf),=20?= =?UTF-8?q?frontend=20~99%=20CPU-idle=20so=20the=20slice=20is=20handoff=20?= =?UTF-8?q?latency=20not=20compute;=20premise=20REFUTED,=20gap=20is=20GPU-?= =?UTF-8?q?intrinsic,=20NO=20code?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Executed the frontend attribution the MXFP4 campaign owed. Method step 1 (instrument OUR frontend per-step at c8) ran as a perf profile of the PRODUCTION server binary under sustained c8 streaming, NO rebuild: the built binary (dgx ~/mxfp4-bench/build sha 735f3b8d) has a git diff-IDENTICAL frontend (output_processor/detokenizer/serving_completion/api_server/async_llm) to main efe94402, so its profile IS main's. Workload matched #52 (/v1/completions streaming, 1024-in/128-out, greedy, 8 workers); sustained ITL mean 40.8ms ≈ the #52 c8 TPOT 37.56ms. Box left clean (server gone, both locks free, 78MB perf .data pruned, disk 35G). DECISIVE: the frontend cores drew only 186 on-CPU samples over 22s at 999Hz = ~0.8% of wall-time busy = ~99% IDLE (blocked on poll/futex), while the engine core drew 22K samples (75% libcuda + 21% vdso clock-poll). So the #52 ~0.26ms/step residual (91% of the +2.98ms c8 TPOT gap is GPU SPAN) is thread-handoff/scheduling LATENCY + syscalls, NOT frontend COMPUTE. The one structural divergence, our SlowIncrementalDetokenizer (O(seq) copy) vs vLLM's default FastIncrementalDetokenizer (Rust DecodeStream O(1)), is quantitatively negligible at 128-tok output (caps ~135 strings, ~2us/step); the largest frontend-compute term is nlohmann SSE-frame serialization (~34% of the SSE worker path), but the worker is idle so it does not move TPOT. VERDICT: the mission premise (a ~0.7ms DOMINANT frontend term to close) is REFUTED — the slice is ~0.26ms of idle-handoff latency with NO dominant closable compute mechanism. NO code shipped: mirroring Fast detok (needs HF Rust DecodeStream) or a hand-rolled serializer are low-ROI general-serving levers, not parity levers. The c2-c8 gap is GPU-intrinsic (flash refuted; #52 fair graphed marlin+flash+glue = 91%; #57 recovered marlin). MXFP4 PARITY VERDICT stands UNCHANGED: c1 1.020 PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, mem 2.63x; residual NOT frontend-closable. Records: benchmark-record (perf table + call-graph + named mechanisms file:line both sides), anchored state entry, STATUS/BENCHMARKS/NOW MXFP4 rows updated (STATUS paid by compacting the superseded MARLIN-STRUCT narrative, 284050<=284081). Evidence dgx:~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 4 +- .agents/benchmark-record.md | 84 +++++++++++++++++++++++++++++++++++++ .agents/state.md | 56 +++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 2 +- 5 files changed, 144 insertions(+), 4 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 1e1707ff..f8d1da03 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`. | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF | | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | -| MXFP4 parity | c1 1.020 PASS, mem 2.63x; c2-c8 0.96-0.97 (#57). Kernel side CLOSED (flash refuted) | Last: ~0.7ms frontend slice + num_splits cap + glue | +| MXFP4 parity | c1 1.020 PASS, mem 2.63x; c2-c8 0.96-0.97 (#57). Kernel side CLOSED (flash refuted) | Frontend slice MEASURED 0.26ms/step, ~99% CPU-idle (latency, not compute); NOT a parity lever; c2-c8 gap GPU-intrinsic | | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner | -| MXFP4 parity goal | **`VT_MARLIN_DENSE` FLIPPED DEFAULT-ON** (`row/KERNEL-MARLIN-DENSE-EXEC` `200b4b56`): dense E=1 marlin, 48-CTA vs 128. Ratified near-tie (32B razor gap 0.000, TIGHTER than MoE; goldens regen). Binding beats #51 every axis (c8 0.969, GPU mem 2.63x). Goal still <1.0x c2-c8 (state) | Merge row; next = decode-flash | +| MXFP4 parity goal | `VT_MARLIN_DENSE` default-ON (`KERNEL-MARLIN-DENSE-EXEC` `200b4b56`, merged): dense 48-CTA marlin beats MoE every axis (c8 0.969, mem 2.63x). c2-c8 still <1.0x; frontend REFUTED as the lever (`SERVE-FRONTEND-SLICE`), residual is GPU-intrinsic | Attribution DONE | In-flight branches (default-OFF, not pushed): `laguna-fp4proj-prod` (fp4), laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. Records: diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index fe4cd088..2f850a7e 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -12781,3 +12781,87 @@ remaining c8 term is the ~0.7ms/step host/sched slice (engine loop proven clean in #51 — the slice lives in the shared async frontend). Parity verdict unchanged: c1 1.020 / c2-c8 0.962-0.969. Evidence dgx:/tmp/fa2dump.err, /tmp/fa2ncu2.out, /tmp/fa2ab_n{1,3}.out. + +## SERVE-FRONTEND-SLICE: the c2-c8 frontend residual MEASURED (perf, c8 online decode) — the frontend is ~99% CPU-IDLE, so the ~0.26ms/step slice is handoff/scheduling LATENCY + syscalls, NOT frontend compute; detok Slow-vs-Fast is quantitatively negligible at 128-tok output; the mission's "~0.7ms dominant frontend term" premise is REFUTED, the c2-c8 gap is GPU-intrinsic — 2026-08-09 (branch `row/SERVE-FRONTEND-SLICE`, base `origin/main` `efe94402`) + +The campaign that owed the frontend attribution. Method step 1 (instrument OUR +frontend per-step at c8) executed as a `perf` profile of the PRODUCTION server +binary under a sustained c8 streaming load, NO rebuild — the built binary +(dgx `~/mxfp4-bench/build`, sha `735f3b8d`) has a `git diff`-IDENTICAL frontend +(output_processor / detokenizer / serving_completion / api_server / async_llm) +to main `efe94402`, so its frontend profile IS main's. Box left clean (server +gone, both locks free, 78 MB perf .data removed, disk 35 G). + +WORKLOAD (matches the #52 binding): `/v1/completions` streaming, 1024-in/128-out, +greedy, 8 concurrent workers cycling the c8 corpus, 60 s. Sustained ITL mean +40.8 ms / median 31.4 ms ≈ #52's c8 TPOT 37.56 ms — the load is the real c8 +decode regime. `sudo perf record -F 999 -p PID` (flat, 22 s) + `-F 499 +--call-graph dwarf` (18 s). + +THE DECISIVE NUMBER (flat profile): the GB10 splits work across two PMU core +clusters. The ENGINE core (counter armv8_pmuv3_1) drew 22 K on-CPU samples = +86.5 B cycles, 75.4 % libcuda + 21.4 % vdso (clock_gettime = CUDA busy-poll) — +pure GPU orchestration. The FRONTEND cores (counter armv8_pmuv3_0) drew only +**186 on-CPU samples over 22 s at 999 Hz** → the frontend was on-CPU ~0.8 % of +wall-time = **~99 % IDLE (blocked on __poll / futir waits)**. So the ~0.26 ms/step +residual (#52: TPOT ours 37.56 − vLLM 34.58 = +2.98 ms, GPU SPAN accounts +2.72 ms += 91 %, residual +0.26 ms) is thread-handoff/scheduling LATENCY + syscalls, NOT +frontend COMPUTE — there is no heavy frontend work to remove. + +FRONTEND on-CPU ATTRIBUTION (of counter_0's 186 samples; noisy but directional): +| bucket | share | detail | +|---|---:|---| +| kernel/syscall | 56.6% | `__poll` (httplib), futex (condvar handoffs), socket sendmsg (SSE write), + `nf_conntrack`/`nf_nat` ~5.6% = loopback conntrack, a box iptables artifact absent for real remote clients | +| libc | 23.6% | malloc/free churn ~9.6% (per-token RequestOutput + json tree + SSE string), memcpy/strlen ~4.2% | +| server (our code) | 13.25% | dominated by per-token SSE-frame JSON build+serialize | +| libstdc++ | 0.93% | std::string / map ops | + +CALL-GRAPH (dwarf) of the httplib content-provider worker: inside +`CompletionSseStream::next` (serving_completion.cpp:39-114) **~34 % goes to +`to_json(...CompletionStreamResponse)`** — and OF THAT ~26 % is nlohmann +`basic_json(initializer_list)` CONSTRUCTION + `~basic_json` teardown (nlohmann's +`std::map`-backed object allocates a node tree per token), only ~8 % is the actual +string dump. Then `RequestOutput::~RequestOutput` + `__libc_free` (freeing the +per-token full `prompt_token_ids` copy, output_processor.cpp:253-254). Detok is +INVISIBLE: the only tokenizer symbols (`vt::tok::BpeMerge`/`MergeKey` ~1.85 %) are +per-REQUEST INPUT tokenization (which vLLM also does), NOT per-token decode; the +per-token `DetokenizeIncrementally`/`ConvertTokensToString` are below the 0.15 % +floor. + +NAMED MECHANISMS (ours file:line vs vLLM file:line): +1. **Detokenizer default.** Ours: `IncrementalDetokenizer::FromNewRequest` + (`src/vllm/v1/engine/detokenizer.cpp:402-412`) ALWAYS returns + `SlowIncrementalDetokenizer`; `DetokenizeIncrementally` + (`detokenizer.cpp:347`) deep-copies the whole growing `tokens_` vector per + token (O(num_generated), but `tokens_` is seeded with only a ~7-token prompt + window in the ctor `detokenizer.cpp:515-530`, so at 128-tok output it caps + ~135 strings ≈ 2 µs/step — negligible). vLLM: `from_new_request` + (`vllm/v1/engine/detokenizer.py:59-75`) defaults to `FastIncrementalDetokenizer` + (`detokenizer.py:167-236`) = Rust `tokenizers.decoders.DecodeStream.step`, O(1) + per token. Our Slow path MIRRORS vLLM's fallback but not its DEFAULT; measured + impact at this workload is nil. It grows only for long generations. +2. **SSE-frame serialization.** Ours: `nlohmann::json(frame).dump()` + (`serving_completion.cpp:102`) — `std::map`-backed init-list construction, the + largest MEASURED frontend-compute term (~34 % of the SSE worker path). vLLM: + `chunk.model_dump_json(exclude_unset=True)` + (`vllm/entrypoints/openai/completion/serving.py:441`) = pydantic-core (Rust). +3. **Architecture.** Ours: EngineCore in-process thread → output_handler thread → + collector condvar → 8 httplib worker threads (async_llm.cpp:260-311). NO IPC. + vLLM: EngineCore SUBPROCESS → ZMQ msgpack IPC → asyncio single-loop output + handler → completion_stream_generator coroutine. We SAVE the ZMQ IPC vLLM pays; + we PAY multi-thread wakeup latency. The ~0.26 ms residual is the NET. + +VERDICT: the frontend slice is a diffuse ~0.26 ms/step (≈9 % of the +2.98 ms c8 +TPOT gap), and it is scheduling/handoff LATENCY, not compute — there is NO +dominant closable frontend-compute mechanism, and the one structural divergence +(Slow vs Fast detok) is quantitatively irrelevant at 128-tok output. NO code +shipped: mirroring vLLM's Fast detok (needs HF Rust DecodeStream) or replacing +nlohmann with a hand-rolled serializer are LOW-ROI general-serving efficiency +levers (help long-output / high-QPS), NOT MXFP4 parity levers — the frontend is +idle, so neither moves TPOT. The c2-c8 gap is GPU-INTRINSIC (KERNEL-FA2-DECODE-PARAMS +refuted flash as intrinsic-identical; #52 fair graphed-vs-graphed = marlin + +flash + glue = 91 % of the gap; #57 recovered marlin byte-safe). MXFP4 PARITY +VERDICT stands UNCHANGED: c1 1.020 PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, mem +2.63x — and the residual is NOT frontend-closable. Evidence +dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}` (perf .data +pruned for disk). diff --git a/.agents/state.md b/.agents/state.md index 313ffe0c..30713618 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -37517,3 +37517,59 @@ over-splits at c1-c2 where we already pass). Full detail + evidence paths in the benchmark record. MXFP4 stands: c1 1.020 / c2-c8 0.962-0.969 / mem 2.63x. Remaining terms: the ~0.7ms/step frontend slice (dominant, next campaign), the c1-c2 num_splits cap (oracle-gated), the glue tail. + +## SERVE-FRONTEND-SLICE: the c2-c8 frontend residual MEASURED — the frontend is ~99% CPU-IDLE, so the slice is handoff/scheduling LATENCY not compute; premise "~0.7ms dominant frontend term" REFUTED; the c2-c8 gap is GPU-intrinsic; NO code shipped + + +Executed the frontend attribution the campaign owed, on `row/SERVE-FRONTEND-SLICE` +off `origin/main` `efe94402`. Method step 1 (instrument OUR frontend per-step at +c8) ran as a `perf` profile of the PRODUCTION server binary under sustained c8 +streaming — NO rebuild: the built binary (dgx `~/mxfp4-bench/build`, sha +`735f3b8d`) has a `git diff`-IDENTICAL frontend (output_processor / detokenizer / +serving_completion / api_server / async_llm) to main, so its profile IS main's. +Workload matches #52 (`/v1/completions` streaming, 1024-in/128-out, greedy, 8 +workers, 60 s); sustained ITL mean 40.8 ms ≈ #52 c8 TPOT 37.56 ms. Box left clean +(server gone, both locks free, 78 MB perf .data pruned, disk 35 G). + +DECISIVE MEASUREMENT: the GB10's two PMU clusters split the work. The ENGINE core +drew 22 K on-CPU samples (86.5 B cycles, 75 % libcuda + 21 % vdso clock-poll) = GPU +orchestration. The FRONTEND cores drew only **186 on-CPU samples over 22 s at +999 Hz → ~0.8 % of wall-time busy = ~99 % IDLE** (blocked on `__poll` / futex). So +the #52 ~0.26 ms/step residual (TPOT ours 37.56 − vLLM 34.58 = +2.98 ms, GPU SPAN ++2.72 ms = 91 %, residual +0.26 ms) is thread-handoff/scheduling LATENCY + +syscalls, NOT frontend COMPUTE. Of the tiny frontend compute that exists: kernel/ +syscall 56.6 % (poll/futex/sendmsg + loopback conntrack, a box artifact), libc +23.6 % (malloc/free churn + memcpy), our code 13.25 % (per-token SSE-frame JSON +build), libstdc++ 0.93 %. Call-graph: inside `CompletionSseStream::next` ~34 % is +`to_json(CompletionStreamResponse)` (nlohmann `std::map` init-list construct+teardown +~26 %, dump only ~8 %) + `RequestOutput::~RequestOutput` free. DETOK is INVISIBLE +(only tokenizer symbols are per-REQUEST input `BpeMerge` ~1.85 %, which vLLM also +pays; per-token decode below the 0.15 % floor). + +NAMED MECHANISMS (ours file:line vs vLLM): (1) detok default — ours +`SlowIncrementalDetokenizer` always (`detokenizer.cpp:402-412`), O(seq) copy at +`:347` but `tokens_` seeded with a ~7-token window (`:515-530`) so caps ~135 +strings ≈ 2 µs/step at 128-tok output = negligible; vLLM defaults to +`FastIncrementalDetokenizer` = Rust `DecodeStream.step` O(1) +(`detokenizer.py:59-75,167-236`). (2) SSE serialize — ours nlohmann +`json(frame).dump()` (`serving_completion.cpp:102`, std::map-backed) vs vLLM +pydantic-core `model_dump_json` (`completion/serving.py:441`). (3) architecture — +ours in-process threads + condvar handoffs (NO IPC) vs vLLM EngineCore subprocess ++ ZMQ; we save the IPC vLLM pays, pay thread-wakeup latency; the 0.26 ms is the NET. + +VERDICT: the mission's premise (a ~0.7 ms DOMINANT frontend term to close) is +REFUTED — the slice is ~0.26 ms (#52 same-tool superseded the old cross-tool +0.7 ms), it is idle-handoff LATENCY not compute, and there is NO dominant closable +frontend-compute mechanism; the one structural divergence (Slow vs Fast detok) is +irrelevant at 128-tok output. NO code shipped: mirroring Fast detok (needs HF Rust +DecodeStream) or a hand-rolled serializer are LOW-ROI general-serving levers (help +long-output / high-QPS), not parity levers — the frontend is idle so neither moves +TPOT. The c2-c8 gap is GPU-INTRINSIC (flash refuted; #52 fair graphed marlin+flash+ +glue = 91 %; #57 recovered marlin). MXFP4 PARITY VERDICT stands UNCHANGED: c1 1.020 +PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, mem 2.63x; the residual is NOT +frontend-closable. RESIDUAL MAP (honest ROI): decode-flash num_splits cap +(oracle-gated, c1-c2 where we already pass) + norm/quant/act glue fusion (portable, +modest) are the only remaining GPU-side terms; both are the same intrinsic/glue +class already recorded, neither crosses c2-c8 to 1.0x alone. Full perf table + +call-graph in the benchmark record. Evidence +dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}`. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 98d62f5f..c2118931 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -309,7 +309,7 @@ built on it rather than keeping the flattering one. | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | -| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE FLIPPED DEFAULT-ON**: dense marlin 48-CTA (vs MoE 128), nsys c8 117.8->85.8us/call; ratified 32B near-tie (razor gap 0.000, goldens regen). Goal <1.0x c2-c8, next decode-flash | +| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE default-ON**: dense 48-CTA marlin; 32B near-tie ratified. SERVE-FRONTEND-SLICE: c2-c8 slice 0.26ms/step, frontend ~99% CPU-idle (latency not compute), NOT a parity lever; gap GPU-intrinsic | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx | | Pre-Ampere breadth (Turing `sm_75` / Volta `sm_70` / Pascal) | **No number owed; nothing runs on these arches.** 2026-08-06 `sm_75`: 20/20 TUs PASS (0 err/warn), WMMA bodies + all 3 selectors arch-gated; GB10 SASS byte-identical. [Detail](../.agents/benchmark-record.md) | Port the llama.cpp `fattn-tile`/`fattn-vec` fp16 body. Perf floor when a card exists is **llama.cpp on the same card** (vLLM does not run there) | diff --git a/docs/STATUS.md b/docs/STATUS.md index de6141a2..2cc760ee 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -84,7 +84,7 @@ token-for-token correctness against the pinned oracle. | Safetensors loading | Supported | Both gate models plus every registered dense/MoE family | | GGUF loading (F32/F16/BF16/Q4_0/Q8_0/Q2_K/Q3_K/Q4_K/Q5_K/Q6_K/IQ2_XXS/IQ3_XXS/IQ2_S/MXFP4/NVFP4) | Supported; compute-in-quant (keep-quant) on CPU AND now CUDA for the six K-block encodings PLUS Q2_K/IQ2_XXS/IQ3_XXS (DeepSeek-V4 W8, 2026-07-29 - the FIRST CUDA keep-quant GGUF k-quant GEMM `KERNEL-QUANT-CIQ-GEMM-CUDA`, MMVQ-style dequant-in-kernel, GB10-gated 92401/92401 vs the CPU oracle, so a CUDA runner keeps blocks compressed and dots them on the GPU instead of the ARM cores); **NVFP4 now COMPUTES IN FP4 on CUDA for the dense-MLP and full-attention projections (2026-07-29, `CLAIM-GGUF-NVFP4-COMPUTE`), no longer materialize-only** | Weights in six block encodings stay compressed from file to matmul on CPU (no BF16 expansion). NVFP4 (ggml type 40) DEQUANTIZES, including the per-tensor (per-expert) `.scale` sidecar the container keeps outside the blocks; gated BIT-EXACT against the compressed-tensors NVFP4 path on real Qwen3.6-27B bytes from both containers. **It no longer expands to bf16 on CUDA:** an NVFP4 matmul/expert weight is REPACKED at load into the same (`weight_packed [N,K/2]`, `weight_scale [N,K/16]`) operand pair the compressed-tensors path produces - a pure byte permutation, gated BYTE-IDENTICAL against that container - and the existing `vt::MatmulNvfp4*` kernels run on it, so no new kernel exists and no numerics are re-derived. Covers the dense MLP + full-attention q/k/v/o and the MoE shared/routed experts; the GDN `in_proj_*` family and `ssm_out` still expand (the V-head reorder rewrites their layout) and a CPU build still expands everything - the documented `part` subset. **MEASURED on GB10 (2026-07-29), same-binary A/B, one `flock`, idle box, 2 reps per arm:** peak RSS **50.8 -> 25.7 GiB**, load-and-generate **1:58 -> 0:41**, and the 256 projections that move cost 35 840 MiB expanded against 10 080 MiB fp4-resident (3.56x). **The divergence against the safetensors sibling CLOSES:** the fp4 arm is token-IDENTICAL over 24 greedy tokens where the bf16 arm of the same binary diverges at index 4, which retires the reading that that divergence was permanent. It is REPORTED, not gated: the two containers are not the same model - the GGUF NVFP4-quantizes 192 GDN `in_proj` tensors the safetensors keeps BF16 (mean relative weight error ~0.18) and their activation global scales differ - so identity is not guaranteed and a cross-container throughput arm is not valid. SACRED gates unmoved: `test_qwen27_paged_engine` 235/235, `test_qwen36_paged_engine` 315/315. **The MoE (35B) stacked-expert arm is now HARDWARE-GATED too (2026-07-29)**, superseding the gap recorded here: the real 35B A3B NVFP4 GGUF loads and generates through the fp4 path, its 120 routed-expert stacks x 256 experts repack to the modelopt safetensors' own operands with ZERO differing bytes over 840 sampled (tensor, expert) slabs, and all 840 per-expert `.scale[e]` are bit-identical to that expert's `weight_scale_2` - the per-expert scale INDEXING, mutation-proved against both a `scales[0]`-for-every-expert and an expert-0-slab-for-every-expert mutant. Same-binary A/B: peak RSS 68.5 -> 22.7 GiB (3.01x), load-and-generate 1:51.9 -> 0:28.8, tokens IDENTICAL (correct here, since the 35B routed experts run the W4A16 grouped GEMM in both arms). Recorded as OPEN, not smoothed over: this case's 24-token greedy stream is NOT run-to-run stable (one of three `use_a16` runs and one of four safetensors-reference runs differed), so the binding results are the weight-level byte identity and the residency audit, not a token-exactness claim; `test_qwen36_paged_engine` is token-exact at ITS engine params, so the instability belongs to this case's configuration and attributing it is owed work. That run also found and FIXED a latent defect the MoE arm made reachable: the two fp4 fused MoE blocks issued the router GEMM assuming the safetensors `[K,N]` gate layout and threw `matmul: inner dims mismatch` on the GGUF's `[N,K]` one; `MoeRouterLogits` now branches on `nk` (inert for the safetensors path, SACRED gates unmoved). **Q2_K (id 10) + IQ2_XXS (id 16) DEQUANTIZE (2026-07-29, `CLAIM-DSV4-GGUF-LOADER`):** the ~2-bit types the single-Spark `DeepSeek-V4-Flash-GGUF UD-IQ2_XXS`/`UD-Q2_K_XL` vehicles use, ported 1:1 from llama.cpp `ggml-quants.c` (`iq2xxs_grid` codebook + signs; Q2_K nibble sub-scale/min), unit-gated on hand-derived bytes (`test_gguf_dequant` 15/15). Dequant-only (no vec_dot -> expand-bf16). A V4-GGUF model still cannot RUN: the V4-GGUF name map (tensor-manifest-blocked) + the V4 forward (W3-W8) remain. **Multi-shard split GGUF READING landed (2026-08-03, `CLAIM-GGUF-SPLIT-SHARDS`):** `GgufFile::Open` now transparently stitches llama.cpp `gguf-split` shards (`...-00001-of-00003.gguf`) — every shard mmap'd, tensor tables merged, KV metadata taken from shard `00001`, and the sibling shard mappings kept alive by the primary so keep-quant mmap-borrows stay valid across shards (`OwnsSpan` is shard-aware); `VT_GGUF_NO_SPLIT=1` opts out; unit-gated (`test_gguf` split-merge / no-split / count-mismatch cases, 33/33 local). This unblocks the real 3-shard `unsloth/DeepSeek-V4-Flash-0731 UD-IQ2_M` (~91 GiB), whose layout is the NATIVE `deepseek4` arch — per-block `ffn_gate_tid2eid` hash tables (hash layers 0/1/2) + `hc_*` MHC + DSA compressor/indexer are all PRESENT (name-map 1328/1328), `vocab_size` derives from `token_embd` — NOT a standard llama.cpp conversion, so no loader-layout change is owed. It now loads THROUGH 1324/1328 tensors; the sole remaining gap is 4 routed-expert slabs quantized with IQ2_S (id 22, ×2) + MXFP4 (id 39, ×2) — encodings we have GGUF block traits for but no keep-quant vec_dot, so they hit the expand→dequant path which lacks them. Dequant-expanding those 4 big expert tensors to bf16 would add ~17 GiB (~106 GiB total → GB10 OOM-reboot risk), so the memory-safe fix is an IQ2_S+MXFP4 keep-quant kernel (CPU dequant dispatch + the `iq2s_grid` codebook + a CUDA `DotSuperblock`), spec'd as the next brick **IQ2_S (id 22) + MXFP4 (id 39) DEQUANTIZE + KEEP-QUANT on CPU (2026-08-03, `CLAIM-DSV4-UDIQ2M-QUANT`, off-GPU):** the extra per-tensor "dynamic" encodings the `unsloth/DeepSeek-V4-Flash-GGUF UD-IQ2_M` checkpoint mixes into its last routed-expert slabs (IQ2_S `ffn_gate/up` dotting Q8_K, MXFP4 `ffn_down` dotting Q8_0) — ported 1:1 from llama.cpp `ggml-quants.c` @ 237ad9b96 (`iq2s_grid` 1024-entry codebook + DIRECT sign bytes; MXFP4 `kvalues_mxfp4` + `e8m0_to_fp32_half` micro-scaling, distinct from the compressed-tensors `E8M0ToF32` NVFP4 path). CPU dequant + keep-quant `vec_dot`, unit-gated on hand-derived golden bytes (`test_gguf_dequant` 17/17), an INDEPENDENT f64 dequant-then-dot + GEMM NMSE (`test_ops_quant_dot` 19/19), and keep-quant routing (`test_gguf_keep_quant` 37/37) — all CPU-green, so UD-IQ2_M's four previously-`unsupported ggml type 22/39` slabs now load COMPRESSED (no ~17 GiB bf16 expansion that OOM-reboots the box). CUDA: the IQ2_S device `DotSuperblock` is wired into the Q8_K grouped-MoE GEMM and now **CUDA-BUILT + LINKED on GB10 (sm_121a, CUDA 13.0, `-Werror`, 2026-08-03 integration)** — it compiles clean and the merged binary links; MXFP4's device dot (`DotMXFP4`) is written but NOT wired (Q8_0-activation needs a separate 32-block GEMM) so it is marked `[[maybe_unused]]` to keep the ready math without tripping nvcc #177-D, and on GPU MXFP4 CPU-fallbacks like Q4_0/Q8_0. The V4-GGUF forward + a real UD-IQ2_M GPU load/coherence run are owed | | AWQ / GPTQ quantization | W0 spike + W1 CPU INT4 dequant primitive; not yet loadable end to end | INT4 unpack+dequant-to-bf16 for BOTH community formats, mirroring vLLM 1:1 (AWQ reverse-order `awq_triton.py`; GPTQ `qdq_4.cuh` with zero_offset v1/v2 + act-order g_idx). Unit-gated RED-first (hand-computed known bytes + double-precision roundtrip). NOT wired to a loader, no GPU Marlin compute, no model run yet: config recognizer (W2), Marlin GPU GEMM riding the vendored NVFP4 Marlin (W4), CPU e2e (W3), GPTQ 8/2/3-bit (W5) and MoE (W6) are named next bricks. See [.agents/specs/awq-gptq-quant.md](../.agents/specs/awq-gptq-quant.md) | -| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49). **`MARLIN-STRUCT`: decode-graph (`VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH`) + gate_up FUSION (NVFP4 to MXFP4) DEFAULT-ON.** Decode marlin 180->144 GEMM/step (vLLM-structural), byte-token-exact (0.6B/4B 184/184 graph ON==OFF, async 82/82, #44 3/3, 32B-NVFP4A16 142/142). nsys c8: marlin CTA 144 vs vLLM 48 dominates (+1,177us). **`KERNEL-MARLIN-DENSE-EXEC`: `VT_MARLIN_DENSE` FLIPPED DEFAULT-ON** — dense marlin 48-CTA (vs MoE 128, nsys 117.8->85.8us/call), byte-faithful (ratified 32B near-tie gap 0.000; unit 263/263, 6/6 graphed+eager, goldens regen); binding beats #51 every axis (c8 0.969, mem 2.63x). Goal <1.0x c2-c8. state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | +| MXFP4 (compressed-tensors `mxfp4-pack-quantized`) | Compute PROVEN (#38); GQA-swap ON (#49). decode-graph + gate_up FUSION default-ON (`MARLIN-STRUCT`, byte-token-exact). **`KERNEL-MARLIN-DENSE-EXEC`: `VT_MARLIN_DENSE` FLIPPED DEFAULT-ON** — dense marlin 48-CTA (vs MoE 128, nsys 117.8->85.8us/call), byte-faithful (ratified 32B near-tie gap 0.000; unit 263/263, 6/6 graphed+eager, goldens regen); binding beats #51 every axis (c1 1.020, c8 0.969, mem 2.63x). **`SERVE-FRONTEND-SLICE`: the c2-c8 frontend residual MEASURED (perf, c8) = 0.26ms/step but the frontend is ~99% CPU-IDLE — handoff LATENCY + syscalls, NOT compute; detok/JSON negligible; NOT a parity lever. The c2-c8 gap is GPU-intrinsic.** state.md | Shared with DeepSeek-V4-Flash + Kimi-K3 MXFP4 paths. CPU E8M0 dequant 5/5·1142. GPU W4A4 + MoE-expert e2e later | | CPU backend vs llama.cpp | At or ahead on every axis (GGUF) | Prefill 1.18x ahead, decode at parity, peak memory 1.01x, byte-identical greedy tokens. Single-stream only; no concurrent-serving comparison has been measured | | Paged KV cache + prefix caching | Supported | Block-paged full attention, hybrid full-attention + GDN state groups, automatic prefix caching (APC) on by default for dense models (cache-ON gated end to end: token-identical output, cache hits, faster TTFT) | | fp8 KV cache (`cache_dtype=fp8`) | In progress (W1 CPU brick), not yet usable end-to-end | HIGH-priority memory/throughput lever (halves the KV footprint). W0 spike + W1 CPU brick landed (`KV-FP8` ACTIVE): fp8-e4m3 K/V STORE (`Quantize(hp/scale)`) + the paged-attention READ dequant (`Dequant(fp8)*scale`) + the `cache_dtype` config parse, all CPU-gated RED-first (`test_ops_fp8_kv_cache` 8/8·511; a wrong store direction fails 3/480). Storage is 1-byte fp8 (`DType::kI8`) + a `Fp8KVCacheDataType` interpretation enum, per-tensor k/v scales (mirroring vLLM `BaseKVCacheMethod`). The CUDA store + fp8 paged-attention read (the GPU memory-halving path, DGX-blocked), the runner/spec integration (half-sized KV blocks + checkpoint-scale threading + `--kv-cache-dtype`/`--calculate-kv-scales`), fp8_e5m2 and per-head scales are named W2-W5 in [.agents/specs/fp8-kv-cache.md](../.agents/specs/fp8-kv-cache.md). No model can run with an fp8 KV cache yet |