mudler · localai-bot · Aug 6, 2026
diff --git a/.agents/NOW.md b/.agents/NOW.md
@@ -22,9 +22,9 @@ checkpoint on `upstream/main` at `59674cf1d`.
 | Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **Full-model GB10 e2e RUNS** (bf16-resident §13): CPU+CUDA 13/13·656, no OOM. **Token gate NEAR-TIE 106/128** (6/8 token-exact) | device GDN/MLA islands + bf16 stream; 1.59 tok/s; default OFF |
 | 35B fresh grid | **BOUND** @`1ea26427`: tput 0.93-1.03x, c16 0.93x. INTAKE + Option A both **RESOLVED NEGATIVE** (H2D-out-of-capture tput WASH) | Real lever left: prefill glue (task #61) |
 | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` |
-| MXFP4 parity | c1 1.020 PASS, mem 2.63x; c2-c8 0.96-0.97 (#57). Kernel side CLOSED (flash refuted) | Last: ~0.7ms frontend slice + num_splits cap + glue |
+| MXFP4 parity | c1 1.020 PASS, mem 2.63x; c2-c8 0.96-0.97 (#57). Kernel side CLOSED (flash refuted) | Frontend slice MEASURED 0.26ms/step, ~99% CPU-idle (latency, not compute); NOT a parity lever; c2-c8 gap GPU-intrinsic |
 | ROW-SERVE-ASYNC-DENSE-MIRROR | **LANDED+dgx-VERIFIED** (`f9c969ae`): #31 async mirror on classic dense Qwen3; gate RED→GREEN, SACRED 184/184 | Residual: sibling scope one-liner |
-| MXFP4 parity goal | **`VT_MARLIN_DENSE` FLIPPED DEFAULT-ON** (`row/KERNEL-MARLIN-DENSE-EXEC` `200b4b56`): dense E=1 marlin, 48-CTA vs 128. Ratified near-tie (32B razor gap 0.000, TIGHTER than MoE; goldens regen). Binding beats #51 every axis (c8 0.969, GPU mem 2.63x). Goal still <1.0x c2-c8 (state) | Merge row; next = decode-flash |
+| MXFP4 parity goal | `VT_MARLIN_DENSE` default-ON (`KERNEL-MARLIN-DENSE-EXEC` `200b4b56`, merged): dense 48-CTA marlin beats MoE every axis (c8 0.969, mem 2.63x). c2-c8 still <1.0x; frontend REFUTED as the lever (`SERVE-FRONTEND-SLICE`), residual is GPU-intrinsic | Attribution DONE |
 
 In-flight branches (default-OFF, not pushed): `laguna-fp4proj-prod` (fp4),
 laguna bf16/legacy/pipeline-gemv, `ds4-hc-expand-fuse`. Records:

diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md
@@ -12781,3 +12781,87 @@ remaining c8 term is the ~0.7ms/step host/sched slice (engine loop proven
 clean in #51 — the slice lives in the shared async frontend). Parity verdict
 unchanged: c1 1.020 / c2-c8 0.962-0.969. Evidence dgx:/tmp/fa2dump.err,
 /tmp/fa2ncu2.out, /tmp/fa2ab_n{1,3}.out.
+
+## SERVE-FRONTEND-SLICE: the c2-c8 frontend residual MEASURED (perf, c8 online decode) — the frontend is ~99% CPU-IDLE, so the ~0.26ms/step slice is handoff/scheduling LATENCY + syscalls, NOT frontend compute; detok Slow-vs-Fast is quantitatively negligible at 128-tok output; the mission's "~0.7ms dominant frontend term" premise is REFUTED, the c2-c8 gap is GPU-intrinsic — 2026-08-09 (branch `row/SERVE-FRONTEND-SLICE`, base `origin/main` `efe94402`)
+
+The campaign that owed the frontend attribution. Method step 1 (instrument OUR
+frontend per-step at c8) executed as a `perf` profile of the PRODUCTION server
+binary under a sustained c8 streaming load, NO rebuild — the built binary
+(dgx `~/mxfp4-bench/build`, sha `735f3b8d`) has a `git diff`-IDENTICAL frontend
+(output_processor / detokenizer / serving_completion / api_server / async_llm)
+to main `efe94402`, so its frontend profile IS main's. Box left clean (server
+gone, both locks free, 78 MB perf .data removed, disk 35 G).
+
+WORKLOAD (matches the #52 binding): `/v1/completions` streaming, 1024-in/128-out,
+greedy, 8 concurrent workers cycling the c8 corpus, 60 s. Sustained ITL mean
+40.8 ms / median 31.4 ms ≈ #52's c8 TPOT 37.56 ms — the load is the real c8
+decode regime. `sudo perf record -F 999 -p PID` (flat, 22 s) + `-F 499
+--call-graph dwarf` (18 s).
+
+THE DECISIVE NUMBER (flat profile): the GB10 splits work across two PMU core
+clusters. The ENGINE core (counter armv8_pmuv3_1) drew 22 K on-CPU samples =
+86.5 B cycles, 75.4 % libcuda + 21.4 % vdso (clock_gettime = CUDA busy-poll) —
+pure GPU orchestration. The FRONTEND cores (counter armv8_pmuv3_0) drew only
+**186 on-CPU samples over 22 s at 999 Hz** → the frontend was on-CPU ~0.8 % of
+wall-time = **~99 % IDLE (blocked on __poll / futir waits)**. So the ~0.26 ms/step
+residual (#52: TPOT ours 37.56 − vLLM 34.58 = +2.98 ms, GPU SPAN accounts +2.72 ms
+= 91 %, residual +0.26 ms) is thread-handoff/scheduling LATENCY + syscalls, NOT
+frontend COMPUTE — there is no heavy frontend work to remove.
+
+FRONTEND on-CPU ATTRIBUTION (of counter_0's 186 samples; noisy but directional):
+| bucket | share | detail |
+|---|---:|---|
+| kernel/syscall | 56.6% | `__poll` (httplib), futex (condvar handoffs), socket sendmsg (SSE write), + `nf_conntrack`/`nf_nat` ~5.6% = loopback conntrack, a box iptables artifact absent for real remote clients |
+| libc | 23.6% | malloc/free churn ~9.6% (per-token RequestOutput + json tree + SSE string), memcpy/strlen ~4.2% |
+| server (our code) | 13.25% | dominated by per-token SSE-frame JSON build+serialize |
+| libstdc++ | 0.93% | std::string / map ops |
+
+CALL-GRAPH (dwarf) of the httplib content-provider worker: inside
+`CompletionSseStream::next` (serving_completion.cpp:39-114) **~34 % goes to
+`to_json(...CompletionStreamResponse)`** — and OF THAT ~26 % is nlohmann
+`basic_json(initializer_list)` CONSTRUCTION + `~basic_json` teardown (nlohmann's
+`std::map`-backed object allocates a node tree per token), only ~8 % is the actual
+string dump. Then `RequestOutput::~RequestOutput` + `__libc_free` (freeing the
+per-token full `prompt_token_ids` copy, output_processor.cpp:253-254). Detok is
+INVISIBLE: the only tokenizer symbols (`vt::tok::BpeMerge`/`MergeKey` ~1.85 %) are
+per-REQUEST INPUT tokenization (which vLLM also does), NOT per-token decode; the
+per-token `DetokenizeIncrementally`/`ConvertTokensToString` are below the 0.15 %
+floor.
+
+NAMED MECHANISMS (ours file:line vs vLLM file:line):
+1. **Detokenizer default.** Ours: `IncrementalDetokenizer::FromNewRequest`
+   (`src/vllm/v1/engine/detokenizer.cpp:402-412`) ALWAYS returns
+   `SlowIncrementalDetokenizer`; `DetokenizeIncrementally`
+   (`detokenizer.cpp:347`) deep-copies the whole growing `tokens_` vector per
+   token (O(num_generated), but `tokens_` is seeded with only a ~7-token prompt
+   window in the ctor `detokenizer.cpp:515-530`, so at 128-tok output it caps
+   ~135 strings ≈ 2 µs/step — negligible). vLLM: `from_new_request`
+   (`vllm/v1/engine/detokenizer.py:59-75`) defaults to `FastIncrementalDetokenizer`
+   (`detokenizer.py:167-236`) = Rust `tokenizers.decoders.DecodeStream.step`, O(1)
+   per token. Our Slow path MIRRORS vLLM's fallback but not its DEFAULT; measured
+   impact at this workload is nil. It grows only for long generations.
+2. **SSE-frame serialization.** Ours: `nlohmann::json(frame).dump()`
+   (`serving_completion.cpp:102`) — `std::map`-backed init-list construction, the
+   largest MEASURED frontend-compute term (~34 % of the SSE worker path). vLLM:
+   `chunk.model_dump_json(exclude_unset=True)`
+   (`vllm/entrypoints/openai/completion/serving.py:441`) = pydantic-core (Rust).
+3. **Architecture.** Ours: EngineCore in-process thread → output_handler thread →
+   collector condvar → 8 httplib worker threads (async_llm.cpp:260-311). NO IPC.
+   vLLM: EngineCore SUBPROCESS → ZMQ msgpack IPC → asyncio single-loop output
+   handler → completion_stream_generator coroutine. We SAVE the ZMQ IPC vLLM pays;
+   we PAY multi-thread wakeup latency. The ~0.26 ms residual is the NET.
+
+VERDICT: the frontend slice is a diffuse ~0.26 ms/step (≈9 % of the +2.98 ms c8
+TPOT gap), and it is scheduling/handoff LATENCY, not compute — there is NO
+dominant closable frontend-compute mechanism, and the one structural divergence
+(Slow vs Fast detok) is quantitatively irrelevant at 128-tok output. NO code
+shipped: mirroring vLLM's Fast detok (needs HF Rust DecodeStream) or replacing
+nlohmann with a hand-rolled serializer are LOW-ROI general-serving efficiency
+levers (help long-output / high-QPS), NOT MXFP4 parity levers — the frontend is
+idle, so neither moves TPOT. The c2-c8 gap is GPU-INTRINSIC (KERNEL-FA2-DECODE-PARAMS
+refuted flash as intrinsic-identical; #52 fair graphed-vs-graphed = marlin +
+flash + glue = 91 % of the gap; #57 recovered marlin byte-safe). MXFP4 PARITY
+VERDICT stands UNCHANGED: c1 1.020 PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, mem
+2.63x — and the residual is NOT frontend-closable. Evidence
+dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}` (perf .data
+pruned for disk).
diff --git a/.agents/state.md b/.agents/state.md
@@ -37517,3 +37517,59 @@ over-splits at c1-c2 where we already pass). Full detail + evidence paths in
 the benchmark record. MXFP4 stands: c1 1.020 / c2-c8 0.962-0.969 / mem 2.63x.
 Remaining terms: the ~0.7ms/step frontend slice (dominant, next campaign),
 the c1-c2 num_splits cap (oracle-gated), the glue tail.
+
+## SERVE-FRONTEND-SLICE: the c2-c8 frontend residual MEASURED — the frontend is ~99% CPU-IDLE, so the slice is handoff/scheduling LATENCY not compute; premise "~0.7ms dominant frontend term" REFUTED; the c2-c8 gap is GPU-intrinsic; NO code shipped
+<!-- state: 2026-08-09T20:00 -->
+
+Executed the frontend attribution the campaign owed, on `row/SERVE-FRONTEND-SLICE`
+off `origin/main` `efe94402`. Method step 1 (instrument OUR frontend per-step at
+c8) ran as a `perf` profile of the PRODUCTION server binary under sustained c8
+streaming — NO rebuild: the built binary (dgx `~/mxfp4-bench/build`, sha
+`735f3b8d`) has a `git diff`-IDENTICAL frontend (output_processor / detokenizer /
+serving_completion / api_server / async_llm) to main, so its profile IS main's.
+Workload matches #52 (`/v1/completions` streaming, 1024-in/128-out, greedy, 8
+workers, 60 s); sustained ITL mean 40.8 ms ≈ #52 c8 TPOT 37.56 ms. Box left clean
+(server gone, both locks free, 78 MB perf .data pruned, disk 35 G).
+
+DECISIVE MEASUREMENT: the GB10's two PMU clusters split the work. The ENGINE core
+drew 22 K on-CPU samples (86.5 B cycles, 75 % libcuda + 21 % vdso clock-poll) = GPU
+orchestration. The FRONTEND cores drew only **186 on-CPU samples over 22 s at
+999 Hz → ~0.8 % of wall-time busy = ~99 % IDLE** (blocked on `__poll` / futex). So
+the #52 ~0.26 ms/step residual (TPOT ours 37.56 − vLLM 34.58 = +2.98 ms, GPU SPAN
++2.72 ms = 91 %, residual +0.26 ms) is thread-handoff/scheduling LATENCY +
+syscalls, NOT frontend COMPUTE. Of the tiny frontend compute that exists: kernel/
+syscall 56.6 % (poll/futex/sendmsg + loopback conntrack, a box artifact), libc
+23.6 % (malloc/free churn + memcpy), our code 13.25 % (per-token SSE-frame JSON
+build), libstdc++ 0.93 %. Call-graph: inside `CompletionSseStream::next` ~34 % is
+`to_json(CompletionStreamResponse)` (nlohmann `std::map` init-list construct+teardown
+~26 %, dump only ~8 %) + `RequestOutput::~RequestOutput` free. DETOK is INVISIBLE
+(only tokenizer symbols are per-REQUEST input `BpeMerge` ~1.85 %, which vLLM also
+pays; per-token decode below the 0.15 % floor).
+
+NAMED MECHANISMS (ours file:line vs vLLM): (1) detok default — ours
+`SlowIncrementalDetokenizer` always (`detokenizer.cpp:402-412`), O(seq) copy at
+`:347` but `tokens_` seeded with a ~7-token window (`:515-530`) so caps ~135
+strings ≈ 2 µs/step at 128-tok output = negligible; vLLM defaults to
+`FastIncrementalDetokenizer` = Rust `DecodeStream.step` O(1)
+(`detokenizer.py:59-75,167-236`). (2) SSE serialize — ours nlohmann
+`json(frame).dump()` (`serving_completion.cpp:102`, std::map-backed) vs vLLM
+pydantic-core `model_dump_json` (`completion/serving.py:441`). (3) architecture —
+ours in-process threads + condvar handoffs (NO IPC) vs vLLM EngineCore subprocess
++ ZMQ; we save the IPC vLLM pays, pay thread-wakeup latency; the 0.26 ms is the NET.
+
+VERDICT: the mission's premise (a ~0.7 ms DOMINANT frontend term to close) is
+REFUTED — the slice is ~0.26 ms (#52 same-tool superseded the old cross-tool
+0.7 ms), it is idle-handoff LATENCY not compute, and there is NO dominant closable
+frontend-compute mechanism; the one structural divergence (Slow vs Fast detok) is
+irrelevant at 128-tok output. NO code shipped: mirroring Fast detok (needs HF Rust
+DecodeStream) or a hand-rolled serializer are LOW-ROI general-serving levers (help
+long-output / high-QPS), not parity levers — the frontend is idle so neither moves
+TPOT. The c2-c8 gap is GPU-INTRINSIC (flash refuted; #52 fair graphed marlin+flash+
+glue = 91 %; #57 recovered marlin). MXFP4 PARITY VERDICT stands UNCHANGED: c1 1.020
+PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, mem 2.63x; the residual is NOT
+frontend-closable. RESIDUAL MAP (honest ROI): decode-flash num_splits cap
+(oracle-gated, c1-c2 where we already pass) + norm/quant/act glue fusion (portable,
+modest) are the only remaining GPU-side terms; both are the same intrinsic/glue
+class already recorded, neither crosses c2-c8 to 1.0x alone. Full perf table +
+call-graph in the benchmark record. Evidence
+dgx:`~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}`.
diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md
@@ -309,7 +309,7 @@ built on it rather than keeping the flattering one.
 | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s |
 | Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 |
 | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin |
-| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE FLIPPED DEFAULT-ON**: dense marlin 48-CTA (vs MoE 128), nsys c8 117.8->85.8us/call; ratified 32B near-tie (razor gap 0.000, goldens regen). Goal <1.0x c2-c8, next decode-flash |
+| MXFP4 Qwen3-8B (W4A16 Marlin) | **`KERNEL-MARLIN-DENSE-EXEC` x3 (dense-ON default): c1 1.020, c2/c4/c8 0.962/0.966/0.969, GPU mem 2.63x less** (beats #51 1.005/0.925/0.939/0.953 EVERY axis); #44 3/3, 32B-NVFP4A16 6/6 | **VT_MARLIN_DENSE default-ON**: dense 48-CTA marlin; 32B near-tie ratified. SERVE-FRONTEND-SLICE: c2-c8 slice 0.26ms/step, frontend ~99% CPU-idle (latency not compute), NOT a parity lever; gap GPU-intrinsic |
 | SGLang floor arms | Never ran | Both arms of the SGLang comparison |
 | cuBLAS invocation-parity guard | CI guard landed (CPU); `kGemvHeuristicAlgos` refactor build-verify owed | `nvcc` rebuild + SACRED gate on dgx |
 | Pre-Ampere breadth (Turing `sm_75` / Volta `sm_70` / Pascal) | **No number owed; nothing runs on these arches.** 2026-08-06 `sm_75`: 20/20 TUs PASS (0 err/warn), WMMA bodies + all 3 selectors arch-gated; GB10 SASS byte-identical. [Detail](../.agents/benchmark-record.md) | Port the llama.cpp `fattn-tile`/`fattn-vec` fp16 body. Perf floor when a card exists is **llama.cpp on the same card** (vLLM does not run there) |