row/SERVE-FRONTEND-SLICE: frontend premise REFUTED — cores 99% idle, residual is handoff latency - #58
Closed
localai-bot wants to merge 1 commit into
Closed
row/SERVE-FRONTEND-SLICE: frontend premise REFUTED — cores 99% idle, residual is handoff latency#58localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…D (perf), frontend ~99% CPU-idle so the slice is handoff latency not compute; premise REFUTED, gap is GPU-intrinsic, NO code Executed the frontend attribution the MXFP4 campaign owed. Method step 1 (instrument OUR frontend per-step at c8) ran as a perf profile of the PRODUCTION server binary under sustained c8 streaming, NO rebuild: the built binary (dgx ~/mxfp4-bench/build sha 735f3b8) has a git diff-IDENTICAL frontend (output_processor/detokenizer/serving_completion/api_server/async_llm) to main efe9440, so its profile IS main's. Workload matched #52 (/v1/completions streaming, 1024-in/128-out, greedy, 8 workers); sustained ITL mean 40.8ms ≈ the #52 c8 TPOT 37.56ms. Box left clean (server gone, both locks free, 78MB perf .data pruned, disk 35G). DECISIVE: the frontend cores drew only 186 on-CPU samples over 22s at 999Hz = ~0.8% of wall-time busy = ~99% IDLE (blocked on poll/futex), while the engine core drew 22K samples (75% libcuda + 21% vdso clock-poll). So the #52 ~0.26ms/step residual (91% of the +2.98ms c8 TPOT gap is GPU SPAN) is thread-handoff/scheduling LATENCY + syscalls, NOT frontend COMPUTE. The one structural divergence, our SlowIncrementalDetokenizer (O(seq) copy) vs vLLM's default FastIncrementalDetokenizer (Rust DecodeStream O(1)), is quantitatively negligible at 128-tok output (caps ~135 strings, ~2us/step); the largest frontend-compute term is nlohmann SSE-frame serialization (~34% of the SSE worker path), but the worker is idle so it does not move TPOT. VERDICT: the mission premise (a ~0.7ms DOMINANT frontend term to close) is REFUTED — the slice is ~0.26ms of idle-handoff latency with NO dominant closable compute mechanism. NO code shipped: mirroring Fast detok (needs HF Rust DecodeStream) or a hand-rolled serializer are low-ROI general-serving levers, not parity levers. The c2-c8 gap is GPU-intrinsic (flash refuted; #52 fair graphed marlin+flash+glue = 91%; #57 recovered marlin). MXFP4 PARITY VERDICT stands UNCHANGED: c1 1.020 PASS, c2-c8 0.962/0.966/0.969 BELOW-FLOOR, mem 2.63x; residual NOT frontend-closable. Records: benchmark-record (perf table + call-graph + named mechanisms file:line both sides), anchored state entry, STATUS/BENCHMARKS/NOW MXFP4 rows updated (STATUS paid by compacting the superseded MARLIN-STRUCT narrative, 284050<=284081). Evidence dgx:~/frontend-attrib/{report.txt,cg_report.txt,run.log,load.json}. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… handoff latency (#58) row/SERVE-FRONTEND-SLICE squash, records-only. perf on the production server at sustained c8: frontend cores ~0.8% busy; the ~0.26ms/step residual (#52 same-tool; the 0.7ms was the superseded cross-tool figure) is thread-handoff/syscall latency, not compute. We SAVE vLLM's ZMQ IPC. Detok invisible (~2us/step at 128-tok); SSE JSON (nlohmann map churn) is the only our-code hotspot on a nearly-idle core. Slow-vs-Fast detok + pydantic-vs-nlohmann recorded as general-serving levers, not parity levers. MXFP4 verdict unchanged: c1 1.020 PASS / c2-c8 0.962-0.969, GPU-intrinsic; final residual = num_splits cap + glue fusion. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only. perf profile of the production server under sustained c8 streaming: frontend cores ~0.8% busy (186 samples/22s @999hz) — the residual (~0.26ms/step per #52 same-tool, not the stale 0.7ms cross-tool figure) is thread-handoff/syscall latency, not compute. We SAVE vLLM's ZMQ IPC; detok invisible (~2us/step); the only our-code hotspot is per-token SSE JSON (nlohmann map churn, ~13% of a nearly-idle core). Slow-vs-Fast detok + pydantic-vs-nlohmann named with file:line as low-ROI general-serving levers, NOT parity levers. MXFP4 verdict unchanged: c1 1.020 PASS / c2-c8 0.962-0.969, gap GPU-intrinsic; final residual = num_splits cap (c1-c2 only) + glue fusion (modest).
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys