row/QUANT-CT-MXFP4-C8-DIFF: first same-tool c8 node-trace — residual is MARLIN-structural (58%) - #52
Closed
localai-bot wants to merge 1 commit into
Closed
row/QUANT-CT-MXFP4-C8-DIFF: first same-tool c8 node-trace — residual is MARLIN-structural (58%)#52localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
… dominant; eager gap graph-closeable; #50 settled (QUANT-CT-MXFP4-C8-DIFF) Fresh same-tool (nsys --cuda-graph-trace=node BOTH sides) c8 decode-window per-step diff on the POST-SLIVER binary (box d3b412f == main 4dd4e20/#51). vLLM captured via offline LLM() + VLLM_ENABLE_V1_MULTIPROCESSING=0 (the online EngineCore subprocess is not nsys-capturable at c8), same 8 corpus prompts, matched M=8 (flash gridZ=64 both). Per-step (median, ours EAGER vs vLLM graphed): SPAN 31.21 vs 28.49 ms/step (+2.72; measured TPOT +2.98, GPU 91%). Terms: marlin +897us (180 vs 144 GEMM), inter-kernel GAP +880us (ours 2us/launch eager vs vLLM 0.5us graph-replay), flash +621us (IDENTICAL grid 1x3x64, +11% per-call), glue +256us. VERDICT (a): dominant term = MARLIN — ours routes 5 grouped-MoE-E1 GEMM/layer (gate_up UNFUSED, 144-CTA) vs vLLM 4 dense (gate_up FUSED, 48-CTA), +5.5% eager/+8.5% graphed per step. SETTLES #50: the ubench grouped==dense parity was vLLM-op-vs-vLLM-op at one isolated shape, blind to our kernel's +36 GEMM/step. Fix = grouped->dense-direct + gate_up fuse (delicate ~2000-line port, scoped). Secondary: the eager launch-gap is closed by the existing default-OFF decode graph (VLLM_CPP_QWEN3_DENSE_DECODE_GRAPH), tested byte-coherent, but nets only -334us (bandwidth contention reclaims +547) = +1.3% TPOT; SACRED gate owed. No code shipped. Docs: STATUS (shrink -6, ratchet lowered), BENCHMARKS row, benchmark-record + state (full tables), NOW. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
…l 58% (#52) row/QUANT-CT-MXFP4-C8-DIFF squash, records-only. First same-tool nsys both engines at c8 (vLLM offline in-process, identical prompts, M=8). Fair decomposition of the ~3ms TPOT gap: MARLIN +1,377us (58%) = STRUCTURAL - 5 grouped-E1 GEMMs/layer (gate+up unfused, 144 CTAs) vs vLLM's 4 dense fused (48 CTAs), +25% GEMM count/step; settles #50 (the isolated-shape ubench was blind to count+CTA structure). Flash +658us (28%, +11%/call at IDENTICAL grid, unresolved params diff). Glue +290us (12%, portable-fusion class). The #51 binding ran EAGER: classic-dense decode graph is opt-in OFF; graph-ON = byte-coherent, gap 1184->305us (=vLLM), TPOT -1.3%, SACRED gate owed before flip. STATUS ratchet lowered 287838->287832. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
Collaborator
Author
|
Landed on main as the squash (mudler authorship). 🤖 Generated with Claude Code |
This was referenced Aug 6, 2026
Closed
mudler
added a commit
that referenced
this pull request
Aug 6, 2026
… handoff latency (#58) row/SERVE-FRONTEND-SLICE squash, records-only. perf on the production server at sustained c8: frontend cores ~0.8% busy; the ~0.26ms/step residual (#52 same-tool; the 0.7ms was the superseded cross-tool figure) is thread-handoff/syscall latency, not compute. We SAVE vLLM's ZMQ IPC. Detok invisible (~2us/step at 128-tok); SSE JSON (nlohmann map churn) is the only our-code hotspot on a nearly-idle core. Slow-vs-Fast detok + pydantic-vs-nlohmann recorded as general-serving levers, not parity levers. MXFP4 verdict unchanged: c1 1.020 PASS / c2-c8 0.962-0.969, GPU-intrinsic; final residual = num_splits cap + glue fusion. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode]
This was referenced Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records-only. First same-tool nsys node-trace both engines at c8 (vLLM captured offline in-process; identical prompts, matched M=8). Fair graphed-vs-graphed decomposition of the ~3ms TPOT gap: MARLIN +1,377us (58%) — STRUCTURAL: 5 grouped-E1 GEMMs/layer (gate+up unfused, 144 CTAs) vs vLLM 4 dense fused (48 CTAs), +25% GEMM count/step (settles #50: the isolated-shape ubench was blind to count+CTA structure); flash +658us (28%, +11%/call at IDENTICAL grid, params diff unresolved); glue +290us (12%, portable-fusion class). BONUS: the binding ran EAGER — the classic-dense decode graph is opt-in default-OFF; enabling = byte-coherent +1.3% TPOT (gap 1184->305us, = vLLM's), needs its SACRED gate. Next: (1) decode-graph gate+flip, (2) the dense-direct marlin + gate_up fusion port (now per-step-justified), (3) flash params diff.
🤖 Generated with Claude Code
https://claude.ai/code/session_014fZAcg1WcU8V629k6HWKys