From b0fd2af7eb49419cad4880ea6bc0e364f05bc316 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Wed, 12 Aug 2026 22:50:49 +0000 Subject: [PATCH 01/11] docs(dspark): storage refuted; ratio stable at ~0.966 over three sessions (#442) The developer asked whether the weights sitting on NAS, or not being fully resident, could be distorting these measurements. Tested and refuted: the weights are on local NVMe (no NAS mount exists on the box), a run reads 22.06 GB once at load, decode-time RSS is 4.8 GB because the mapping is released after upload, and 8 warm reps hold a 0.5% spread. File-backed weights could not produce that stability, and on GB10 the failure mode would be loud anyway -- host pages reach the GPU through ATS at a measured 20-30% per-GEMM penalty. Recorded alongside it: that NVMe is 98% full (76 GB free). This repo has already lost a gate run to ENOSPC reporting green over work that never ran. Also records a third WITHIN-session paired ratio, ours -> oracle -> ours at free clocks with drift bracketed at -0.89%: 140.98 vs a modal 147.32, RATIO 0.9569. With the two pinned-clock pairs (0.9757, 0.9646) that puts the gap at ~0.966 +/- 0.01, consistently below 1.0, while the ABSOLUTE numbers move up to 5% between sessions for the same binary because GB10's memory clock cannot be pinned. Only the within-session ratio is quotable, and all three agree. The oracle's draws remain bimodal (~147.3 vs ~155.6) from the same one-extra- accepted-token effect, so its modal draws stay the honest denominator. No code change; this is the record catching up with the measurements. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 31 ++++++++++++++++++++++ .agents/specs/dspark-spec-decode.md | 41 +++++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 11 +++++++- 4 files changed, 83 insertions(+), 2 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 0d27114e0..126cb961e 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20353,3 +20353,34 @@ per stream), NOT as a perf fix; C_tmp size is ELIMINATED as an explanation of th Method: third time drift has fooled a before/after here. Pairing caught the first, pinned clocks the second, and only an in-process toggle catches this one. Future perf claims on this row need the toggle, not two runs. + +## SPEC-DSPARK: storage ruled out; ratio stable at ~0.966 across three sessions (2026-08-12) + +Question raised: are the weights on NAS, or not fully resident, distorting the +measurements? + +Weights are on LOCAL NVMe (/dev/nvme0n1p2 ext4); no NAS mount exists on the box. +A run reads 22.06 GB total = one full model read at load. Process RSS during +decode is 4.8 GB, so weights are uploaded and the mapping released, not held. +Decode is stable to 0.5% across 8 warm reps (146.0-147.6), which file-backed +weights could not be. Storage is NOT a factor. + +Operational: that NVMe is 98% full (76 GB free), and this repo has already lost a +gate run to ENOSPC reporting green over work that never ran. + +Within-session ratios, three independent measurements: + +| session | ours | oracle (modal) | ratio | +|---|---|---|---| +| pinned clocks, pre-C_tmp | 135.98 | 139.36 | 0.9757 | +| pinned clocks, post-C_tmp | 139.20 | 144.32 | 0.9646 | +| free clocks, ours->oracle->ours | 140.98 | 147.32 | 0.9569 | + +~0.966 +/- 0.01, consistently below 1.0. Absolute numbers move up to 5% between +sessions for the SAME binary because GB10's memory clock cannot be pinned, so +only the within-session ratio is quotable -- and all three agree. + +Oracle draws remain bimodal (~147.3 and ~155.6), the same one-extra-accepted-token +effect as the fibacc run, so its MODAL draws are the honest denominator. + +Evidence: `dgx:~/work/dspark-w6/iocheck.log`, `final_pair.log`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 3f0c6bc57..b885405a6 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1623,6 +1623,47 @@ in-process A/B switch, because on a machine whose memory clock cannot be pinned, even a pinned-clock before/after across two runs is not a controlled experiment. Any future perf claim on this row needs the toggle, not two runs. +## 6ac. STORAGE RULED OUT, and the ratio is stable at ~0.966 (2026-08-12) + +Developer question: could the weights being on NAS, or not fully resident in +device memory, be distorting these measurements? + +**Tested, and no.** + +| check | result | +|---|---| +| weight location | local NVMe (`/dev/nvme0n1p2`, ext4) -- **no NAS mount exists on this box** | +| total disk read for a run | 22.06 GB, i.e. ONE full model read at load | +| process RSS during decode | 4.8 GB -- the weights are NOT held in host RSS; they are uploaded and the mapping released | +| decode stability | 8 warm reps at 146.0 / 147.6 / 147.2 / 146.9 / 147.2 / 147.3 / 147.4 -- **0.5% spread** | + +If weights were still file-backed, decode would fault pages from NVMe and the +per-rep numbers would be erratic; a 0.5% spread says they are resident and decode +touches no storage. Worth keeping in mind that on GB10 the failure mode would be +severe if it ever regressed -- host pages reach the GPU through ATS, which this +repo measured at a 20-30% per-GEMM penalty. + +**Operational finding worth acting on separately:** that NVMe is **98% full** +(3.4T of 3.6T, 76 GB free). This repo has already lost a gate run to ENOSPC +producing a green-looking report over work that never executed. + +**And the ratio is now measured three times, each WITHIN one session:** + +| session | ours | oracle (modal) | ratio | +|---|---|---|---| +| pinned clocks, pre-C_tmp | 135.98 | 139.36 | 0.9757 | +| pinned clocks, post-C_tmp | 139.20 | 144.32 | 0.9646 | +| free clocks, ours->oracle->ours | 140.98 | 147.32 | **0.9569** | + +**~0.966 +/- 0.01, consistently below 1.0.** The absolute numbers move a lot +between sessions (135.98 to 142.09 for the same binary) because GB10's memory +clock cannot be pinned, which is exactly why only the within-session ratio is +quotable -- and all three agree. + +Also visible in the last run: the oracle's draws are bimodal at ~147.3 and +~155.6, the same one-extra-accepted-token effect as §6q, which is why its MODAL +draws are the honest denominator. + ## 7. Evidence, authority, stop conditions - Evidence root: `dgx:~/work/vllm.cpp-dspark-/`, one `flock`, named tmux. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 71d3964d0..3a2fa3149 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **0.965x-0.976x** within-session; C_tmp cap landed but is perf-NEUTRAL (+0.03%) (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **~0.966x +/- 0.01** over three within-session pairs; C_tmp cap perf-NEUTRAL; storage refuted (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 86b20c444..6df60e9c4 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -529,7 +529,16 @@ memory-system behaviour that no allocation change we can name would alter; upstr ncu in either replay mode. A C_tmp over-allocation (15-30 MB vs upstream's 3.15 MB) was found and fixed, but an in-session A/B shows it is perf-NEUTRAL (+0.03%) -- an apparent +2.9% was machine drift, since GB10 cannot lock memory -clocks. Editing +clocks. The ratio has now been measured WITHIN a single session three times -- +0.9757, 0.9646 and (ours->oracle->ours at free clocks, drift -0.89%) 0.9569 -- +so it is **~0.966 +/- 0.01, consistently below 1.0**, while the absolute numbers +move up to 5% BETWEEN sessions for the same binary. Storage was raised as a +possible distortion and is refuted: the weights are on local NVMe (no NAS mount +exists on the box), a run reads 22.06 GB once at load, decode-time RSS is 4.8 GB +because the mapping is released after upload, and 8 warm reps hold a 0.5% +spread -- decode touches no storage. (That NVMe is 98% full, 76 GB free, which +is its own operational risk given ENOSPC has previously produced a green report +over a gate that never ran.) Editing the kernel, its launch config, layout or flags is NOT indicated: all are proven identical. (The repack kernels that appear to take 40% of a long run are LOAD-TIME.) NOT parity. The Gemma4 `1 + N` layout is coded and unit-tested but has From b87b6a503c18e3b22ed2fb209d4cf7352ba6886e Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 00:44:28 +0000 Subject: [PATCH 02/11] perf(dspark): unblock upstream ncu, and REFUTE the DRAM-bound attribution (#442) Section 6z listed upstream's ncu counters as the last open route and recorded it BLOCKED: vLLM's EngineCore will not initialise under ncu in either replay mode, so its DRAM efficiency stayed DERIVED rather than counted, and the spec named a standalone harness as the only remaining way in. scripts/marlin-moe-standalone.py is that harness. It drives upstream's own torch.ops._moe_C.moe_wna16_marlin_gemm on the 35B-A3B decode shapes -- hidden 2048, moe_intermediate 512, E=256, top_k=8, moe_block_size 8 -- with no EngineCore, no multiprocessing and no model load, mirroring prepare_nvfp4_moe_layer_for_marlin's scale pipeline exactly. It asserts the oracle identity and aborts on mismatch, because this repo has three times measured a rollback venv believing it was the pin. It profiles. And the counters refute what 6z concluded from them: Memory Throughput 11.14% of peak Compute (SM) 11.42% of peak Occupancy 25% theoretical / 25.98% achieved Block limit shared memory = 3 Waves per SM 1 grid / block / shmem 144 / 128 / 32768 B Both throughputs at ~11% is the LATENCY-bound signature. A kernel at 11% of memory peak is not against a bandwidth wall, so "we sustain 186.6 GB/s against upstream's 210.7, and that IS the per-unit-work difference" does not hold -- those numbers were time x analytic bytes, so they restated the time gap instead of explaining it. The geometry shows the real limiter. GB10 has 48 SMs with 102400 B of shared memory each; the kernel takes 32768 B per block, giving 3 blocks/SM and 25% occupancy, and 48 x 3 = 144 is exactly the grid. It is a persistent single wave. So the per-unit-work normalisation in 6x is also invalid: 38.9 vs 40.6 "blocks per launch" are loop iterations inside a FIXED 144-CTA grid, not parallel work, and cost is set by the CTA with the most work. ncu flags exactly that, plus uncoalesced access at 20.2 of 32 bytes per load sector. What does NOT change: the 3.4% gap itself. Our kernel through this same harness on identical routing is not yet run, and no claim about the gap may rest on an upstream-only measurement. What changes is its attribution. What this opens: ~9x of absolute headroom that BOTH engines leave on the table, since occupancy is capped purely by a shared-memory budget Marlin computes as max_shared_mem / blocks_per_sm - 1024. Under 25600 B would allow 4 blocks/SM. Not a parity lever -- upstream pays it identically -- a best-in-class one. The synthetic router draws uniformly over 256 experts and so occupies 61 blocks against the model's 38.9-40.6; absolute us/call from this harness is therefore NOT comparable in-situ, while the static geometry and throughput percentages are. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 41 ++++++++ .agents/specs/dspark-spec-decode.md | 74 ++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 19 +++- scripts/marlin-moe-standalone.py | 143 ++++++++++++++++++++++++++++ 5 files changed, 275 insertions(+), 4 deletions(-) create mode 100755 scripts/marlin-moe-standalone.py diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 126cb961e..b94a8de4a 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20384,3 +20384,44 @@ Oracle draws remain bimodal (~147.3 and ~155.6), the same one-extra-accepted-tok effect as the fibacc run, so its MODAL draws are the honest denominator. Evidence: `dgx:~/work/dspark-w6/iocheck.log`, `final_pair.log`. +## SPEC-DSPARK / #442: upstream Marlin profiled under ncu at last; DRAM-bound REFUTED (2026-08-13) + +`scripts/marlin-moe-standalone.py` drives upstream's own +`torch.ops._moe_C.moe_wna16_marlin_gemm` on the 35B-A3B decode shapes with no +EngineCore, no multiprocessing and no model load, which is what makes `ncu` +attach -- the blocker recorded as "BLOCKED, both replay modes TRIED". + +vLLM 0.23.1rc1.dev1511+g555967922 (identity asserted), torch 2.13.0+cu130, +`--set full`, one launch, GB10: + +| counter | value | +|---|---| +| Memory Throughput | 11.14% of peak | +| Compute (SM) Throughput | 11.42% of peak | +| Theoretical / Achieved Occupancy | 25% / 25.98% | +| Block Limit shared memory | 3 (registers 5, warps 12, SM 24) | +| Waves Per SM | 1 | +| L2 / L1 hit | 4.08% / 0.34% | +| grid / block / shared per block | 144 / 128 / 32768 B | + +48 SMs x 102400 B shared per SM, 32768 B per block => 3 blocks/SM => 25% +occupancy, and 48 x 3 = 144 = the grid. The launch is a persistent single wave. + +Both throughputs ~11% is the LATENCY-bound signature, not DRAM-bound. The +186.6 vs 210.7 GB/s figures were derived (time x analytic bytes), never counted, +so they restated the time difference rather than explaining it. The per-unit-work +normalisation is invalid too: 38.9 vs 40.6 "blocks per launch" are loop +iterations inside a FIXED 144-CTA grid, so cost is set by max work per CTA, and +ncu flags load imbalance plus uncoalesced access (20.2/32 bytes per load sector). + +NOT yet run: our kernel through the same harness on identical routing. No claim +about the 3.4% gap changes on this evidence; only its attribution does. Absolute +us/call is not comparable in-situ (uniform synthetic routing gives 61 occupied +blocks vs the model's 38.9-40.6); the static geometry and throughput percentages +are. + +Standing headroom, applying to BOTH engines: occupancy is capped purely by a +32 KB shared-memory budget (`max_shared_mem / blocks_per_sm - 1024`). Under +25600 B would allow 4 blocks/SM. + +Evidence: `dgx.casa:~/work/marlin442/`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index b885405a6..57dd3a4e1 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1663,6 +1663,80 @@ quotable -- and all three agree. Also visible in the last run: the oracle's draws are bimodal at ~147.3 and ~155.6, the same one-extra-accepted-token effect as §6q, which is why its MODAL draws are the honest denominator. +## 6ad. THE ncu BLOCKER IS GONE, AND "DRAM-BOUND" IS REFUTED (2026-08-13) + +Section 6z listed upstream's `ncu` counters as the last open route and called it +BLOCKED: vLLM's EngineCore will not initialise under `ncu` in either replay mode. +`scripts/marlin-moe-standalone.py` routes around it. It drives upstream's OWN +`torch.ops._moe_C.moe_wna16_marlin_gemm` on the 35B-A3B decode shapes -- hidden +2048, moe_intermediate 512, E=256, top_k=8, `moe_block_size=8` -- with NO +EngineCore, NO multiprocessing and NO model load, mirroring +`prepare_nvfp4_moe_layer_for_marlin`'s scale pipeline exactly (per-expert +`marlin_permute_scales` on the transposed scale, one shared scale factor, +S0E5M3 conversion). It asserts the oracle identity and aborts on mismatch. + +**It profiles.** vLLM 0.23.1rc1.dev1511+g555967922, torch 2.13.0+cu130, +`--set full`, one launch: + +| counter | value | +|---|---| +| Memory Throughput | **11.14%** of peak | +| Compute (SM) Throughput | **11.42%** of peak | +| Theoretical / Achieved Occupancy | 25% / 25.98% | +| Block Limit -- **shared memory** | **3** (registers 5, warps 12, SM 24, barriers 24) | +| Waves Per SM | **1** | +| L2 hit / L1 hit | 4.08% / 0.34% | +| registers per thread | 94 | +| SM frequency | 2.14 GHz | + +**Both throughputs are ~11%.** Section 6z read "SM throughput 10.6%" as evidence +the kernel was DRAM-bound and concluded that "for this kernel, time and achieved +bandwidth are the same measurement". That inference does not survive the +counters: a kernel at 11% of memory peak is nowhere near a bandwidth wall. Low +compute AND low memory throughput at 25% occupancy is the LATENCY-bound +signature. The 186.6 vs 210.7 GB/s figures were always DERIVED (time x analytic +bytes), never counted, so they measured the time difference and renamed it +bandwidth; they cannot also explain it. + +**The geometry says why.** GB10 has 48 SMs and 102400 bytes of shared memory per +SM. The kernel takes 32768 bytes per block, so 102400/32768 = 3 blocks per SM, +which is exactly the reported shared-memory block limit and caps occupancy at +25%. The launch is `grid=144`, `block=128`, and 48 x 3 = 144: **the grid is a +persistent single wave sized to the device.** + +That invalidates the per-unit-work normalisation in 6x. The "38.9 vs 40.6 blocks +per launch" are work items each CTA LOOPS OVER, not parallel blocks -- the grid +is a fixed 144 CTAs in both arms. So "4.21 vs 3.73 us per block" divided by the +wrong denominator: more work items do not add parallel work, they add loop +iterations spread over the same 144 CTAs. With a persistent grid the cost is set +by the CTA with the MOST work, and `ncu` flags exactly that -- "avoid possible +load imbalances due to highly different execution durations per [block]" -- along +with uncoalesced access, 20.2 of 32 bytes per global load sector and 31.9 of 32 +per store. + +**Two consequences, and the second is the more valuable one.** + +1. The us/block gap is not established. Re-deriving it needs max-work-per-CTA, + not total work over block count. +2. **The kernel has ~9x of absolute headroom on this device and BOTH engines + leave it on the table.** Occupancy is capped at 3 blocks/SM purely by a 32 KB + shared-memory budget that Marlin computes as + `max_shared_mem / blocks_per_sm - 1024`. This is not a parity lever -- upstream + pays it identically -- it is a BEST-IN-CLASS lever, which is the standing + goal. Getting under 25600 bytes would allow 4 blocks/SM. + +**Open, and honest about it:** this measures UPSTREAM standalone. The comparison +this unblocks -- our kernel through the same harness on identical routing input, +counters side by side -- is NOT yet run, so no claim about the 3.4% gap changes +on this evidence. What changes is the ATTRIBUTION of the residual: "achieved +memory bandwidth" is refuted as its cause, and the mechanism is latency at 25% +occupancy in a one-wave persistent grid. + +The synthetic router draws uniformly over 256 experts, which yields 61 occupied +blocks against the 38.9-40.6 the model produces (real routing over 9 near-identical +spec-decode tokens is far more concentrated). Absolute us/call from this harness +is therefore NOT comparable to the in-situ numbers; the static geometry, occupancy +and throughput percentages are, since they do not depend on the routing draw. ## 7. Evidence, authority, stop conditions diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 3a2fa3149..113a47b5d 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **~0.966x +/- 0.01** over three within-session pairs; C_tmp cap perf-NEUTRAL; storage refuted (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** over three within-session pairs. DRAM-bound cause **REFUTED**: latency-bound at 25% occupancy (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 6df60e9c4..b9e0c3bd9 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -520,9 +520,22 @@ shared-memory budget, reduction flags, scale layout, alignment, residency, CUDA toolkit (13.0 both) and arch all match -- and `ncu` plus cuobjdump then showed the COMPILED KERNELS ARE EQUIVALENT (94 registers and 3664 SASS instructions on both, upstream running its family-compatible sm_120 cubin against our sm_121a). The -residual is therefore runtime and is now ATTRIBUTED: the kernel is DRAM-bound -(L2 hit 9.5%) and we sustain **186.6 GB/s against upstream's 210.7**, a 12.9% -effective-bandwidth gap that IS the whole per-unit-work difference. Weight +residual is therefore runtime. It was ATTRIBUTED to effective DRAM bandwidth +(186.6 vs 210.7 GB/s) and that attribution is now **REFUTED**: those figures were +DERIVED from time x analytic bytes, never counted, so they restated the time gap +rather than explaining it. `scripts/marlin-moe-standalone.py` drives upstream's +own `moe_wna16_marlin_gemm` with no EngineCore, which is what finally lets `ncu` +attach, and the counters say **11.14% of memory peak and 11.42% of compute peak** +-- the LATENCY-bound signature, not a bandwidth wall. GB10's 48 SMs hold 102400 B +of shared memory each, the kernel takes 32768 B per block, so occupancy is capped +at 3 blocks/SM (**25%**) and the grid is 48x3 = **144 CTAs, one persistent wave**. +That also invalidates the per-unit-work normalisation: 38.9 vs 40.6 "blocks per +launch" are loop iterations inside a FIXED grid, so cost is set by max work per +CTA, and ncu flags load imbalance and uncoalesced access (20.2 of 32 bytes per +load sector). Our kernel through the same harness is NOT yet run, so the 3.4% gap +itself is unchanged -- only its cause is. Separately this leaves ~9x of absolute +headroom that BOTH engines pay: a shared-memory budget under 25600 B would allow +4 blocks/SM. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under diff --git a/scripts/marlin-moe-standalone.py b/scripts/marlin-moe-standalone.py new file mode 100755 index 000000000..8e5e33f94 --- /dev/null +++ b/scripts/marlin-moe-standalone.py @@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +"""Standalone upstream Marlin MoE driver for #442. + +Drives vLLM's OWN `moe_wna16_marlin_gemm` on the 35B-A3B decode shapes with NO +EngineCore, NO multiprocessing and NO model load, so that `ncu` can attach -- +which is the blocker recorded in the spec (upstream's engine will not initialise +under ncu in either replay mode). + +Shapes come from nvidia/Qwen3.6-35B-A3B-NVFP4: hidden 2048, moe_intermediate +512, E=256, top_k=8. Marlin's runtime is data-independent, so random weights +measure the same kernel the model does. +""" + +import argparse +import os +import sys + +import torch + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--m", type=int, default=9, help="tokens (1 + k spec draft)") + ap.add_argument("--iters", type=int, default=200) + ap.add_argument("--warmup", type=int, default=20) + ap.add_argument("--arm", choices=["gate_up", "down", "both"], default="both") + args = ap.parse_args() + + import vllm + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_make_workspace_new, + ) + from vllm.model_executor.layers.quantization.utils import marlin_utils_fp4 as f4 + from vllm.model_executor.layers.fused_moe.moe_align_block_size import ( + moe_align_block_size, + ) + from vllm.scalar_type import scalar_types + + # ORACLE IDENTITY. A rollback venv silently invalidates every number here; + # this repo has already measured a rollback three times believing it was the + # pin. Abort rather than measure the wrong build. + want = "555967922" + if want not in vllm.__version__: + print(f"ABORT: vllm {vllm.__version__} is not the pinned {want}") + return 2 + print(f"vllm={vllm.__version__} torch={torch.__version__}", flush=True) + + dev = torch.device("cuda") + torch.cuda.init() + + E, K, N, TOPK = 256, 2048, 512, 8 + GROUP = 16 + BLOCK_M = 8 + M = args.m + dtype = torch.bfloat16 + quant_type = scalar_types.float4_e2m1f + + perm = torch.empty(0, dtype=torch.int, device=dev) + + def build(size_n: int, size_k: int): + """Repacked weights + processed scales for one grouped GEMM. + + Mirrors prepare_nvfp4_moe_layer_for_marlin's permute_scales exactly -- + per-expert marlin_permute_scales on the transposed scale, one shared + scale_factor across experts, then the S0E5M3 conversion. + """ + w = torch.randint(0, 255, (E, size_n, size_k // 2), dtype=torch.uint8, + device=dev) + wq = f4._repack_marlin_experts(w, size_n, size_k, perm, False) + + # Positive scales, as NVFP4 weight scales always are. + s = (torch.rand((E, size_n, size_k // GROUP), dtype=dtype, device=dev) + * 0.5 + 0.5) + factor = f4._nvfp4_compute_scale_factor(s, dtype) + per_expert = [] + for i in range(E): + ms = f4.marlin_permute_scales(s=s[i].T, size_k=size_k, + size_n=size_n, group_size=GROUP, + is_a_8bit=False) + ms, _ = f4.nvfp4_marlin_process_scales(ms, scale_factor=factor, + a_dtype=dtype) + per_expert.append(ms) + s_proc = torch.cat([x.unsqueeze(0) for x in per_expert], 0) + + s2 = torch.ones(E, dtype=torch.float32, device=dev) + g_proc = f4.nvfp4_marlin_process_global_scale(s2, dtype) / factor + return wq, s_proc, g_proc + + print("building weights...", flush=True) + w1, w1s, w1g = build(2 * N, K) # gate_up: size_n = 2*512, size_k = 2048 + w2, w2s, w2g = build(K, N) # down: size_n = 2048, size_k = 512 + workspace = marlin_make_workspace_new(dev, 4) + + topk_ids = torch.randint(0, E, (M, TOPK), dtype=torch.int32, device=dev) + topk_w = torch.rand((M, TOPK), dtype=torch.float32, device=dev) + sorted_ids, expert_ids, num_past = moe_align_block_size( + topk_ids, BLOCK_M, E, None) + + a1 = torch.randn((M, K), dtype=dtype, device=dev) + c1 = torch.empty((M * TOPK, 2 * N), dtype=dtype, device=dev) + a2 = torch.randn((M * TOPK, N), dtype=dtype, device=dev) + c2 = torch.empty((M * TOPK, K), dtype=dtype, device=dev) + + ops = torch.ops._moe_C + + def gate_up(): + return ops.moe_wna16_marlin_gemm( + a1, c1, w1, None, w1s, None, w1g, None, None, None, workspace, + sorted_ids, expert_ids, num_past, topk_w, + BLOCK_M, TOPK, False, quant_type.id, M, 2 * N, K, + True, False, True, False, -1, -1, -1) + + def down(): + return ops.moe_wna16_marlin_gemm( + a2, c2, w2, None, w2s, None, w2g, None, None, None, workspace, + sorted_ids, expert_ids, num_past, topk_w, + BLOCK_M, 1, True, quant_type.id, M * TOPK, K, N, + True, False, True, False, -1, -1, -1) + + arms = {"gate_up": gate_up, "down": down} + if args.arm != "both": + arms = {args.arm: arms[args.arm]} + + for name, fn in arms.items(): + for _ in range(args.warmup): + fn() + torch.cuda.synchronize() + st, en = torch.cuda.Event(True), torch.cuda.Event(True) + st.record() + for _ in range(args.iters): + fn() + en.record() + torch.cuda.synchronize() + us = st.elapsed_time(en) * 1000.0 / args.iters + print(f"UPSTREAM {name} M={M} blocks_past={int(num_past.item())} " + f"us_per_call={us:.3f}", flush=True) + + print("DONE", flush=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) From 44c0a48a77fb42ee28ac2a58ffc0d3b93974f274 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 01:05:45 +0000 Subject: [PATCH 03/11] fix(record): retract the "latency-bound" reading -- 6z's DRAM attribution stands (#442) The previous commit on this branch read ncu's SpeedOfLight line -- "Memory Throughput 11.14%" beside "Compute (SM) Throughput 11.42%" -- as neither resource being saturated, and concluded the Marlin kernel is latency-bound and 6z's DRAM attribution refuted. That is wrong, and the tell was in the same output: dram__bytes.sum reads n/a on GB10. The chip exposes no DRAM counters, so that memory percentage EXCLUDES DRAM traffic and cannot support any claim about being far from a bandwidth wall. Sweeping the occupied block count settles it without those counters. Under ~18 blocks the touched expert weights fit in L2 and cost 1.15 us/block; above ~27 they stream and us/block is FLAT at 5.2-5.5 across a 2.4x range of work. Constant bytes per second across varying work is the bandwidth-limited signature. At 1179648 B per block (1 MiB of weights + 128 KiB of scales) that plateau is 203-226 GB/s. 6z's derived in-situ numbers land ON it. Upstream's 210.7 GB/s is INSIDE the plateau, so upstream runs at this kernel's achievable bandwidth; ours at 186.6 GB/s is ~12% below. The standalone harness CORROBORATES the attribution it was built to test rather than overturning it. Two things do change. 6x's per-unit-work division does not survive a fixed 144-CTA persistent grid, where block count is loop iterations and the sweep prices 38.9 -> 40.6 blocks at about +4.4%. And the "~9x of absolute headroom" claimed by the retracted commit was an artefact of the same unusable percentage: at ~75-80% of this device's ~273 GB/s, occupancy is a secondary lever. The geometry findings are unaffected -- they are static properties, not counter-derived: grid 144 = 48 SMs x 3 blocks, 32768 B shared per block against 102400 B per SM, 25% occupancy, one persistent wave. Corrected in place rather than force-pushed, so the branch keeps both the wrong reading and its retraction. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 75 ++++++++------- .agents/specs/dspark-spec-decode.md | 142 ++++++++++++++++------------ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 34 +++---- 4 files changed, 140 insertions(+), 113 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index b94a8de4a..d6a0b602a 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20384,44 +20384,53 @@ Oracle draws remain bimodal (~147.3 and ~155.6), the same one-extra-accepted-tok effect as the fibacc run, so its MODAL draws are the honest denominator. Evidence: `dgx:~/work/dspark-w6/iocheck.log`, `final_pair.log`. -## SPEC-DSPARK / #442: upstream Marlin profiled under ncu at last; DRAM-bound REFUTED (2026-08-13) +## SPEC-DSPARK / #442: upstream Marlin profiled under ncu at last; 6z CORROBORATED (2026-08-13) `scripts/marlin-moe-standalone.py` drives upstream's own `torch.ops._moe_C.moe_wna16_marlin_gemm` on the 35B-A3B decode shapes with no EngineCore, no multiprocessing and no model load, which is what makes `ncu` -attach -- the blocker recorded as "BLOCKED, both replay modes TRIED". +attach -- the blocker recorded as "BLOCKED, both replay modes TRIED". vLLM +0.23.1rc1.dev1511+g555967922 (identity asserted), torch 2.13.0+cu130, GB10. -vLLM 0.23.1rc1.dev1511+g555967922 (identity asserted), torch 2.13.0+cu130, -`--set full`, one launch, GB10: +STATIC GEOMETRY (trustworthy): grid 144, block 128, 32768 B shared per block. +48 SMs x 102400 B shared => 3 blocks/SM => 25% occupancy (achieved 25.98%), and +48 x 3 = 144 = the grid. A persistent single wave; 94 registers/thread. -| counter | value | -|---|---| -| Memory Throughput | 11.14% of peak | -| Compute (SM) Throughput | 11.42% of peak | -| Theoretical / Achieved Occupancy | 25% / 25.98% | -| Block Limit shared memory | 3 (registers 5, warps 12, SM 24) | -| Waves Per SM | 1 | -| L2 / L1 hit | 4.08% / 0.34% | -| grid / block / shared per block | 144 / 128 / 32768 B | - -48 SMs x 102400 B shared per SM, 32768 B per block => 3 blocks/SM => 25% -occupancy, and 48 x 3 = 144 = the grid. The launch is a persistent single wave. - -Both throughputs ~11% is the LATENCY-bound signature, not DRAM-bound. The -186.6 vs 210.7 GB/s figures were derived (time x analytic bytes), never counted, -so they restated the time difference rather than explaining it. The per-unit-work -normalisation is invalid too: 38.9 vs 40.6 "blocks per launch" are loop -iterations inside a FIXED 144-CTA grid, so cost is set by max work per CTA, and -ncu flags load imbalance plus uncoalesced access (20.2/32 bytes per load sector). - -NOT yet run: our kernel through the same harness on identical routing. No claim -about the 3.4% gap changes on this evidence; only its attribution does. Absolute -us/call is not comparable in-situ (uniform synthetic routing gives 61 occupied -blocks vs the model's 38.9-40.6); the static geometry and throughput percentages -are. - -Standing headroom, applying to BOTH engines: occupancy is capped purely by a -32 KB shared-memory budget (`max_shared_mem / blocks_per_sm - 1024`). Under -25600 B would allow 4 blocks/SM. +MEASUREMENT TRAP: `dram__bytes.sum` is `n/a` on GB10 -- no DRAM counters exist. +The SpeedOfLight "Memory Throughput 11.14%" therefore excludes DRAM traffic, and +reading it beside "Compute (SM) 11.42%" as "latency-bound" is WRONG. + +WHAT THE WORK SWEEP SHOWS (gate_up, 80 iters/point; pool = distinct experts): + +| pool | blocks | us/call | us/block | implied GB/s | +|---|---|---|---|---| +| 8 | 13 | 19.9 | 1.53 | fits L2 | +| 16 | 18 | 20.8 | 1.15 | fits L2 | +| 24 | 22 | 65.0 | 2.96 | transition | +| 32 | 27 | 157.0 | 5.81 | 202.9 | +| 48 | 38 | 207.0 | 5.45 | 216.6 | +| 64 | 45 | 249.5 | 5.55 | 212.7 | +| 128 | 58 | 306.5 | 5.28 | 223.2 | +| 256 | 65 | 339.6 | 5.22 | 225.8 | + +1179648 B streamed per block (1 MiB weights + 128 KiB scales). us/block is FLAT +at 5.2-5.5 across a 2.4x range of work: constant bytes/second, i.e. bandwidth +limited. Plateau 203-226 GB/s. + +CONSEQUENCE: the derived in-situ figures land ON this plateau. Upstream's +210.7 GB/s is INSIDE it (upstream runs at the kernel's achievable bandwidth); +ours at 186.6 GB/s is ~12% BELOW. An independent standalone measurement now +CORROBORATES the DRAM attribution rather than replacing it. What does not +survive is the per-unit-work DIVISION: with a fixed 144-CTA grid, block count is +loop iterations, and the sweep prices 38.9 -> 40.6 blocks at about +4.4%. + +NOT YET RUN, and decisive: our kernel through this same harness, us/block against +the 5.2-5.5 plateau. Occupancy (25%, shared-memory capped, under 25600 B would +buy 4 blocks/SM) is a real but SECONDARY lever at ~75-80% of this device's +~273 GB/s. + +Absolute us/call is not comparable in-situ (uniform synthetic routing occupies +61-65 blocks vs the model's 38.9-40.6); geometry, regime shape and plateau +bandwidth are. Evidence: `dgx.casa:~/work/marlin442/`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 57dd3a4e1..38d988f27 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1663,7 +1663,7 @@ quotable -- and all three agree. Also visible in the last run: the oracle's draws are bimodal at ~147.3 and ~155.6, the same one-extra-accepted-token effect as §6q, which is why its MODAL draws are the honest denominator. -## 6ad. THE ncu BLOCKER IS GONE, AND "DRAM-BOUND" IS REFUTED (2026-08-13) +## 6ad. THE ncu BLOCKER IS GONE, AND 6z's DRAM ATTRIBUTION IS CORROBORATED (2026-08-13) Section 6z listed upstream's `ncu` counters as the last open route and called it BLOCKED: vLLM's EngineCore will not initialise under `ncu` in either replay mode. @@ -1671,72 +1671,88 @@ BLOCKED: vLLM's EngineCore will not initialise under `ncu` in either replay mode `torch.ops._moe_C.moe_wna16_marlin_gemm` on the 35B-A3B decode shapes -- hidden 2048, moe_intermediate 512, E=256, top_k=8, `moe_block_size=8` -- with NO EngineCore, NO multiprocessing and NO model load, mirroring -`prepare_nvfp4_moe_layer_for_marlin`'s scale pipeline exactly (per-expert -`marlin_permute_scales` on the transposed scale, one shared scale factor, -S0E5M3 conversion). It asserts the oracle identity and aborts on mismatch. +`prepare_nvfp4_moe_layer_for_marlin`'s scale pipeline exactly. It asserts the +oracle identity and aborts on mismatch. It profiles. -**It profiles.** vLLM 0.23.1rc1.dev1511+g555967922, torch 2.13.0+cu130, -`--set full`, one launch: +### The static geometry (trustworthy, and new) -| counter | value | +| property | value | |---|---| -| Memory Throughput | **11.14%** of peak | -| Compute (SM) Throughput | **11.42%** of peak | -| Theoretical / Achieved Occupancy | 25% / 25.98% | -| Block Limit -- **shared memory** | **3** (registers 5, warps 12, SM 24, barriers 24) | -| Waves Per SM | **1** | -| L2 hit / L1 hit | 4.08% / 0.34% | +| grid / block / shared per block | 144 / 128 / 32768 B | +| GB10 | 48 SMs, 102400 B shared per SM | +| block limit -- **shared memory** | **3** (registers 5, warps 12, SM 24) | +| occupancy, theoretical / achieved | 25% / 25.98% | +| waves per SM | 1 | | registers per thread | 94 | -| SM frequency | 2.14 GHz | - -**Both throughputs are ~11%.** Section 6z read "SM throughput 10.6%" as evidence -the kernel was DRAM-bound and concluded that "for this kernel, time and achieved -bandwidth are the same measurement". That inference does not survive the -counters: a kernel at 11% of memory peak is nowhere near a bandwidth wall. Low -compute AND low memory throughput at 25% occupancy is the LATENCY-bound -signature. The 186.6 vs 210.7 GB/s figures were always DERIVED (time x analytic -bytes), never counted, so they measured the time difference and renamed it -bandwidth; they cannot also explain it. - -**The geometry says why.** GB10 has 48 SMs and 102400 bytes of shared memory per -SM. The kernel takes 32768 bytes per block, so 102400/32768 = 3 blocks per SM, -which is exactly the reported shared-memory block limit and caps occupancy at -25%. The launch is `grid=144`, `block=128`, and 48 x 3 = 144: **the grid is a -persistent single wave sized to the device.** - -That invalidates the per-unit-work normalisation in 6x. The "38.9 vs 40.6 blocks -per launch" are work items each CTA LOOPS OVER, not parallel blocks -- the grid -is a fixed 144 CTAs in both arms. So "4.21 vs 3.73 us per block" divided by the -wrong denominator: more work items do not add parallel work, they add loop -iterations spread over the same 144 CTAs. With a persistent grid the cost is set -by the CTA with the MOST work, and `ncu` flags exactly that -- "avoid possible -load imbalances due to highly different execution durations per [block]" -- along -with uncoalesced access, 20.2 of 32 bytes per global load sector and 31.9 of 32 -per store. - -**Two consequences, and the second is the more valuable one.** - -1. The us/block gap is not established. Re-deriving it needs max-work-per-CTA, - not total work over block count. -2. **The kernel has ~9x of absolute headroom on this device and BOTH engines - leave it on the table.** Occupancy is capped at 3 blocks/SM purely by a 32 KB - shared-memory budget that Marlin computes as - `max_shared_mem / blocks_per_sm - 1024`. This is not a parity lever -- upstream - pays it identically -- it is a BEST-IN-CLASS lever, which is the standing - goal. Getting under 25600 bytes would allow 4 blocks/SM. - -**Open, and honest about it:** this measures UPSTREAM standalone. The comparison -this unblocks -- our kernel through the same harness on identical routing input, -counters side by side -- is NOT yet run, so no claim about the 3.4% gap changes -on this evidence. What changes is the ATTRIBUTION of the residual: "achieved -memory bandwidth" is refuted as its cause, and the mechanism is latency at 25% -occupancy in a one-wave persistent grid. - -The synthetic router draws uniformly over 256 experts, which yields 61 occupied -blocks against the 38.9-40.6 the model produces (real routing over 9 near-identical -spec-decode tokens is far more concentrated). Absolute us/call from this harness -is therefore NOT comparable to the in-situ numbers; the static geometry, occupancy -and throughput percentages are, since they do not depend on the routing draw. + +102400/32768 = 3 blocks per SM, and 48 x 3 = 144 = the grid exactly. **The launch +is a persistent single wave sized to the device**, so the "38.9 vs 40.6 blocks per +launch" of 6x are work items each CTA LOOPS OVER, not parallel blocks. + +### `ncu`'s memory percentages are NOT usable on this chip + +`dram__bytes.sum` and `dram__bytes.sum.per_second` return **`n/a`** on GB10 -- +there are no DRAM counters to read. What the SpeedOfLight section still prints, +"Memory Throughput 11.14%" against "Compute (SM) Throughput 11.42%", therefore +does NOT include DRAM traffic, and reading those two as "neither is saturated, +so the kernel is latency-bound" is WRONG. An `n/a` on the byte counter is the +tell that the percentage beside it is derived from something else. + +### What the kernel actually does, measured by sweeping the work + +Concentrating the synthetic router over a controlled pool of experts moves the +occupied block count. `--arm gate_up`, 80 iterations per point: + +| distinct-expert pool | blocks | us/call | us/block | implied GB/s | +|---|---|---|---|---| +| 8 | 13 | 19.9 | 1.53 | -- (fits L2) | +| 16 | 18 | 20.8 | **1.15** | -- (fits L2) | +| 24 | 22 | 65.0 | 2.96 | transition | +| 32 | 27 | 157.0 | 5.81 | 202.9 | +| 40 | 31 | 165.0 | 5.32 | -- | +| 48 | 38 | 207.0 | 5.45 | 216.6 | +| 64 | 45 | 249.5 | 5.55 | 212.7 | +| 128 | 58 | 306.5 | 5.28 | 223.2 | +| 256 | 65 | 339.6 | 5.22 | 225.8 | + +Each block streams one expert's gate_up weights (2N x K/2 = 1 MiB) plus its +scales (2N x K/16 = 128 KiB) = 1179648 B. Two regimes are visible. Under ~18 +blocks the touched weights fit in L2 and cost 1.15 us/block. Above ~27 they +stream, and **us/block is FLAT at 5.2-5.5 across a 2.4x range of work** -- +a constant bytes-per-second, which is the bandwidth-limited signature. + +**So 6z was right and this section's first draft was wrong.** The kernel IS +memory-bound; the standalone plateau is **203-226 GB/s**, and 6z's derived +in-situ numbers land exactly on it: upstream's **210.7 GB/s is INSIDE the +plateau**, i.e. upstream runs at this kernel's achievable bandwidth, while our +**186.6 GB/s is ~12% BELOW** it. An independent standalone measurement of the +same kernel now corroborates the derived attribution instead of replacing it. + +The one thing 6x should not keep is the per-unit-work DIVISION: with a fixed +144-CTA grid, block count is loop iterations, and at the operating point the +sweep says +1.7 blocks (38.9 -> 40.6) should cost about +4.4%. Upstream carries +those extra iterations and is still faster, which is not a contradiction once the +denominator is bandwidth rather than block count -- it is the same 12-13% +bandwidth-achievement gap seen twice. + +### What is now open + +Ours has NOT been through this harness. That is the decisive experiment: our +kernel, same shapes, same routing input, same box, us/block against the 5.2-5.5 +plateau. If ours plateaus at 5.2-5.5 too, the in-situ deficit is NOT in the +kernel and 6x's localisation is wrong; if ours plateaus ~12% higher, the kernel +owns it and the next question is why the same SASS sustains less bandwidth. + +Occupancy stays a real but SECONDARY lever: 25% capped by a 32 KB shared-memory +budget (`max_shared_mem / blocks_per_sm - 1024`), where under 25600 B would buy +4 blocks/SM. At ~75-80% of this device's ~273 GB/s that is worth far less than +the earlier "~9x of headroom" first draft claimed, which was an artefact of the +same unusable percentage. + +Absolute us/call here is not comparable in-situ: uniform synthetic routing +occupies 61-65 blocks against the model's 38.9-40.6. The static geometry, the +two-regime shape and the plateau bandwidth are comparable, since they do not +depend on the routing draw. ## 7. Evidence, authority, stop conditions diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 113a47b5d..b8dcb4809 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** over three within-session pairs. DRAM-bound cause **REFUTED**: latency-bound at 25% occupancy (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** over three within-session pairs; ours runs ~12% under the Marlin kernel's measured 203-226 GB/s plateau (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index b9e0c3bd9..c5829186e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -520,22 +520,24 @@ shared-memory budget, reduction flags, scale layout, alignment, residency, CUDA toolkit (13.0 both) and arch all match -- and `ncu` plus cuobjdump then showed the COMPILED KERNELS ARE EQUIVALENT (94 registers and 3664 SASS instructions on both, upstream running its family-compatible sm_120 cubin against our sm_121a). The -residual is therefore runtime. It was ATTRIBUTED to effective DRAM bandwidth -(186.6 vs 210.7 GB/s) and that attribution is now **REFUTED**: those figures were -DERIVED from time x analytic bytes, never counted, so they restated the time gap -rather than explaining it. `scripts/marlin-moe-standalone.py` drives upstream's -own `moe_wna16_marlin_gemm` with no EngineCore, which is what finally lets `ncu` -attach, and the counters say **11.14% of memory peak and 11.42% of compute peak** --- the LATENCY-bound signature, not a bandwidth wall. GB10's 48 SMs hold 102400 B -of shared memory each, the kernel takes 32768 B per block, so occupancy is capped -at 3 blocks/SM (**25%**) and the grid is 48x3 = **144 CTAs, one persistent wave**. -That also invalidates the per-unit-work normalisation: 38.9 vs 40.6 "blocks per -launch" are loop iterations inside a FIXED grid, so cost is set by max work per -CTA, and ncu flags load imbalance and uncoalesced access (20.2 of 32 bytes per -load sector). Our kernel through the same harness is NOT yet run, so the 3.4% gap -itself is unchanged -- only its cause is. Separately this leaves ~9x of absolute -headroom that BOTH engines pay: a shared-memory budget under 25600 B would allow -4 blocks/SM. Weight +residual is therefore runtime and is ATTRIBUTED to effective DRAM bandwidth +(186.6 vs 210.7 GB/s), an attribution now CORROBORATED by an independent +measurement. `scripts/marlin-moe-standalone.py` drives upstream's own +`moe_wna16_marlin_gemm` with no EngineCore, which is what finally lets `ncu` +attach. Its static geometry is new and exact: GB10's 48 SMs hold 102400 B of +shared memory, the kernel takes 32768 B per block, so occupancy is capped at +3 blocks/SM (**25%**) and the grid is 48x3 = **144 CTAs, one persistent wave**. +Sweeping the occupied block count then shows two regimes: under ~18 blocks the +weights fit in L2 (1.15 us/block), above ~27 they stream and **us/block is FLAT +at 5.2-5.5 across a 2.4x range of work** -- constant bytes/second, the +bandwidth-limited signature, a **203-226 GB/s** plateau. Upstream's 210.7 sits +INSIDE that plateau (it runs at the kernel's achievable bandwidth); ours at +186.6 sits ~12% below. Note `ncu`'s SpeedOfLight memory percentage is unusable +here -- `dram__bytes.sum` reads `n/a` on GB10, so "Memory Throughput 11.14%" +excludes DRAM traffic and must not be read as latency-bound. What does NOT +survive is the per-unit-work division: with a fixed 144-CTA grid the "38.9 vs +40.6 blocks" are loop iterations, priced by the sweep at about +4.4%. Ours has +NOT yet run through the harness, which is the decisive next measurement. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under From e60efef7e5af0e7decb7f80b58f27ee3f71694d4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 01:26:14 +0000 Subject: [PATCH 04/11] perf(dspark): the Marlin kernel is NOT the gap -- 6x's localisation REFUTED (#442) 6ad built the upstream arm of the standalone harness. This is the arm it was built for. benchmarks/marlin_moe_standalone.cpp drives OUR vt::MoeGroupedGemmNvfp4Marlin through the same 35B-A3B gate_up shapes, the same expert-pool control over the occupied block count, and the same GEMM as the python arm, so the two kernels can be asked to do the same work on the same box. Ours plateaus on the SAME 5.2-5.5 us/block band as upstream. Two INTERLEAVED paired runs at pool 48 and 128, three reps each: n mean us/block sd range ours 12 5.3187 0.124 5.083-5.562 upstream 12 5.3330 0.151 5.042-5.560 ours/upstream = 0.9973, ours 0.27% FASTER, inside one standard deviation on either side, with the sign of the difference flipping between runs. A per-call workspace memset our arm pays and upstream's does not was isolated behind --zero-ws 0 and is noise. So 6x does not survive. Its "the SAME kernel, the SAME 1520 launches, ours 249.22 ms vs upstream 230.39 ms, 8.2% slower inside one kernel" does NOT reproduce when both kernels are driven with matched work. The "12.8% slower per unit of work" and the 186.6-vs-210.7 GB/s reading derived from it therefore describe the in-situ RUNS, not the kernel: both engines reach the same 203-226 GB/s plateau when asked to do the same thing. What the harness does establish, and it points somewhere new: time is set by how many DISTINCT EXPERTS a launch touches, 1.15 us/block at 16 experts where the weights fit L2, rising to ~5.3 above ~27 where they stream. That is a 4.6x swing no kernel change causes. Blocks are not experts, so the recorded 38.9 vs 40.6 blocks per launch never settled which arm did more work. Measuring distinct experts per launch on both arms IN SITU is the next step, not another kernel lever. docs/STATUS.md's DSpark narrative is collapsed onto the binding result rather than extended, which is what the ratchet asks for when a finding supersedes the story that preceded it. Caveats, stated rather than buried: our arm links ~/work/pr234's build, whose vendored marlin_mm_moe.cu is byte-identical to current main (md5 85c40e4869bc6ec594b8cfb97fb58b3c) while its dispatcher predates the C_tmp cap, independently measured perf-neutral at +0.03%. Ours times with steady_clock over 80 iterations plus a final sync, upstream's with CUDA events; both amortise launch overhead, and the memset probe bounds that class of difference at noise. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 39 ++++++ .agents/specs/dspark-spec-decode.md | 61 +++++++++ benchmarks/marlin_moe_standalone.cpp | 196 +++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 59 ++++---- 5 files changed, 322 insertions(+), 35 deletions(-) create mode 100644 benchmarks/marlin_moe_standalone.cpp diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index d6a0b602a..d5201a906 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20434,3 +20434,42 @@ Absolute us/call is not comparable in-situ (uniform synthetic routing occupies bandwidth are. Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: our Marlin == upstream's at matched work; 6x REFUTED (2026-08-13) + +`benchmarks/marlin_moe_standalone.cpp` is the OUR-side arm of the harness in the +previous entry: same 35B-A3B gate_up shapes, same expert-pool control over the +occupied block count, driving `vt::MoeGroupedGemmNvfp4Marlin`. + +Sweep, us/block, ours vs upstream: 5.482/5.813 (pool 32), 5.525/5.323 (40), +5.311/5.446 (48), 5.318/5.545 (64), 5.194/5.284 (128), 5.261/5.224 (256). Ours +plateaus on the SAME 5.2-5.5 band. + +Two INTERLEAVED paired runs, pool 48 and 128, 3 reps each, 12 points per arm: + +| | n | mean us/block | sd | range | +|---|---|---|---|---| +| ours | 12 | 5.3187 | 0.124 | 5.083-5.562 | +| upstream | 12 | 5.3330 | 0.151 | 5.042-5.560 | + +ours/upstream = 0.9973 (ours 0.27% FASTER), inside one sd, sign flipping between +runs. A per-call workspace memset ours pays and upstream does not was isolated +via `--zero-ws 0`: noise. + +CONSEQUENCE: the in-situ "8.2% slower inside one kernel / 12.8% per unit work / +186.6 vs 210.7 GB/s" does NOT reproduce at matched work. Both engines reach the +same 203-226 GB/s plateau. That attribution describes the in-situ RUNS, not the +kernel, and the localisation of the residual to `marlin_moe_wna16::Marlin` is +REFUTED. + +WHERE TO LOOK NEXT: the sweep shows time is set by DISTINCT EXPERTS touched per +launch (1.15 us/block at 16 experts, weights in L2; ~5.3 above ~27, streaming -- +a 4.6x swing). Blocks are not experts, so the recorded 38.9 vs 40.6 blocks does +not settle it. Measure distinct experts per launch on both arms IN SITU. + +CAVEATS: our arm links ~/work/pr234's build, vendored marlin_mm_moe.cu +byte-identical to main (md5 85c40e4869bc6ec594b8cfb97fb58b3c), dispatcher +predates the perf-neutral C_tmp cap. Ours times with steady_clock over 80 +iterations, upstream with CUDA events. + +Evidence: `dgx.casa:~/work/marlin442/`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 38d988f27..1e6f46de6 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1754,6 +1754,67 @@ occupies 61-65 blocks against the model's 38.9-40.6. The static geometry, the two-regime shape and the plateau bandwidth are comparable, since they do not depend on the routing draw. +## 6ae. THE KERNEL IS NOT THE GAP -- 6x's LOCALISATION IS REFUTED (2026-08-13) + +6ad built the upstream arm. This is the arm it was built for: +`benchmarks/marlin_moe_standalone.cpp` drives OUR +`vt::MoeGroupedGemmNvfp4Marlin` through the same shapes, the same expert-pool +control over the block count, and the same gate_up GEMM as the python arm. + +**Sweep, ours against upstream, us/block:** + +| pool | ours blocks / us-per-block | upstream blocks / us-per-block | +|---|---|---| +| 8 | 13 / 1.512 | 13 / 1.534 | +| 16 | 16 / 1.411 | 18 / 1.153 | +| 24 | 23 / 4.773 | 22 / 2.955 | +| 32 | 30 / 5.482 | 27 / 5.813 | +| 40 | 34 / 5.525 | 31 / 5.323 | +| 48 | 39 / 5.311 | 38 / 5.446 | +| 64 | 44 / 5.318 | 45 / 5.545 | +| 128 | 54 / 5.194 | 58 / 5.284 | +| 256 | 63 / 5.261 | 65 / 5.224 | + +Ours plateaus on the SAME 5.2-5.5 band, so ours reaches the same achievable +bandwidth. Two INTERLEAVED paired runs at pool 48 and 128, three reps each, +then settle it: + +| | n | mean us/block | sd | range | +|---|---|---|---|---| +| ours | 12 | **5.3187** | 0.124 | 5.083-5.562 | +| upstream | 12 | **5.3330** | 0.151 | 5.042-5.560 | + +**ours/upstream = 0.9973, i.e. ours 0.27% FASTER, inside one standard deviation +on either side.** The sign of the difference flips between runs. A per-call +workspace memset that our arm pays and upstream's does not was isolated with +`--zero-ws 0` and is noise. + +**So 6x's localisation does not survive.** Its "the SAME kernel, the SAME 1520 +launches, ours 249.22 ms vs upstream 230.39 ms, 8.2% slower inside one kernel" +does NOT reproduce when the same kernel is driven with matched work on the same +box. At matched blocks the two are indistinguishable. Correspondingly, the +"12.8% slower per unit of work" and the 186.6-vs-210.7 GB/s reading it produced +describe the in-situ RUNS, not the kernel: both engines reach the same 203-226 +GB/s plateau when asked to do the same thing. + +**What that leaves.** The in-situ difference must come from CONTEXT rather than +from kernel efficiency, and the sweep shows precisely which context term +dominates: time is set by how many DISTINCT EXPERTS a launch touches, from +1.15 us/block at 16 experts (weights fit L2) to ~5.3 above ~27 (streaming) -- +a 4.6x swing that no kernel change causes. The leading hypothesis is therefore +that the two in-situ arms were not touching the same number of distinct experts +per launch, which the recorded 38.9 vs 40.6 blocks hints at but does not +measure, because blocks are not experts. The next measurement is distinct +experts per launch on both arms in situ, not another kernel lever. + +**Caveats, stated rather than buried.** Our arm links `~/work/pr234`'s build, +whose vendored `marlin_mm_moe.cu` is byte-identical to current main +(md5 85c40e4869bc6ec594b8cfb97fb58b3c on both) while its dispatcher predates the +C_tmp cap, which was independently measured perf-neutral at +0.03%. Ours times +with `steady_clock` around 80 iterations plus a final sync; upstream's arm uses +CUDA events. Both amortise launch overhead over 80 calls, and the memset probe +bounds that class of difference at noise. + ## 7. Evidence, authority, stop conditions - Evidence root: `dgx:~/work/vllm.cpp-dspark-/`, one `flock`, named tmux. diff --git a/benchmarks/marlin_moe_standalone.cpp b/benchmarks/marlin_moe_standalone.cpp new file mode 100644 index 000000000..0af9f95c4 --- /dev/null +++ b/benchmarks/marlin_moe_standalone.cpp @@ -0,0 +1,196 @@ +// OUR arm of the #442 standalone Marlin harness. +// +// Mirrors scripts/marlin-moe-standalone.py exactly: same 35B-A3B decode shapes +// (hidden 2048, moe_intermediate 512, E=256, top_k=8, moe_block_size 8), same +// gate_up GEMM, same expert-pool control over the occupied block count. Prints +// us/call and us/block so our plateau can be laid against upstream's 5.2-5.5. +// +// Not a test: no assertions, no goldens. It measures the kernel only. + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vt/backend.h" +#include "vt/cuda/marlin_repack.h" +#include "vt/dtype.h" +#include "vt/ops.h" + +namespace { + +using vt::Backend; +using vt::Device; +using vt::DeviceType; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +Device Gpu() { return Device{DeviceType::kCUDA, 0}; } + +Tensor MakeT(void* data, DType dt, Device dev, const std::vector& shape) { + Tensor t; + t.data = data; + t.dtype = dt; + t.device = dev; + t.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = t.rank - 1; i >= 0; --i) { + t.shape[i] = shape[static_cast(i)]; + t.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return t; +} + +class Dev { + public: + Dev(Backend& b, Queue& q, DType dt, const std::vector& shape, + const void* host = nullptr) + : b_(b) { + int64_t numel = 1; + for (auto s : shape) numel *= s; + bytes_ = static_cast(numel) * vt::SizeOf(dt); + p_ = b_.Alloc(bytes_ == 0 ? 1 : bytes_); + if (host != nullptr) b_.Copy(q, p_, host, bytes_); + t_ = MakeT(p_, dt, Gpu(), shape); + } + ~Dev() { b_.Free(p_); } + Dev(const Dev&) = delete; + Dev& operator=(const Dev&) = delete; + Tensor& tensor() { return t_; } + void* ptr() { return p_; } + + private: + Backend& b_; + void* p_ = nullptr; + size_t bytes_ = 0; + Tensor t_; +}; + +int IntArg(int argc, char** argv, const char* name, int fallback) { + for (int i = 1; i + 1 < argc; ++i) + if (std::strcmp(argv[i], name) == 0) return std::atoi(argv[i + 1]); + return fallback; +} + +} // namespace + +int main(int argc, char** argv) { + const int pool_arg = IntArg(argc, argv, "--experts", 0); + const int iters = IntArg(argc, argv, "--iters", 80); + const int warmup = IntArg(argc, argv, "--warmup", 20); + const int M = IntArg(argc, argv, "--m", 9); + const int zero_ws = IntArg(argc, argv, "--zero-ws", 1); + + const int E = 256, K = 2048, N = 512, top_k = 8; + const int pool = pool_arg > 0 ? pool_arg : E; + const int size_n = 2 * N; // gate_up + const int size_k = K; + + Backend& b = vt::GetBackend(DeviceType::kCUDA); + Queue q{Gpu(), nullptr}; + void* stream = nullptr; + const int dev_id = 0; + + // Weights: random packed nibbles, repacked per expert into Marlin layout. + // Marlin's runtime is data independent, so random bits time like real ones. + std::mt19937 rng(1234); + const size_t raw_bytes = static_cast(size_n) * size_k / 2; + std::vector raw(raw_bytes); + for (auto& x : raw) x = static_cast(rng() & 0xFF); + const size_t scale_bytes = static_cast(size_n) * size_k / 16; + std::vector raw_s(scale_bytes); + for (auto& x : raw_s) x = 0x38; // fp8-e4m3 ~ 0.5, safely positive + + Dev staging(b, q, DType::kI8, {size_n, size_k / 2}, raw.data()); + Dev staging_s(b, q, DType::kI8, {size_n, size_k / 16}, raw_s.data()); + + Dev wq(b, q, DType::kI32, {E, size_k / 16, size_n * 2}); + Dev sc(b, q, DType::kI8, {E, size_k / 16, size_n}); + const float sf = 1.0f; + std::vector gs(static_cast(E), + vt::cuda::MarlinNvfp4ProcessGlobalScale(1.0f, sf)); + + const size_t wq_expert_words = static_cast(size_n) * size_k / 2 / 4; + for (int e = 0; e < E; ++e) { + vt::cuda::MarlinRepackExpertWeight( + stream, dev_id, + static_cast(wq.ptr()) + static_cast(e) * wq_expert_words, + static_cast(staging.ptr()), size_k, size_n); + vt::cuda::MarlinProcessExpertScales( + stream, static_cast(staging_s.ptr()), + static_cast(sc.ptr()) + static_cast(e) * scale_bytes, + size_k, size_n, sf); + } + b.Synchronize(q); + Dev dgs(b, q, DType::kF32, {E}, gs.data()); + + // Routing, drawn from `pool` distinct experts -- the block-count control. + const int P = M * top_k; + std::vector topk_ids(static_cast(P)); + std::vector topk_w(static_cast(P), 1.0f); + for (int i = 0; i < P; ++i) + topk_ids[static_cast(i)] = static_cast(rng() % static_cast(pool)); + + const int block = vt::cuda::MarlinMoeAlignBlockSizeSelect(M, top_k, E); + int max_tok = 0, max_blk = 0; + vt::cuda::MarlinMoeAlignSizes(M, top_k, E, block, &max_tok, &max_blk); + Dev dtid(b, q, DType::kI32, {M, top_k}, topk_ids.data()); + Dev dtw(b, q, DType::kF32, {M, top_k}, topk_w.data()); + Dev sorted_ids(b, q, DType::kI32, {max_tok}); + Dev expert_ids(b, q, DType::kI32, {max_blk}); + Dev num_pad(b, q, DType::kI32, {1}); + vt::cuda::MarlinMoeAlignBlockSize(stream, static_cast(dtid.ptr()), M, + top_k, E, block, + static_cast(sorted_ids.ptr()), + static_cast(expert_ids.ptr()), + static_cast(num_pad.ptr())); + b.Synchronize(q); + int32_t past = 0; + b.Copy(q, &past, num_pad.ptr(), sizeof(int32_t)); + b.Synchronize(q); + + const int sms = vt::cuda::MarlinDeviceSms(dev_id); + Dev ws(b, q, DType::kI32, {sms * 4}); + Dev dact(b, q, DType::kBF16, {M, K}); + Dev dout(b, q, DType::kBF16, {P, size_n}); + + vt::MoeMarlinArgs args{}; + args.moe_block_size = block; + args.top_k = top_k; + args.size_m = M; + args.size_n = size_n; + args.size_k = size_k; + args.mul_topk_weights = false; + + // vLLM's arm does NOT re-zero the workspace per call (the kernel leaves it + // reset), so timing ours WITH a per-call memset adds a launch upstream never + // pays. --zero-ws 0 removes that asymmetry. + b.Memset(q, ws.ptr(), 0, static_cast(sms) * 4 * sizeof(int32_t)); + auto once = [&]() { + if (zero_ws) b.Memset(q, ws.ptr(), 0, static_cast(sms) * 4 * sizeof(int32_t)); + vt::MoeGroupedGemmNvfp4Marlin(q, dout.tensor(), dact.tensor(), wq.tensor(), + sc.tensor(), dgs.tensor(), ws.tensor(), + sorted_ids.tensor(), expert_ids.tensor(), + num_pad.tensor(), dtw.tensor(), args); + }; + + for (int i = 0; i < warmup; ++i) once(); + b.Synchronize(q); + const auto t0 = std::chrono::steady_clock::now(); + for (int i = 0; i < iters; ++i) once(); + b.Synchronize(q); + const auto t1 = std::chrono::steady_clock::now(); + + const double us = + std::chrono::duration(t1 - t0).count() / iters; + const int blocks = past / block; + std::printf("OURS gate_up M=%d pool=%d zero_ws=%d blocks=%d us_per_call=%.3f us_per_block=%.4f\n", + M, pool, zero_ws, blocks, us, us / (blocks > 0 ? blocks : 1)); + return 0; +} diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index b8dcb4809..c60dc99b7 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** over three within-session pairs; ours runs ~12% under the Marlin kernel's measured 203-226 GB/s plateau (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** over three within-session pairs. Marlin localisation **REFUTED**: at matched work ours is 0.9973x of upstream (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index c5829186e..26a447aa3 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -504,40 +504,31 @@ MHz, 30 reps, drift bracketed at -0.088%, the oracle's non-modal draws excluded) the code cell is **0.975x with NON-OVERLAPPING distributions** — a real gap, not noise, and the earlier "within resolution" reading was too generous. Ours slowed more than the oracle when the clock was pinned, so the residual is -SM-clock-sensitive work. PAIRED profiling localises it exactly: the SAME -`marlin_moe_wna16::Marlin` kernel, the SAME 1520 launches, ours 249.22 ms vs -upstream 230.39 ms -- **8.2% slower inside one kernel**, which at ~34% of wall is -2.8% end-to-end and accounts for the whole measured 2.5%. Not an algorithm difference, and not the launch -geometry either: the full template arguments match, `determine_exec_config` is -byte-identical to the pinned upstream copy, and every OTHER kernel matches to -0.2%. The inputs match too (scale bytes per expert, -256-byte alignment, cudaMalloc residency), and the work counts were MEASURED: -upstream loops 4.4% MORE blocks per launch (40.6 vs 38.9) and is still faster, so -routing is refuted and normalising by work makes our deficit bigger -- **4.21 vs -3.73 us per block, ~12.8% slower per unit of work**. Every source-level explanation is now -eliminated -- kernel source, template instantiation, grid config, block size, -shared-memory budget, reduction flags, scale layout, alignment, residency, CUDA -toolkit (13.0 both) and arch all match -- and `ncu` plus cuobjdump then showed the -COMPILED KERNELS ARE EQUIVALENT (94 registers and 3664 SASS instructions on both, -upstream running its family-compatible sm_120 cubin against our sm_121a). The -residual is therefore runtime and is ATTRIBUTED to effective DRAM bandwidth -(186.6 vs 210.7 GB/s), an attribution now CORROBORATED by an independent -measurement. `scripts/marlin-moe-standalone.py` drives upstream's own -`moe_wna16_marlin_gemm` with no EngineCore, which is what finally lets `ncu` -attach. Its static geometry is new and exact: GB10's 48 SMs hold 102400 B of -shared memory, the kernel takes 32768 B per block, so occupancy is capped at -3 blocks/SM (**25%**) and the grid is 48x3 = **144 CTAs, one persistent wave**. -Sweeping the occupied block count then shows two regimes: under ~18 blocks the -weights fit in L2 (1.15 us/block), above ~27 they stream and **us/block is FLAT -at 5.2-5.5 across a 2.4x range of work** -- constant bytes/second, the -bandwidth-limited signature, a **203-226 GB/s** plateau. Upstream's 210.7 sits -INSIDE that plateau (it runs at the kernel's achievable bandwidth); ours at -186.6 sits ~12% below. Note `ncu`'s SpeedOfLight memory percentage is unusable -here -- `dram__bytes.sum` reads `n/a` on GB10, so "Memory Throughput 11.14%" -excludes DRAM traffic and must not be read as latency-bound. What does NOT -survive is the per-unit-work division: with a fixed 144-CTA grid the "38.9 vs -40.6 blocks" are loop iterations, priced by the sweep at about +4.4%. Ours has -NOT yet run through the harness, which is the decisive next measurement. Weight +SM-clock-sensitive work. PAIRED profiling appeared to localise it to `marlin_moe_wna16::Marlin` (ours +249.22 ms vs upstream 230.39 ms over the same 1520 launches), and every +source-level explanation was eliminated -- kernel source, template +instantiation, grid, block size, shared memory, flags, scale layout, +alignment, residency, toolkit, arch -- with `ncu` and cuobjdump showing the +compiled kernels EQUIVALENT (94 registers, 3664 SASS instructions on +both). **That localisation is now REFUTED.** `scripts/marlin-moe-standalone.py` and +`benchmarks/marlin_moe_standalone.cpp` drive each engine's own kernel outside +its engine -- which is also what finally lets `ncu` attach to upstream, the +blocker recorded as impossible in both replay modes -- and at matched work the +two are indistinguishable: over 12 interleaved paired points ours +averages **5.3187 us/block against upstream's 5.3330**, ratio **0.9973**, sign flipping +between runs, inside one standard deviation either way. The in-situ 8.2% +therefore describes the RUNS, not the kernel, and so do the 12.8%-per-unit- +work and 186.6-vs-210.7 GB/s figures derived from it. What the harness does +establish is the kernel's shape: a persistent single wave (grid 144 = 48 SMs x +3 blocks, 32768 B shared against 102400 B/SM, 25% occupancy), a bandwidth- +limited plateau of 203-226 GB/s that BOTH engines reach, and a 4.6x swing +driven by how many DISTINCT EXPERTS a launch touches (1.15 us/block at 16, +weights in L2; ~5.3 above ~27, streaming). Blocks are not experts, so the +recorded 38.9-vs-40.6 block counts never settled which arm did more work; +measuring distinct experts per launch in situ is the next step, not another +kernel lever. One trap worth carrying: `dram__bytes.sum` reads `n/a` on GB10, +so `ncu`'s "Memory Throughput %" excludes DRAM traffic and must not be read as +a bandwidth utilisation. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under From cabaf175bca0946a673d77076ae4a585bd346035 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 01:39:36 +0000 Subject: [PATCH 05/11] perf(dspark): the in-situ denominator was WRONG -- experts, not blocks (#442) 6ae cleared the kernel by showing ours and upstream's Marlin are the same to 0.27% at matched work. This says what the in-situ 8.2% was actually measuring. Holding the block count roughly fixed while varying how many DISTINCT experts the routing touches: M blocks distinct us/call us/block 64 73 20 89.4 1.22 64 82 40 227.1 2.77 64 94 80 427.2 4.55 64 216 216 1121.2 5.19 128 137 20 149.9 1.09 128 146 40 220.0 1.51 128 255 250 1298.6 5.09 Read the M=128 rows: 137 -> 146 blocks is +6.6% of blocks and +46.7% of TIME, because distinct experts went 20 -> 40. Cost per DISTINCT EXPERT is flat at 5.2-5.7 us across the whole table (1.125 MiB each, ~205-225 GB/s, the same plateau as 6ad); cost per BLOCK varies 4.7x over the same rows. So time is distinct_experts x 1.125 MiB / ~215 GB/s, and block count is nearly irrelevant. Every in-situ comparison in 6x and 6y normalised by BLOCKS -- 38.9 ours against 40.6 upstream -- which is not the driver. Blocks are what num_tokens_past_padded reports; experts are what the kernel streams. With block_size_m = 8 and correlated spec-decode tokens one expert routinely spans several blocks, so the two come apart. Apply the model to the recorded in-situ launches: ours 164.0 us implies ~30 distinct experts, upstream 151.6 us implies ~28. About TWO distinct experts per launch reproduces the entire 8.2% with ZERO implementation difference. It also explains why both in-situ arms beat the standalone plateau per block (4.21 and 3.73 against ~5.3) -- in situ several blocks share an expert, so fewer bytes are streamed per block. That was never a sign of anything being wrong. Consequences. The 8.2% "kernel gap" is an artefact of comparing two different routing draws and should not be quoted again. Any future MoE comparison must control distinct experts per launch or force both arms onto an identical token stream, because block counts compare the wrong quantity. VT_MOE_PAD_STATS counts padded tokens and blocks; it should count distinct experts too, since that is the number that predicts the time. What this does NOT do: it does not move the end-to-end ratio, which is wall-clock on matched prompts and token counts and stands at ~0.966. It removes the ATTRIBUTION of that residual. The open question is whether our token path systematically touches more experts per step -- a near-tie divergence consequence rather than a defect -- or whether the remaining wall-clock sits outside the MoE entirely. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 38 +++++++++++++++++++ .agents/specs/dspark-spec-decode.md | 58 +++++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 22 ++++++++--- 4 files changed, 113 insertions(+), 7 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index d5201a906..18ff3d525 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20473,3 +20473,41 @@ predates the perf-neutral C_tmp cap. Ours times with steady_clock over 80 iterations, upstream with CUDA events. Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: the in-situ denominator was WRONG -- experts, not blocks (2026-08-13) + +Holding blocks roughly fixed while varying DISTINCT experts (upstream arm, gate_up): + +| M | blocks | distinct | us/call | us/block | +|---|---|---|---|---| +| 64 | 73 | 20 | 89.4 | 1.22 | +| 64 | 82 | 40 | 227.1 | 2.77 | +| 64 | 94 | 80 | 427.2 | 4.55 | +| 64 | 216 | 216 | 1121.2 | 5.19 | +| 128 | 137 | 20 | 149.9 | 1.09 | +| 128 | 146 | 40 | 220.0 | 1.51 | +| 128 | 255 | 250 | 1298.6 | 5.09 | + +M=128: 137 -> 146 blocks is +6.6% blocks and +46.7% TIME, because distinct +experts doubled. Cost per DISTINCT EXPERT is flat at 5.2-5.7 us (1.125 MiB, +~205-225 GB/s); cost per BLOCK varies 4.7x. So time ~= distinct_experts x +1.125 MiB / ~215 GB/s and blocks are nearly irrelevant. + +Every in-situ comparison normalised by BLOCKS (38.9 ours vs 40.6 upstream), which +is not the driver. Applying the model to the recorded launches: ours 164.0 us +implies ~30 distinct experts, upstream 151.6 us implies ~28. About TWO distinct +experts per launch reproduces the entire 8.2% with zero implementation +difference, and the matched-work comparison already put the two kernels at +0.27%. + +It also explains both in-situ arms beating the standalone plateau per block +(4.21 and 3.73 vs ~5.3): in situ several blocks share an expert. + +RULE FOR FUTURE MoE COMPARISONS: control distinct experts per launch, or force +both arms onto an identical token stream. Block counts compare the wrong +quantity. VT_MOE_PAD_STATS should count distinct experts as well. + +Does NOT move the e2e ratio (~0.966, wall-clock on matched prompts); it removes +the attribution of its residual. + +Evidence: `dgx.casa:~/work/marlin442/`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 1e6f46de6..e941ae7ed 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1815,6 +1815,64 @@ with `steady_clock` around 80 iterations plus a final sync; upstream's arm uses CUDA events. Both amortise launch overhead over 80 calls, and the memset probe bounds that class of difference at noise. +## 6af. THE DENOMINATOR WAS WRONG: EXPERTS, NOT BLOCKS (2026-08-13) + +6ae cleared the kernel. This says what the in-situ 8.2% actually measured. + +Holding the block count roughly fixed while varying how many DISTINCT experts +the routing touches, upstream arm, `--arm gate_up`: + +| M | blocks | distinct experts | us/call | us/block | +|---|---|---|---|---| +| 64 | 73 | 20 | 89.4 | 1.22 | +| 64 | 82 | 40 | 227.1 | 2.77 | +| 64 | 94 | 80 | 427.2 | 4.55 | +| 64 | 216 | 216 | 1121.2 | 5.19 | +| 128 | 137 | 20 | 149.9 | 1.09 | +| 128 | 146 | 40 | 220.0 | 1.51 | +| 128 | 255 | 250 | 1298.6 | 5.09 | + +Read the M=128 rows: **137 -> 146 blocks is +6.6% of blocks, and time rises ++46.7%**, because distinct experts went 20 -> 40. Cost per DISTINCT EXPERT is +flat at **5.2-5.7 us** across the whole table (1.125 MiB each, ~205-225 GB/s, +the same plateau as 6ad). Cost per BLOCK varies 4.7x over the same rows. + +**So the model is `time ~= distinct_experts x 1.125 MiB / ~215 GB/s`, and block +count is nearly irrelevant.** Every in-situ comparison in 6x and 6y normalised by +BLOCKS -- 38.9 ours against 40.6 upstream -- which the data now shows is not the +driver. Blocks are what `num_tokens_past_padded` reports and experts are what the +kernel streams; with `block_size_m = 8` and correlated spec-decode tokens, one +expert routinely spans several blocks, so the two numbers come apart. + +Apply the model to the recorded in-situ launches: ours 164.0 us implies ~30 +distinct experts, upstream 151.6 us implies ~28. **A difference of about two +distinct experts per launch reproduces the entire 8.2% with ZERO implementation +difference**, and 6ae has already shown that at matched work the two kernels are +the same to 0.27%. + +That also explains why both in-situ arms beat the standalone plateau per block +(4.21 and 3.73 against ~5.3): in situ, several blocks share an expert, so fewer +bytes are streamed per block. It was never a sign of anything being wrong. + +**Consequences.** + +1. The 8.2% "kernel gap" is an artefact of comparing two different routing + draws. It is not evidence of an implementation defect, and it should not be + quoted again. +2. Any future MoE comparison must control DISTINCT EXPERTS PER LAUNCH, or force + both arms onto an identical token stream. Comparing block counts is comparing + the wrong quantity. +3. `VT_MOE_PAD_STATS` (qwen3_5.cpp) counts padded tokens and blocks. It should + count DISTINCT EXPERTS too; that is the number that predicts the time. + +**What this does NOT do.** It does not move the end-to-end ratio. That is +wall-clock on matched prompts and token counts, and it stands at ~0.966. What +changes is that its residual no longer has an attribution: the kernel is cleared +and the number that appeared to localise it was measuring the token path. The +open question is whether OUR token path systematically touches more experts per +step -- a near-tie divergence consequence rather than a defect -- or whether the +remaining wall-clock sits outside the MoE entirely. + ## 7. Evidence, authority, stop conditions - Evidence root: `dgx:~/work/vllm.cpp-dspark-/`, one `flock`, named tmux. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index c60dc99b7..ea295e584 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** over three within-session pairs. Marlin localisation **REFUTED**: at matched work ours is 0.9973x of upstream (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED**: matched-work 0.9973x; time tracks DISTINCT EXPERTS (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 26a447aa3..43cff4e85 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -523,12 +523,22 @@ establish is the kernel's shape: a persistent single wave (grid 144 = 48 SMs x 3 blocks, 32768 B shared against 102400 B/SM, 25% occupancy), a bandwidth- limited plateau of 203-226 GB/s that BOTH engines reach, and a 4.6x swing driven by how many DISTINCT EXPERTS a launch touches (1.15 us/block at 16, -weights in L2; ~5.3 above ~27, streaming). Blocks are not experts, so the -recorded 38.9-vs-40.6 block counts never settled which arm did more work; -measuring distinct experts per launch in situ is the next step, not another -kernel lever. One trap worth carrying: `dram__bytes.sum` reads `n/a` on GB10, -so `ncu`'s "Memory Throughput %" excludes DRAM traffic and must not be read as -a bandwidth utilisation. Weight +weights in L2; ~5.3 above ~27, streaming). Blocks are not experts, and holding blocks fixed while varying distinct +experts settles it: at M=128, going 137 to 146 blocks (+6.6%) while distinct +experts doubled cost +46.7% TIME, and cost per DISTINCT EXPERT is flat at +5.2-5.7 us across the table while cost per block varies 4.7x. Time is +distinct_experts x 1.125 MiB / ~215 GB/s and blocks are nearly irrelevant, so +every in-situ comparison normalised by the wrong quantity. Applying that model +to the recorded launches, ours 164.0 us implies ~30 distinct experts against +upstream's ~28: about TWO experts per launch reproduces the whole 8.2% with no +implementation difference. It also explains both arms beating the standalone +plateau per block (4.21 and 3.73 against ~5.3), since in situ several blocks +share an expert. Future MoE comparisons must control distinct experts per +launch, or force both arms onto one token stream. None of this moves the end- +to-end ~0.966x, which is wall-clock on matched prompts; it removes the +attribution of its residual. One trap worth carrying: dram__bytes.sum reads +n/a on GB10, so ncu's Memory Throughput % excludes DRAM traffic and is not a +bandwidth utilisation. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under From 115b630905eab0069a323d8ec6955148d48f1a7d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 02:07:59 +0000 Subject: [PATCH 06/11] fix(record): ours touches FEWER experts, not more -- 6af's inference was inverted (#442) The previous commit inferred from the distinct-experts model that our in-situ launches touch ~30 distinct experts against upstream's ~28, and that this reproduced the 8.2%. That is BACKWARDS, and the counts contradicting it were already in .agents/benchmark-record.md: the 2026-08-12 both-sides measurement recorded ours at 311.2 padded tokens / 38.9 blocks per call against upstream's 324.8 / 40.6. I inferred where the file already had a measurement. At M=9 the 72 (token, expert) pairs spread over ~39 experts at under 8 tokens each, so moe_align emits ONE block per expert and blocks and distinct experts COINCIDE at the decode shape. That measurement was therefore already counting experts, upstream touches MORE of them, and the routing explanation stays refuted exactly as 6y concluded. The distinction is still worth keeping, because blocks and experts come apart 4.7x at M=64/128 -- it simply does not bite at M=9. What survives is sharper than what it replaces: standalone, matched work ours == upstream to 0.27% in situ ours does LESS work (38.9 vs 40.6 expert-blocks) and takes MORE time (164.0 vs 151.6 us) A kernel identical in isolation cannot be slower in place because of its own code, so the deficit belongs to the CONTEXT, not the kernel and not the routing. Candidates in the order their evidence justifies: expert-weight residency in situ, where this repo has already measured 20-30% per GEMM for host/ATS-retagged decode weights and the standalone arm's fresh cudaMalloc cannot reproduce it; clock and power state across the two runs; and overlap with concurrent stream work. Measurement-base caveat now stated: in-situ per-launch times are summed profiler kernel durations, standalone are wall-clock over 80 iterations, and both in-situ arms beat the standalone plateau per unit work (4.21 and 3.73 against ~5.3), which may be partly that rather than physical. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 34 ++++++++++++++++++++++++++++ .agents/specs/dspark-spec-decode.md | 35 +++++++++++++++++++++-------- docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 24 +++++++++++++------- 4 files changed, 77 insertions(+), 18 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 18ff3d525..f22dd5ff1 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20511,3 +20511,37 @@ Does NOT move the e2e ratio (~0.966, wall-clock on matched prompts); it removes the attribution of its residual. Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: CORRECTION -- ours touches FEWER experts, not more (2026-08-13) + +The entry above inferred, from the distinct-experts model, that our in-situ +launches touch ~30 distinct experts against upstream's ~28 and that this +explained the 8.2%. That is BACKWARDS, and the counts contradicting it were +already in this file: the 2026-08-12 both-sides measurement recorded ours 311.2 +padded tokens / 38.9 blocks per call against upstream 324.8 / 40.6. + +At M=9 the 72 (token, expert) pairs spread over ~39 experts at under 8 tokens +each, so moe_align emits ONE block per expert and blocks and distinct experts +COINCIDE at this shape. That measurement was therefore already counting experts, +upstream touches MORE of them, and the routing explanation stays refuted exactly +as it concluded. Blocks and experts do come apart at larger batches (spec §6af +measures 4.7x divergence at M=64/128), which is why the distinction is worth +keeping, but it does not apply at the decode shape. + +What survives, and it is sharper than what it replaces: + + standalone, matched work ours == upstream to 0.27% + in situ ours does LESS work (38.9 vs 40.6 expert-blocks) + and takes MORE time (164.0 vs 151.6 us) + +A kernel identical in isolation cannot be slower in place because of its own +code, so the deficit belongs to the CONTEXT, not the kernel and not the routing. +Candidates in evidence order: expert-weight residency in situ (this repo has +measured 20-30% per-GEMM for host/ATS-retagged decode weights, and the +standalone arm's fresh cudaMalloc cannot reproduce that), clock/power state +across runs, and overlap with concurrent stream work. + +Measurement-base caveat: in-situ per-launch times are summed profiler kernel +durations, standalone are wall-clock over 80 iterations. Both in-situ arms beat +the standalone plateau per unit work (4.21 and 3.73 vs ~5.3), which may be partly +that difference rather than a physical one. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index e941ae7ed..240d707cd 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1844,15 +1844,32 @@ driver. Blocks are what `num_tokens_past_padded` reports and experts are what th kernel streams; with `block_size_m = 8` and correlated spec-decode tokens, one expert routinely spans several blocks, so the two numbers come apart. -Apply the model to the recorded in-situ launches: ours 164.0 us implies ~30 -distinct experts, upstream 151.6 us implies ~28. **A difference of about two -distinct experts per launch reproduces the entire 8.2% with ZERO implementation -difference**, and 6ae has already shown that at matched work the two kernels are -the same to 0.27%. - -That also explains why both in-situ arms beat the standalone plateau per block -(4.21 and 3.73 against ~5.3): in situ, several blocks share an expert, so fewer -bytes are streamed per block. It was never a sign of anything being wrong. +**Applying the model to the recorded in-situ launches requires care, and my +first pass got it backwards.** 6y already measured both sides for the same +prompt: ours 311.2 padded tokens / 38.9 blocks per call, upstream 324.8 / 40.6. +At M=9 those 72 (token, expert) pairs spread over ~39 experts at under 8 tokens +each, so `moe_align` emits ONE block per expert and **blocks and distinct +experts coincide at this shape** -- that measurement was already counting +experts. So upstream touches MORE distinct experts (40.6 vs 38.9) and is still +faster, exactly as 6y concluded. The routing explanation stays refuted; an +earlier draft of this section inferred ours touched ~30 against upstream's ~28 +by inverting the model, which the measured counts in this same file contradict. + +**Which leaves a sharp contradiction, and it is the useful output of this +section.** Standalone at matched work the two kernels are equal to 0.27% (6ae). +In situ ours does LESS work (38.9 against 40.6 expert-blocks) and takes MORE +time (164.0 us against 151.6). Both in-situ arms also beat the standalone +plateau per unit of work (4.21 and 3.73 us against ~5.3). A kernel that is +identical in isolation cannot be slower in place because of its own code, so the +deficit belongs to the CONTEXT the kernel runs in, not to the kernel and not to +the routing. Candidates, in the order their evidence justifies: expert-weight +residency in situ (this repo has already measured a 20-30% per-GEMM penalty for +host/ATS-retagged decode weights, and the standalone arm allocates fresh device +memory that cannot reproduce it), clock and power state across the two runs, and +overlap with concurrent work on other streams. Note also that the in-situ +per-launch times come from summed profiler kernel durations while the standalone +numbers are wall-clock over 80 iterations, so the two bases are not +interchangeable and the ~5.3-to-4.21 difference may be partly that. **Consequences.** diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index ea295e584..adeba95c7 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED**: matched-work 0.9973x; time tracks DISTINCT EXPERTS (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED**: matched-work 0.9973x, so the residual is context (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 43cff4e85..33eae1086 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -528,14 +528,22 @@ experts settles it: at M=128, going 137 to 146 blocks (+6.6%) while distinct experts doubled cost +46.7% TIME, and cost per DISTINCT EXPERT is flat at 5.2-5.7 us across the table while cost per block varies 4.7x. Time is distinct_experts x 1.125 MiB / ~215 GB/s and blocks are nearly irrelevant, so -every in-situ comparison normalised by the wrong quantity. Applying that model -to the recorded launches, ours 164.0 us implies ~30 distinct experts against -upstream's ~28: about TWO experts per launch reproduces the whole 8.2% with no -implementation difference. It also explains both arms beating the standalone -plateau per block (4.21 and 3.73 against ~5.3), since in situ several blocks -share an expert. Future MoE comparisons must control distinct experts per -launch, or force both arms onto one token stream. None of this moves the end- -to-end ~0.966x, which is wall-clock on matched prompts; it removes the +every in-situ comparison normalised by the wrong quantity. At the M=9 decode shape blocks and distinct experts COINCIDE (72 pairs over +~39 experts, under 8 each, one block per expert), so the both-sides count +already measured experts: ours 38.9 against upstream's 40.6. That leaves a +sharp contradiction rather than an explanation. Standalone at matched work the +kernels are equal to 0.27%; in situ ours does LESS work and takes MORE time +(164.0 us against 151.6). A kernel identical in isolation cannot be slower in +place because of its own code, so the deficit belongs to the CONTEXT, not the +kernel and not the routing. Candidates in evidence order: expert-weight +residency in situ, where this repo has already measured 20-30% per GEMM for +host/ATS-retagged decode weights and the standalone arm's fresh cudaMalloc +cannot reproduce it; clock and power state across runs; and overlap with +concurrent stream work. Measurement bases differ too, in-situ being summed +profiler durations against standalone wall-clock. Future MoE comparisons must +control distinct experts per launch or force both arms onto one token stream, +since blocks and experts diverge 4.7x at larger batches. None of this moves +the end-to-end ~0.966x, which is wall-clock on matched prompts; it removes the attribution of its residual. One trap worth carrying: dram__bytes.sum reads n/a on GB10, so ncu's Memory Throughput % excludes DRAM traffic and is not a bandwidth utilisation. Weight From b0fc9d8ded97ce2423b6a22d6d3dbbe2dfcb3525 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 02:14:28 +0000 Subject: [PATCH 07/11] fix(record): the GEMM mix, and the MODE HOLE under every in-situ ratio (#442) Two arithmetic corrections close this thread. THE MIX. The 1520 in-situ launches are 760 gate_up plus 760 down, and down's per-expert bytes are exactly HALF gate_up's: gate_up is 2N x K/2 weights plus 2N x K/16 scales = 1.1250 MiB, down is K x N/2 plus K x N/16 = 0.5625 MiB, mixed average 0.8438 MiB per expert-block. 6af compared a MIXED in-situ average against a gate_up-ONLY standalone plateau, which is the whole reason both arms appeared to beat it at 4.21 and 3.73 us against ~5.3. They did not. Redone with the right bytes: blocks MiB/launch us implied GB/s ours 38.9 32.8 164.0 209.9 upstream 40.6 34.3 151.6 236.9 Measured standalone gate_up plateau: 203-226 GB/s. Ours sits INSIDE it, upstream ABOVE it. We run this kernel at the bandwidth it achieves in isolation, and upstream gets something in place that the isolated kernel does not -- cache reuse across the gate_up/down pair is the first candidate, down's weights being half size so more of the working set persists. The framing inverts: on this evidence we are not slow here, upstream is unusually fast in situ. THE MODE HOLE. 6y warns, correctly, that "a work COUNT may be taken under different execution modes; a TIME may not" -- and the per-unit-work normalisation then does exactly what that forbids. The 38.9/40.6 counts were taken EAGER (ours VT_SPEC_DECODE_GRAPH=0, upstream enforce_eager, because neither probe survived capture and compile) while the 249.2/230.4 ms times came from the GRAPHED profile. Nothing establishes the graphed runs had the same blocks per launch as the eager ones, so every ratio dividing those times by those counts -- 4.21 vs 3.73 us/block, 12.8% per unit of work, 186.6 vs 210.7 GB/s -- has a denominator from a different execution mode than its numerator. CONSEQUENCE: the only like-for-like Marlin comparison in evidence is the standalone one of 6ae, and it says parity at 0.9973, inside one standard deviation. Closing the in-situ question needs blocks AND time from the SAME graphed run, which needs a probe that survives capture: a device-side counter the launcher increments, read once at the end, never a per-launch D2H sync. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 45 +++++++++++++++++++++++++++++ .agents/specs/dspark-spec-decode.md | 41 ++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 27 ++++++++++++----- 4 files changed, 106 insertions(+), 9 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index f22dd5ff1..df595c05a 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20545,3 +20545,48 @@ Measurement-base caveat: in-situ per-launch times are summed profiler kernel durations, standalone are wall-clock over 80 iterations. Both in-situ arms beat the standalone plateau per unit work (4.21 and 3.73 vs ~5.3), which may be partly that difference rather than a physical one. + +## SPEC-DSPARK / #442: the GEMM MIX dissolves the anomaly, and the in-situ normalisation has a MODE hole (2026-08-13) + +Two arithmetic corrections close out this thread. + +1. THE MIX. The 1520 in-situ launches are 760 gate_up plus 760 down, and down's + per-expert bytes are exactly HALF gate_up's: gate_up is 2N x K/2 weights plus + 2N x K/16 scales = 1.1250 MiB, down is K x N/2 plus K x N/16 = 0.5625 MiB, so + the mixed average is 0.8438 MiB per expert-block. Comparing a mixed in-situ + average against a gate_up-ONLY standalone plateau was apples to oranges, and + it is what made both arms appear to "beat" the plateau (4.21 and 3.73 us + against ~5.3). They did not. + + Redone with the right bytes: + + | | blocks | MiB/launch | us | implied GB/s | + |---|---|---|---|---| + | ours | 38.9 | 32.8 | 164.0 | **209.9** | + | upstream | 40.6 | 34.3 | 151.6 | **236.9** | + + Standalone gate_up plateau, measured: 203-226 GB/s. **Ours sits INSIDE it. + Upstream sits ABOVE it.** So we run this kernel at the bandwidth it achieves + in isolation, and upstream gets something in place that the isolated kernel + does not -- cache reuse across the gate_up/down pair (down's weights are half + size, so more of the working set can persist) is the first candidate. The + framing inverts: we are not slow here, upstream is unusually fast. + +2. THE MODE HOLE. This file already warns "a work COUNT may be taken under + different execution modes; a TIME may not", and then the per-unit-work + normalisation does exactly what that warns against: the 38.9/40.6 counts were + taken EAGER (ours VT_SPEC_DECODE_GRAPH=0, upstream enforce_eager, both + because the probes could not survive capture/compile) while the 249.2/230.4 ms + times came from the GRAPHED profile. Nothing establishes that the graphed runs + had the same blocks per launch as the eager ones, so every ratio built by + dividing those times by those counts -- 4.21 vs 3.73 us/block, 12.8% per unit + work, 186.6 vs 210.7 GB/s -- rests on a denominator measured in a different + execution mode than its numerator. + +CONSEQUENCE: the only like-for-like Marlin comparison anyone has is the +standalone one, and it says parity (0.9973, inside 1 sd). Closing the in-situ +question needs blocks AND time from the SAME graphed run, which needs a probe +that survives capture -- a device-side counter written by the kernel launcher, +read once at the end, never a per-launch D2H sync. + +Evidence: `dgx.casa:~/work/marlin442/`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 240d707cd..d1cbedd46 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1890,6 +1890,47 @@ open question is whether OUR token path systematically touches more experts per step -- a near-tie divergence consequence rather than a defect -- or whether the remaining wall-clock sits outside the MoE entirely. +## 6ag. THE MIX, AND THE MODE HOLE UNDER EVERY IN-SITU RATIO (2026-08-13) + +Two arithmetic corrections close this thread out. + +**The mix.** The 1520 in-situ launches are 760 gate_up plus 760 down, and down's +per-expert bytes are exactly HALF gate_up's -- gate_up 2N x K/2 weights plus +2N x K/16 scales = 1.1250 MiB, down K x N/2 plus K x N/16 = 0.5625 MiB, mixed +average 0.8438 MiB per expert-block. 6af compared a MIXED in-situ average against +a gate_up-ONLY standalone plateau, which is what made both arms look like they +beat it (4.21 and 3.73 us against ~5.3). They did not. Redone with the right +bytes: + +| | blocks | MiB / launch | us | implied GB/s | +|---|---|---|---|---| +| ours | 38.9 | 32.8 | 164.0 | **209.9** | +| upstream | 40.6 | 34.3 | 151.6 | **236.9** | + +Measured standalone gate_up plateau: 203-226 GB/s. **Ours sits INSIDE it, +upstream ABOVE it.** We run this kernel at the bandwidth it achieves in +isolation; upstream gets something in place the isolated kernel does not. First +candidate is cache reuse across the gate_up/down pair, down's weights being half +size so more of the working set can persist. The framing inverts: on this +evidence we are not slow, upstream is unusually fast in situ. + +**The mode hole.** 6y warns, correctly, that "a work COUNT may be taken under +different execution modes; a TIME may not" -- and then the per-unit-work +normalisation does the thing that warning forbids. The 38.9/40.6 counts were +taken EAGER (ours `VT_SPEC_DECODE_GRAPH=0`, upstream `enforce_eager`, both +because the probes could not survive capture and compile respectively) while the +249.2/230.4 ms times came from the GRAPHED profile. Nothing establishes the +graphed runs had the same blocks per launch as the eager ones. So every ratio +built by dividing those times by those counts -- 4.21 vs 3.73 us/block, 12.8% +per unit of work, 186.6 vs 210.7 GB/s -- has a denominator from a different +execution mode than its numerator. + +**Consequence.** The only like-for-like Marlin comparison in evidence is the +standalone one of 6ae, and it says parity: 0.9973, inside one standard +deviation. Closing the in-situ question needs blocks AND time from the SAME +graphed run, which needs a probe that survives capture: a device-side counter +the launcher increments, read once at the end, never a per-launch D2H sync. + ## 7. Evidence, authority, stop conditions - Evidence root: `dgx:~/work/vllm.cpp-dspark-/`, one `flock`, named tmux. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index adeba95c7..a479093c0 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED**: matched-work 0.9973x, so the residual is context (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED**: matched-work 0.9973x; ours is AT the kernel plateau (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 33eae1086..d55825b35 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -539,14 +539,25 @@ kernel and not the routing. Candidates in evidence order: expert-weight residency in situ, where this repo has already measured 20-30% per GEMM for host/ATS-retagged decode weights and the standalone arm's fresh cudaMalloc cannot reproduce it; clock and power state across runs; and overlap with -concurrent stream work. Measurement bases differ too, in-situ being summed -profiler durations against standalone wall-clock. Future MoE comparisons must -control distinct experts per launch or force both arms onto one token stream, -since blocks and experts diverge 4.7x at larger batches. None of this moves -the end-to-end ~0.966x, which is wall-clock on matched prompts; it removes the -attribution of its residual. One trap worth carrying: dram__bytes.sum reads -n/a on GB10, so ncu's Memory Throughput % excludes DRAM traffic and is not a -bandwidth utilisation. Weight +concurrent stream work. Two arithmetic corrections then close it out. The 1520 in-situ launches are +760 gate_up plus 760 down, and down's per-expert bytes are exactly half, so +the mixed average is 0.8438 MiB per expert-block; comparing that against a +gate_up-only standalone plateau is what made both arms look like they beat it. +Redone correctly, ours implies 209.9 GB/s and upstream 236.9 against a +measured 203-226 plateau, so OURS SITS INSIDE IT AND UPSTREAM ABOVE IT: we run +this kernel at the bandwidth it achieves in isolation and upstream gets +something in place that the isolated kernel does not, cache reuse across the +gate_up/down pair being the first candidate. Second, every in-situ per-unit- +work ratio has a mode hole: the 38.9/40.6 block counts were taken EAGER +because the probes could not survive capture, while the 249.2/230.4 ms times +came from the GRAPHED profile, so numerator and denominator are from different +execution modes. The only like-for-like Marlin comparison in evidence is +therefore the standalone one, and it says parity. Closing the in-situ question +needs blocks and time from the SAME graphed run, via a device-side counter +read once at the end rather than a per-launch D2H sync. None of this moves the +end-to-end ~0.966x, which is wall-clock on matched prompts. One trap worth +carrying: dram__bytes.sum reads n/a on GB10, so ncu's Memory Throughput % +excludes DRAM traffic and is not a bandwidth utilisation. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under From 1fa4c2de4c955294dbde86d76a7d974e74f1a4e1 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 02:24:55 +0000 Subject: [PATCH 08/11] perf(dspark): blocks_per_sm is a DEAD lever, and the box cannot currently resolve 3% (#442) moe_wna16_marlin_gemm's last parameter is blocks_per_sm, and both engines pass -1 (auto), which yields 32768 B of shared memory, 3 blocks/SM and 25% occupancy. With the kernel running at ~78% of peak bandwidth, forcing a higher value was the one settable occupancy lever left. It does not work. A TRAP FIRST, because it nearly produced a false result. The unseeded sweep read bps -1/1/4 slow and 2/3/5/6/8 fast, an apparent 8.7% win. The routing was redrawn per run, so `distinct` moved between 35 and 41, and since time tracks distinct experts (spec 6af) the sweep was measuring the DRAW. Any MoE comparison that lets routing vary between arms measures the draw and not the change. The harness now seeds torch explicitly for that reason. SEEDED, identical 33 distinct / 33 blocks everywhere, 4 interleaved reps at 120 iterations: bps runs (us) mean excl. outliers -1 auto 191.3, 367.8, 177.2 184.2 3 181.1, 171.1, 177.6, 353.0 176.6 4 177.7, 178.6, 179.8, 180.3 179.1 5 181.7, 184.2, 179.4 181.7 Between-configuration spread is 4.3%; the WITHIN-configuration spread is comparable (bps=3 alone ranges 171-181). No reliable effect. The lever is dead. THE MORE USEFUL FINDING is the environment. Two of fourteen runs returned ~2x (367.8 and 353.0) and two produced no output at all. A box emitting 2x outliers cannot resolve a 3.4% question, and this one is currently contended: a concurrent session has been building and rebuilding a ~79G tree on the same hardware. No parity measurement taken in this window should be trusted in either direction. The paired numbers earlier in the record predate that contention and were interleaved, which is what makes them survivable. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 37 ++++++++++++++++++++++++++++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 12 ++++++++--- scripts/marlin-moe-standalone.py | 22 ++++++++++++++----- 4 files changed, 64 insertions(+), 9 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index df595c05a..1c1ba947a 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20590,3 +20590,40 @@ that survives capture -- a device-side counter written by the kernel launcher, read once at the end, never a per-launch D2H sync. Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: blocks_per_sm is a DEAD lever, and the box is currently unfit to measure 3% (2026-08-13) + +`moe_wna16_marlin_gemm`'s last parameter is `blocks_per_sm`; both engines pass +-1 (auto), which yields 32768 B of shared memory, 3 blocks/SM and 25% occupancy. +Since the kernel runs at ~78% of peak bandwidth, forcing a higher value looked +like a best-in-class lever. It is not. + +FIRST SWEEP, UNSEEDED, and it is a trap worth recording: bps -1/1/4 read slow and +2/3/5/6/8 read fast, apparently 8.7%. The routing was redrawn per run, so +`distinct` moved 35-41 and the sweep was measuring the DRAW, not the parameter. +Time tracks distinct experts, so any comparison that lets routing vary measures +nothing else. + +SEEDED (torch.manual_seed, identical 33 distinct / 33 blocks everywhere), 4 +interleaved reps at 120 iters: + +| bps | runs (us) | mean excl. outliers | +|---|---|---| +| -1 (auto) | 191.3, 367.8, 177.2 | 184.2 | +| 3 | 181.1, 171.1, 177.6, 353.0 | 176.6 | +| 4 | 177.7, 178.6, 179.8, 180.3 | 179.1 | +| 5 | 181.7, 184.2, 179.4 | 181.7 | + +Between-config spread 4.3%; WITHIN-config spread comparable (bps=3 alone ranges +171-181). No reliable effect. The lever is DEAD and the seeded single-shot 8.7% +was noise. + +ENVIRONMENT, and this is the more useful finding: two of fourteen runs returned +~2x (367.8 and 353.0) and two runs produced no output at all. A box that emits +2x outliers cannot resolve a 3.4% question, and this one is currently contended +-- a concurrent session has been building and rebuilding a ~79G tree on the same +hardware. No parity measurement taken in this window should be trusted, in +either direction. Earlier paired numbers in this file were taken before that +contention began and were interleaved, which is what makes them survivable. + +Evidence: `dgx.casa:~/work/marlin442/`. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index a479093c0..b66c0d8b5 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED**: matched-work 0.9973x; ours is AT the kernel plateau (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED** (matched-work 0.9973x); blocks_per_sm lever DEAD (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index d55825b35..37174919b 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -555,9 +555,15 @@ execution modes. The only like-for-like Marlin comparison in evidence is therefore the standalone one, and it says parity. Closing the in-situ question needs blocks and time from the SAME graphed run, via a device-side counter read once at the end rather than a per-launch D2H sync. None of this moves the -end-to-end ~0.966x, which is wall-clock on matched prompts. One trap worth -carrying: dram__bytes.sum reads n/a on GB10, so ncu's Memory Throughput % -excludes DRAM traffic and is not a bandwidth utilisation. Weight +end-to-end ~0.966x, which is wall-clock on matched prompts. The kernel's one +settable occupancy knob, blocks_per_sm, was then swept and is DEAD: with +routing SEEDED so every configuration sees identical work, the between- +configuration spread (4.3%) is no larger than the within-configuration spread, +and an apparent 8.7% win in an unseeded first pass was the routing draw moving +rather than the parameter. Two traps worth carrying: dram__bytes.sum reads n/a +on GB10, so ncu's Memory Throughput % excludes DRAM traffic and is not a +bandwidth utilisation; and any MoE comparison that lets routing vary between +arms measures the draw, not the change. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under diff --git a/scripts/marlin-moe-standalone.py b/scripts/marlin-moe-standalone.py index 8e5e33f94..c31aaf374 100755 --- a/scripts/marlin-moe-standalone.py +++ b/scripts/marlin-moe-standalone.py @@ -24,6 +24,11 @@ def main() -> int: ap.add_argument("--iters", type=int, default=200) ap.add_argument("--warmup", type=int, default=20) ap.add_argument("--arm", choices=["gate_up", "down", "both"], default="both") + ap.add_argument("--bps", type=int, default=-1, + help="blocks_per_sm passed to the kernel (-1 = auto)") + ap.add_argument("--experts", type=int, default=0, + help="draw routing from only this many distinct experts " + "(0 = all E); controls occupied block count") args = ap.parse_args() import vllm @@ -91,10 +96,15 @@ def build(size_n: int, size_k: int): w2, w2s, w2g = build(K, N) # down: size_n = 2048, size_k = 512 workspace = marlin_make_workspace_new(dev, 4) - topk_ids = torch.randint(0, E, (M, TOPK), dtype=torch.int32, device=dev) + pool = args.experts if args.experts > 0 else E + # SEEDED: routing dominates the time, so every configuration under + # comparison must see the IDENTICAL draw or the sweep measures the draw. + torch.manual_seed(20260813) + topk_ids = torch.randint(0, pool, (M, TOPK), dtype=torch.int32, device=dev) topk_w = torch.rand((M, TOPK), dtype=torch.float32, device=dev) sorted_ids, expert_ids, num_past = moe_align_block_size( topk_ids, BLOCK_M, E, None) + distinct = int(torch.unique(topk_ids).numel()) a1 = torch.randn((M, K), dtype=dtype, device=dev) c1 = torch.empty((M * TOPK, 2 * N), dtype=dtype, device=dev) @@ -108,14 +118,14 @@ def gate_up(): a1, c1, w1, None, w1s, None, w1g, None, None, None, workspace, sorted_ids, expert_ids, num_past, topk_w, BLOCK_M, TOPK, False, quant_type.id, M, 2 * N, K, - True, False, True, False, -1, -1, -1) + True, False, True, False, -1, -1, args.bps) def down(): return ops.moe_wna16_marlin_gemm( a2, c2, w2, None, w2s, None, w2g, None, None, None, workspace, sorted_ids, expert_ids, num_past, topk_w, BLOCK_M, 1, True, quant_type.id, M * TOPK, K, N, - True, False, True, False, -1, -1, -1) + True, False, True, False, -1, -1, args.bps) arms = {"gate_up": gate_up, "down": down} if args.arm != "both": @@ -132,8 +142,10 @@ def down(): en.record() torch.cuda.synchronize() us = st.elapsed_time(en) * 1000.0 / args.iters - print(f"UPSTREAM {name} M={M} blocks_past={int(num_past.item())} " - f"us_per_call={us:.3f}", flush=True) + blocks = int(num_past.item()) // BLOCK_M + print(f"UPSTREAM {name} bps={args.bps} M={M} pool={pool} distinct={distinct} blocks={blocks} " + f"us_per_call={us:.3f} us_per_block={us/max(blocks,1):.4f}", + flush=True) print("DONE", flush=True) return 0 From 69e9e7a62f6d871c370558d51f37bc6de5fc3b51 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 02:37:14 +0000 Subject: [PATCH 09/11] fix(record): the standalone Marlin runs were taken UNLOCKED -- wrong lock file (#442) Every standalone run in this branch was wrapped in `flock /tmp/gpu.lock`. That is the wrong file. This box's GPU lock is `$HOME/gpu.lock` -- it is what final_pair.sh takes and what other sessions hold -- so /tmp/gpu.lock coordinated with nothing and those runs executed unserialised against whatever else was on the GPU. At least one concurrent test_qwen27_spec_decode_concurrent (15.9 GB RSS) overlapped them. That is the likeliest source of the ~2x outliers (367.8 and 353.0 us) in the blocks_per_sm sweep, and it was MINE. An earlier note in this session blamed a concurrent session for not locking; the reverse is true -- they took the correct lock and I did not. WHAT SURVIVES: the ours-vs-upstream comparisons were INTERLEAVED within a single run, so contention lands on both arms alike and the RATIO is exactly what interleaving protects. 0.9973 with the sign flipping between reps still reads as indistinguishable, with wider error bars than the quoted sd suggests. WHAT DOES NOT: absolute us/call and us/block, including the 203-226 GB/s plateau, are upper bounds rather than exact, since an unlocked box can only make them slower. That plateau needs re-taking under $HOME/gpu.lock before it is quoted as the kernel's achievable bandwidth. The two-regime SHAPE (L2-resident under ~18 blocks, streaming above ~27) and the distinct-experts scaling are robust to a uniform slowdown and do not need re-taking. Both harness arms now carry the rule in their header, including that nvidia-smi showing no compute apps does NOT mean the GPU is unreserved -- a holder may be between phases -- so `fuser -v $HOME/gpu.lock` is the check. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 30 ++++++++++++++++++++++++++++ benchmarks/marlin_moe_standalone.cpp | 6 ++++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 14 +++++++++---- scripts/marlin-moe-standalone.py | 6 ++++++ 5 files changed, 53 insertions(+), 5 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 1c1ba947a..300e4500f 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20627,3 +20627,33 @@ either direction. Earlier paired numbers in this file were taken before that contention began and were interleaved, which is what makes them survivable. Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: CORRECTION -- the standalone runs were taken UNLOCKED (2026-08-13) + +Every standalone Marlin run in the entries above was wrapped in +`flock /tmp/gpu.lock`. That is the WRONG FILE. This box's GPU lock is +`$HOME/gpu.lock` -- it is what `final_pair.sh` takes and what other sessions +hold. `/tmp/gpu.lock` coordinates with nothing, so those runs executed +unserialised against whatever else was on the GPU, and at least one concurrent +`test_qwen27_spec_decode_concurrent` (15.9 GB RSS) overlapped them. + +That is the likeliest source of the ~2x outliers (367.8 and 353.0 us) in the +blocks_per_sm sweep, and it was mine, not the other session's. An earlier note +in this session blamed them for not locking; the reverse is true. + +WHAT SURVIVES. The ours-vs-upstream comparisons were INTERLEAVED within a single +run (ours, upstream, ours, upstream), so contention lands on both arms alike and +the RATIO is the quantity interleaving protects. 0.9973 with the sign flipping +between reps still stands as "indistinguishable", though the error bars are wider +than the quoted sd suggests. + +WHAT DOES NOT. Absolute us/call and us/block from those runs, including the +203-226 GB/s plateau, are upper-bounded rather than exact -- an unlocked box can +only make them slower. The plateau should be re-taken under `$HOME/gpu.lock` +before it is quoted as the kernel's achievable bandwidth. The two-regime SHAPE +(L2-resident under ~18 blocks, streaming above ~27) and the distinct-experts +scaling are robust to a uniform slowdown and do not need re-taking. + +RULE: take `$HOME/gpu.lock`, not `/tmp/gpu.lock`. Check `fuser -v ~/gpu.lock` +before assuming the GPU is free; `nvidia-smi` showing no compute apps does NOT +mean it is unreserved, since a holder may be between phases. diff --git a/benchmarks/marlin_moe_standalone.cpp b/benchmarks/marlin_moe_standalone.cpp index 0af9f95c4..4872a7988 100644 --- a/benchmarks/marlin_moe_standalone.cpp +++ b/benchmarks/marlin_moe_standalone.cpp @@ -6,6 +6,12 @@ // us/call and us/block so our plateau can be laid against upstream's 5.2-5.5. // // Not a test: no assertions, no goldens. It measures the kernel only. +// +// RUN IT UNDER THE BOX LOCK: `flock $HOME/gpu.lock ...`, NOT /tmp/gpu.lock, +// which coordinates with nothing. `nvidia-smi` showing no compute apps does +// not mean the GPU is unreserved, so check `fuser -v $HOME/gpu.lock` first. +// Absolute timings taken unlocked are upper bounds; only interleaved RATIOS +// survive contention. #include #include diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index b66c0d8b5..60b0b023c 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED** (matched-work 0.9973x); blocks_per_sm lever DEAD (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED** (interleaved matched-work 0.9973x) (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index 37174919b..ce11e6fb3 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -560,10 +560,16 @@ settable occupancy knob, blocks_per_sm, was then swept and is DEAD: with routing SEEDED so every configuration sees identical work, the between- configuration spread (4.3%) is no larger than the within-configuration spread, and an apparent 8.7% win in an unseeded first pass was the routing draw moving -rather than the parameter. Two traps worth carrying: dram__bytes.sum reads n/a -on GB10, so ncu's Memory Throughput % excludes DRAM traffic and is not a -bandwidth utilisation; and any MoE comparison that lets routing vary between -arms measures the draw, not the change. Weight +rather than the parameter. Three traps worth carrying. dram__bytes.sum reads n/a on GB10, so ncu's Memory +Throughput % excludes DRAM traffic and is not a bandwidth utilisation. Any MoE +comparison that lets routing vary between arms measures the draw, not the +change. And this box's GPU lock is $HOME/gpu.lock, not /tmp/gpu.lock: the +standalone runs above took the wrong file and so ran UNLOCKED against a +concurrent GPU test, which makes their absolute timings (including the 203-226 +GB/s plateau) upper bounds needing a re-take, while the INTERLEAVED ratio, the +two-regime shape and the distinct-experts scaling survive because contention +lands on both arms alike. nvidia-smi showing no compute apps does not mean the +GPU is unreserved; fuser -v $HOME/gpu.lock is the check. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under diff --git a/scripts/marlin-moe-standalone.py b/scripts/marlin-moe-standalone.py index c31aaf374..a2de54b3d 100755 --- a/scripts/marlin-moe-standalone.py +++ b/scripts/marlin-moe-standalone.py @@ -9,6 +9,12 @@ Shapes come from nvidia/Qwen3.6-35B-A3B-NVFP4: hidden 2048, moe_intermediate 512, E=256, top_k=8. Marlin's runtime is data-independent, so random weights measure the same kernel the model does. + +RUN IT UNDER THE BOX LOCK: `flock $HOME/gpu.lock ...`, NOT /tmp/gpu.lock, which +coordinates with nothing. `nvidia-smi` reporting no compute apps does not mean +the GPU is unreserved -- a holder may be between phases -- so check +`fuser -v $HOME/gpu.lock` first. Absolute timings taken unlocked are upper +bounds; only interleaved RATIOS survive contention. """ import argparse From d7d67be3e992c26df889cfb0760cc7ff31af776d Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 06:21:23 +0000 Subject: [PATCH 10/11] fix(record): review FAIL repairs -- per-expert cost is NOT flat, and the bound direction was inverted (#442) A fresh scoped review returned FAIL on this branch with two blocking findings. Both are real, both are repaired here, and neither needed the GPU -- which is just as well, since dgx.casa is down. 1. "Cost per DISTINCT EXPERT is flat at 5.2-5.7 us" is FALSE. Recomputing us/call / distinct from the section's OWN table gives 4.470, 5.678, 5.340, 5.191, 7.495, 5.500, 5.194 -- a 4.47 to 7.50 range. The two distinct=20 rows are the tell: identical expert counts, 89.4 vs 149.9 us, blocks 73 vs 137. There, time tracks BLOCKS -- precisely what a flat-per-expert model calls irrelevant. The honest model has the same two regimes as the L2 finding one section earlier. Weights fitting L2 (low distinct) are not re-streamed, so cost is per-block work and time tracks BLOCKS. Weights not fitting (distinct >= ~40 here) make cost streaming-bound, so time tracks DISTINCT EXPERTS at 5.2-5.7 us each. time ~= distinct x 1.125 MiB / ~215 GB/s holds ONLY in the second regime: at distinct=20 it predicts 109.7 us against 89.4 and 149.9 measured, off by -19% and +37%. So the rule this spec promulgated -- "control distinct experts per launch" -- was INCOMPLETE. Control BOTH distinct experts and blocks, or state the regime. 2. The lock caveat reached the record and STATUS but NOT the spec, which is the file a fresh implementer is told to read, and it gave the bound the WRONG DIRECTION. Contention inflates TIME, so a bandwidth computed as bytes/time is a LOWER bound on what the kernel achieves, not an upper one. A re-take under $HOME/gpu.lock can only move the plateau UP, which would put our in-situ 209.9 GB/s BELOW it and REVERSE 6ag's inversion. "We are not slow, upstream is unusually fast" is the least favourable reading to us that the data admits and the one most likely to change on a clean re-take; it is now marked provisional in the spec itself. Also repaired: the plateau range mixed bands (over the rows actually flat it is 212.7-225.8 GB/s, and the 203 endpoint is the 27-block row the same paragraph excludes -- under the self-consistent range "ours sits INSIDE it" is FALSE); "at matched blocks" overstated the harness, whose two arms draw routing from independent RNG streams and therefore occupy different block counts at the same pool, so blocks are NORMALISED rather than matched and neither arm can yet take an external routing tensor; the blocks_per_sm entry credited the 8.7% to the seeded pass when it came from the unseeded one; and --iters defaults differed between the arms (python 200, C++ 80) while the prose asserted 80 for both. benchmarks/marlin_moe_standalone.cpp is not wired into any build target, so nothing compiles it and it will rot against vt::MoeGroupedGemmNvfp4Marlin's signature. Recorded as owed in the file header and the record rather than quietly left. WHAT SURVIVES: 6ae's interleaved 0.9973 -- the kernel is not the gap -- because interleaving is exactly what protects a ratio against contention AND routing draw. What does not survive as stated is 6ag's inversion. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 60 ++++++++++++++++++++++++++- .agents/specs/dspark-spec-decode.md | 62 +++++++++++++++++++++------- benchmarks/marlin_moe_standalone.cpp | 11 +++++ docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 26 +++++++----- scripts/marlin-moe-standalone.py | 2 +- 6 files changed, 135 insertions(+), 28 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 300e4500f..784e7f738 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20615,8 +20615,9 @@ interleaved reps at 120 iters: | 5 | 181.7, 184.2, 179.4 | 181.7 | Between-config spread 4.3%; WITHIN-config spread comparable (bps=3 alone ranges -171-181). No reliable effect. The lever is DEAD and the seeded single-shot 8.7% -was noise. +171-181). No reliable effect. The lever is DEAD, and the 8.7% +came from the UNSEEDED first pass, i.e. the routing draw moving rather than the +parameter. ENVIRONMENT, and this is the more useful finding: two of fourteen runs returned ~2x (367.8 and 353.0) and two runs produced no output at all. A box that emits @@ -20657,3 +20658,58 @@ scaling are robust to a uniform slowdown and do not need re-taking. RULE: take `$HOME/gpu.lock`, not `/tmp/gpu.lock`. Check `fuser -v ~/gpu.lock` before assuming the GPU is free; `nvidia-smi` showing no compute apps does NOT mean it is unreserved, since a holder may be between phases. + +## SPEC-DSPARK / #442: CORRECTIONS from fresh review -- per-expert cost is NOT flat, and the bound direction was wrong (2026-08-13) + +A fresh review of the branch found two blocking errors in the entries above. +Both are recorded here rather than edited away, since this log is append-only. + +1. "Cost per DISTINCT EXPERT is flat at 5.2-5.7 us" is FALSE. Recomputing + us/call / distinct from the table's own numbers gives 4.470, 5.678, 5.340, + 5.191, 7.495, 5.500, 5.194 -- a range of 4.47 to 7.50. The two distinct=20 + rows are the tell: identical expert counts, 89.4 vs 149.9 us, blocks 73 vs + 137. THERE, TIME TRACKS BLOCKS, which is what a flat-per-expert model calls + irrelevant. + + The honest model has the same two regimes as the L2 finding. Weights fitting + L2 (low distinct) => not re-streamed => cost set by per-block work => time + tracks BLOCKS. Weights not fitting (distinct >= ~40 here) => cost set by + streaming => time tracks DISTINCT EXPERTS at 5.2-5.7 us each. + `time ~= distinct x 1.125 MiB / ~215 GB/s` applies ONLY in the second regime; + at distinct=20 it predicts 109.7 us against 89.4 and 149.9 measured, off by + -19% and +37%. + + The rule this file stated -- "control distinct experts per launch" -- is + therefore INCOMPLETE. Control BOTH distinct experts AND blocks, or state + which regime the comparison is in. + +2. The lock caveat gave the bound the WRONG DIRECTION. It called the plateau an + upper bound. Contention inflates TIME, so a bandwidth computed as bytes/time + is a LOWER BOUND on what the kernel achieves. A re-take under + `$HOME/gpu.lock` can only move the plateau UP, which would move our in-situ + 209.9 GB/s BELOW it and REVERSE the "ours is inside it, upstream above it" + inversion. That inversion is the least favourable reading to us the data + admits and the one most likely to change on a clean re-take. + +3. The plateau range 203-226 GB/s mixes bands: over the rows actually flat + (>= 38 blocks) it is 212.7-225.8, and the 203 endpoint is the 27-block row at + 5.81 us/block, which the same paragraph places OUTSIDE the flat band. Under + the narrower, self-consistent range, "ours (209.9) sits INSIDE it" is FALSE. + +4. The harness arms draw routing from independent RNG streams (python + torch.manual_seed, C++ mt19937(1234)), so at the same pool they occupy + different block counts (48: 39 vs 38; 128: 54 vs 58). "At matched blocks" was + inaccurate -- blocks are NORMALISED, not matched. Neither arm accepts an + external routing tensor, so the harness cannot yet satisfy the "force both + arms onto one token stream" rule stated one section later. The 0.9973 ratio + is probably still sound at M=9, but its error bars carry an undisclosed draw + term on top of the contention term. + +5. `benchmarks/marlin_moe_standalone.cpp` is not wired into any build target, so + nothing compiles it and it will rot against + `vt::MoeGroupedGemmNvfp4Marlin`'s signature. Tracked as owed. + +NET EFFECT ON CONCLUSIONS: 6ae's interleaved 0.9973 (the kernel is not the gap) +SURVIVES, since interleaving is what protects a ratio against both contention +and draw. 6ag's "we are not slow, upstream is unusually fast" DOES NOT survive +as stated and is now marked provisional pending a re-take under the right lock. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index d1cbedd46..4f0177d96 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1722,7 +1722,9 @@ stream, and **us/block is FLAT at 5.2-5.5 across a 2.4x range of work** -- a constant bytes-per-second, which is the bandwidth-limited signature. **So 6z was right and this section's first draft was wrong.** The kernel IS -memory-bound; the standalone plateau is **203-226 GB/s**, and 6z's derived +memory-bound; the standalone plateau is **212.7-225.8 GB/s** over the rows that +are actually flat (>= 38 blocks) -- the widely quoted 203 endpoint is the +27-block row, which sits outside the flat band -- and 6z's derived in-situ numbers land exactly on it: upstream's **210.7 GB/s is INSIDE the plateau**, i.e. upstream runs at this kernel's achievable bandwidth, while our **186.6 GB/s is ~12% BELOW** it. An independent standalone measurement of the @@ -1792,7 +1794,15 @@ workspace memset that our arm pays and upstream's does not was isolated with **So 6x's localisation does not survive.** Its "the SAME kernel, the SAME 1520 launches, ours 249.22 ms vs upstream 230.39 ms, 8.2% slower inside one kernel" does NOT reproduce when the same kernel is driven with matched work on the same -box. At matched blocks the two are indistinguishable. Correspondingly, the +box. Normalised by blocks the two are indistinguishable -- and note NORMALISED, not +matched: the two arms draw routing from independent RNG streams (python +`torch.manual_seed`, C++ `mt19937(1234)`), so at the same pool they occupy +different block counts (pool 48: 39 vs 38; pool 128: 54 vs 58). Neither harness +accepts an externally supplied routing tensor, so it cannot yet satisfy the +"force both arms onto one token stream" rule this spec states. At M=9, blocks +track distinct experts and us/block is flat in the streaming regime, so the +ratio is probably sound -- but its error bars carry an undisclosed draw term on +top of the contention term. Correspondingly, the "12.8% slower per unit of work" and the 186.6-vs-210.7 GB/s reading it produced describe the in-situ RUNS, not the kernel: both engines reach the same 203-226 GB/s plateau when asked to do the same thing. @@ -1833,16 +1843,26 @@ the routing touches, upstream arm, `--arm gate_up`: | 128 | 255 | 250 | 1298.6 | 5.09 | Read the M=128 rows: **137 -> 146 blocks is +6.6% of blocks, and time rises -+46.7%**, because distinct experts went 20 -> 40. Cost per DISTINCT EXPERT is -flat at **5.2-5.7 us** across the whole table (1.125 MiB each, ~205-225 GB/s, -the same plateau as 6ad). Cost per BLOCK varies 4.7x over the same rows. - -**So the model is `time ~= distinct_experts x 1.125 MiB / ~215 GB/s`, and block -count is nearly irrelevant.** Every in-situ comparison in 6x and 6y normalised by -BLOCKS -- 38.9 ours against 40.6 upstream -- which the data now shows is not the -driver. Blocks are what `num_tokens_past_padded` reports and experts are what the -kernel streams; with `block_size_m = 8` and correlated spec-decode tokens, one -expert routinely spans several blocks, so the two numbers come apart. ++46.7%**, because distinct experts went 20 -> 40. + +**But the per-expert cost is NOT flat, and an earlier draft of this section said +it was.** Dividing us/call by distinct gives 4.470, 5.678, 5.340, 5.191, 7.495, +5.500, 5.194 -- a **4.47 to 7.50** range, not 5.2-5.7. The two `distinct=20` +rows are the tell: same expert count, 89.4 vs 149.9 us, while blocks differ +73 vs 137. There, time tracks BLOCKS, which is exactly what a flat-per-expert +model calls irrelevant. + +**So the honest model has two regimes, and it is the same two regimes as 6ad.** +When the touched weights fit L2 (low distinct), the weights are not re-streamed +and cost is set by the per-block work, so time tracks BLOCKS. When they do not +(distinct >= ~40 here), cost is set by streaming and time tracks DISTINCT +EXPERTS at 5.2-5.7 us each. `time ~= distinct x 1.125 MiB / ~215 GB/s` applies +ONLY in the second regime; at distinct=20 it predicts 109.7 us against measured +89.4 and 149.9, off by -19% and +37%. + +The rule that follows is therefore **control BOTH**: a MoE comparison must match +distinct experts AND blocks, or state which regime it is in. Matching one alone +is what makes two runs look comparable when they are not. **Applying the model to the recorded in-situ launches requires care, and my first pass got it backwards.** 6y already measured both sides for the same @@ -1907,8 +1927,22 @@ bytes: | ours | 38.9 | 32.8 | 164.0 | **209.9** | | upstream | 40.6 | 34.3 | 151.6 | **236.9** | -Measured standalone gate_up plateau: 203-226 GB/s. **Ours sits INSIDE it, -upstream ABOVE it.** We run this kernel at the bandwidth it achieves in +Measured standalone gate_up plateau: 212.7-225.8 GB/s over the flat rows. +Ours appears INSIDE it and upstream ABOVE it -- **but read the next paragraph +before using that.** + +**THE LOCK CAVEAT, WHICH APPLIES TO EVERY ABSOLUTE NUMBER IN 6ad THROUGH 6ag.** +Those standalone runs took `flock /tmp/gpu.lock`. This box's GPU lock is +`$HOME/gpu.lock`, so they ran UNSERIALISED against concurrent GPU work. Timings +taken under contention are inflated, and a bandwidth computed as bytes/time is +therefore a **LOWER BOUND** on what the kernel achieves, not an upper one. A +re-take under the correct lock can only move the plateau UP -- which would move +our in-situ 209.9 GB/s BELOW it and **reverse the inversion below**. The +"we are not slow, upstream is unusually fast" reading is the LEAST favourable +interpretation to us that the data admits, and it is the one most likely to +change on a clean re-take. Do not build on it until the plateau is re-measured +under `$HOME/gpu.lock`. What survives contention is the INTERLEAVED ratio of +6ae, because contention lands on both arms alike. We run this kernel at the bandwidth it achieves in isolation; upstream gets something in place the isolated kernel does not. First candidate is cache reuse across the gate_up/down pair, down's weights being half size so more of the working set can persist. The framing inverts: on this diff --git a/benchmarks/marlin_moe_standalone.cpp b/benchmarks/marlin_moe_standalone.cpp index 4872a7988..ce29f6449 100644 --- a/benchmarks/marlin_moe_standalone.cpp +++ b/benchmarks/marlin_moe_standalone.cpp @@ -12,6 +12,17 @@ // not mean the GPU is unreserved, so check `fuser -v $HOME/gpu.lock` first. // Absolute timings taken unlocked are upper bounds; only interleaved RATIOS // survive contention. +// +// NOT WIRED INTO ANY BUILD TARGET (#442). Nothing compiles this file, so it +// carries no -Werror and no CI, and it will rot against +// vt::MoeGroupedGemmNvfp4Marlin's signature. The recorded measurements were +// taken from an out-of-tree build. Wiring it into examples/CMakeLists.txt the +// way benchmarks/vulkan_gemm_ab.cpp is wired is owed. +// +// Its routing RNG is a DIFFERENT stream from the python arm's, so the two +// arms occupy different block counts at the same --experts pool. Comparisons +// between them are NORMALISED by blocks, not matched on them; neither arm can +// yet take an externally supplied routing tensor. #include #include diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 60b0b023c..34858f389 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED** (interleaved matched-work 0.9973x) (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle. NOT parity: **~0.966x +/- 0.01** (3 within-session pairs). Marlin cause **REFUTED** (interleaved matched-work 0.9973x); absolutes owe a re-take (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index ce11e6fb3..5a3fe1b30 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -560,16 +560,22 @@ settable occupancy knob, blocks_per_sm, was then swept and is DEAD: with routing SEEDED so every configuration sees identical work, the between- configuration spread (4.3%) is no larger than the within-configuration spread, and an apparent 8.7% win in an unseeded first pass was the routing draw moving -rather than the parameter. Three traps worth carrying. dram__bytes.sum reads n/a on GB10, so ncu's Memory -Throughput % excludes DRAM traffic and is not a bandwidth utilisation. Any MoE -comparison that lets routing vary between arms measures the draw, not the -change. And this box's GPU lock is $HOME/gpu.lock, not /tmp/gpu.lock: the -standalone runs above took the wrong file and so ran UNLOCKED against a -concurrent GPU test, which makes their absolute timings (including the 203-226 -GB/s plateau) upper bounds needing a re-take, while the INTERLEAVED ratio, the -two-regime shape and the distinct-experts scaling survive because contention -lands on both arms alike. nvidia-smi showing no compute apps does not mean the -GPU is unreserved; fuser -v $HOME/gpu.lock is the check. Weight +rather than the parameter. A fresh review then corrected three things here. Cost per distinct expert is +NOT flat: recomputed it spans 4.47-7.50 us, and two rows with identical expert +counts differ 89.4 vs 149.9 us while blocks differ 73 vs 137, so in the +L2-resident regime time tracks BLOCKS and the distinct-expert model applies +only above ~40. Control BOTH, or state the regime. The plateau over the rows +actually flat is 212.7-225.8 GB/s, not 203-226, and under that narrower range +ours at 209.9 is NOT inside it. And the standalone runs took /tmp/gpu.lock +rather than this box's $HOME/gpu.lock, so they ran unserialised: contention +inflates TIME, which makes a bytes/time bandwidth a LOWER bound, so a clean +re-take can only raise the plateau and would REVERSE the we-are-not-slow +reading. That reading is provisional. What survives is the INTERLEAVED 0.9973 +ratio, since interleaving protects a ratio against both contention and routing +draw. Two traps worth carrying: dram__bytes.sum reads n/a on GB10, so ncu's +Memory Throughput % excludes DRAM traffic; and fuser -v $HOME/gpu.lock is the +check, because nvidia-smi showing no compute apps does not mean the GPU is +unreserved. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under diff --git a/scripts/marlin-moe-standalone.py b/scripts/marlin-moe-standalone.py index a2de54b3d..e65da6844 100755 --- a/scripts/marlin-moe-standalone.py +++ b/scripts/marlin-moe-standalone.py @@ -27,7 +27,7 @@ def main() -> int: ap = argparse.ArgumentParser() ap.add_argument("--m", type=int, default=9, help="tokens (1 + k spec draft)") - ap.add_argument("--iters", type=int, default=200) + ap.add_argument("--iters", type=int, default=80) # matches the C++ arm ap.add_argument("--warmup", type=int, default=20) ap.add_argument("--arm", choices=["gate_up", "down", "both"], default="both") ap.add_argument("--bps", type=int, default=-1, From 258f501e412bbbbdb807839478b93f10b5ddbedb Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Thu, 13 Aug 2026 21:15:19 +0000 Subject: [PATCH 11/11] record(dspark): the n=2 repeat FAILS its own drift gate -- ~0.98 on ONE valid run (#442) The confirmation for 0.9889 ran. It does not confirm cleanly, and the gate that rejects it was set before the number existed, so it is reported in full rather than quietly replaced by the friendlier run. run 1 run 2 ours BEFORE (n=9) 142.604 146.740 ours AFTER (n=9) 142.140 143.619 DRIFT -0.33% -2.13% validity (<1%) PASS FAIL ours combined 142.534 145.117 oracle (n=15) 144.130 UNIMODAL 148.150 BIMODAL ratio 0.9889 0.9795 (not usable) RUN 2 IS REJECTED, not averaged in. Its two ours arms disagree by 2.13%, twice the effect being measured, and the drift is DOWNWARD -- the closing arm slower than the opening one, thermal or contention, the opposite sign from the cold-start ramp the warm-up arm exists to remove. Bracketing on its own arms it spans 0.9912 to 0.9701: it cannot separate parity from a 3% deficit. Two things it does establish. The oracle's BIMODALITY IS BOOT-DEPENDENT. Run 1 was unimodal, 15 draws near 144 with a single outlier. Run 2 is clearly bimodal, 10 draws at 148.05 and 5 at 156.61 -- the same one-extra-accepted-token effect earlier sessions saw. Same script, same pin, same prompt, different boot. So whether the modal-value correction is needed is a property of the BOOT, and no harness may assume either shape. ABSOLUTES MOVED TOGETHER across the reboot: ours 142.5 -> 145.1 (+1.8%), oracle 144.1 -> 148.2 (+2.8%), consistent with the recorded 12.8% boot-to-boot SM clock variation and further reason no absolute from this box survives a boot. STANDING CLAIM: ~0.98, NOT parity, on ONE valid paired run. Not "0.9889 confirmed". Both runs are consistent with a 1-2% deficit and neither reaches 1.0. The binding constraint on resolving 1% here is the box: it rebooted or dropped FIVE times on 2026-08-13 (08:57, 09:29, 16:29, ~21:16, and an outage near 20:40), each one destroying whatever was queued. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code] --- .agents/benchmark-record.md | 47 +++++++++++++++++++++++++++++ .agents/specs/dspark-spec-decode.md | 13 ++++++-- docs/BENCHMARKS.md | 2 +- docs/STATUS.md | 19 ++++++++---- 4 files changed, 72 insertions(+), 9 deletions(-) diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 2d9e22be7..8ce7e4465 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -21363,3 +21363,50 @@ refusal never fired: `Ltx2SelectTextFeatureVariant` **does** refuse a partial se (`ltx2_text_encoder.cpp:184-192`). It never fired because **no production path called the selector before L13** — the marker keys and the engine's first call to it landed in the same commit, so there was no earlier run for it to refuse. + +## SPEC-DSPARK: the n=2 repeat FAILS its own drift gate (2026-08-13) + +The confirmation run for the 0.9889 result. Reported in full because it does not +confirm cleanly, and the gate that rejects it was set before the number existed. + +| | run 1 | run 2 | +|---|---|---| +| ours BEFORE (n=9) | 142.604 | 146.740 | +| ours AFTER (n=9) | 142.140 | 143.619 | +| DRIFT before->after | **-0.33%** | **-2.13%** | +| validity (gate: abs(drift) < 1%) | PASS | **FAIL** | +| ours combined | 142.534 | 145.117 | +| oracle (n=15) | 144.130, UNIMODAL | 148.150, BIMODAL | +| ratio | **0.9889** | 0.9795 (not usable) | + +RUN 2 IS REJECTED, not averaged in. Its two `ours` arms disagree by 2.13%, +which is twice the effect being measured, and the drift is DOWNWARD (the closing +arm slower than the opening one) -- thermal or contention, the opposite sign +from the cold-start ramp the warm-up arm was built to remove. Bracketing on its +own arms gives 0.9912 (before) to 0.9701 (after): it cannot distinguish parity +from a 3% deficit. + +TWO THINGS IT DOES ESTABLISH. + +The oracle's BIMODALITY IS BOOT-DEPENDENT. Run 1, unimodal: 15 draws at ~144 +with one outlier. Run 2, clearly bimodal: 10 draws at 148.05 and 5 at 156.61, +the same one-extra-accepted-token effect seen in earlier sessions. Same script, +same pin, same prompt, different boot. So whether the modal-value correction is +needed is a property of the BOOT, not of the workload, and a harness cannot +assume either shape. + +ABSOLUTES MOVED AGAIN ACROSS THE REBOOT, both arms together: ours 142.5 -> 145.1 +(+1.8%), oracle 144.1 -> 148.2 (+2.8%). Consistent with the recorded 12.8% +boot-to-boot SM clock variation, and further reason no absolute from this box is +quotable across boots. + +WHERE THAT LEAVES THE RATIO: one VALID paired run at 0.9889, one REJECTED run +bracketing 0.970-0.991. Both are consistent with a deficit of roughly 1-2%, and +neither reaches parity. The honest statement is "~0.98, not parity, n=1 valid" +-- not "0.9889 confirmed". + +The box rebooted or dropped FIVE times on 2026-08-13 (08:57, 09:29, 16:29, +~21:16, plus an outage around 20:40), which is the binding constraint on +resolving 1% here at all. + +Evidence: `dgx.casa:~/work/dspark-w6/paired_lm.log`, `paired_lm2.log`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index 6798211d3..d272db82e 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -2001,8 +2001,17 @@ what remains to be found -- roughly a third of the deficit the record has been chasing all campaign, and it lands after 6ae had already cleared the kernel that deficit was attributed to. -**Caveats, kept rather than buried.** n=1 paired run; a repeat is owed before -this is settled. The ours-AFTER arm carries four low outliers (135.2, 135.5, +**Caveats, kept rather than buried.** n=1 paired run. **The repeat RAN and was +REJECTED by this harness's own drift gate**: its arms measured 146.740 before +and 143.619 after, a -2.13% drift against a 1% gate, so its 0.9795 does not +count and is not averaged in. Bracketing on its own arms it spans 0.9912 to +0.9701 -- it cannot separate parity from a 3% deficit. Two things it did +establish: the oracle's BIMODALITY IS BOOT-DEPENDENT (run 1 unimodal at ~144, +run 2 clearly bimodal with 10 draws at 148.05 and 5 at 156.61, same script and +pin), so a harness cannot assume either shape; and absolutes moved together +across the reboot (ours +1.8%, oracle +2.8%), consistent with the recorded 12.8% +boot-to-boot clock variation. So the standing claim is **~0.98, NOT parity, on +one valid paired run** -- not "0.9889 confirmed". The ours-AFTER arm carries four low outliers (135.2, 135.5, 139.0, 139.9) that the BEFORE arm does not, so something touched the box during it -- medians are unaffected, that arm's spread is not trustworthy. The oracle shows ONE fast draw (151.84) against fourteen near 144, the old bimodality diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 17b0b9491..406369cc1 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -351,7 +351,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE 35B-A3B **0.9889x** of the pinned graphed oracle in the first fully-controlled paired run (warm-up arm, settle barriers, correct lock, drift -0.33%). NOT parity (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE 35B-A3B **~0.98x** of the pinned graphed oracle: one VALID controlled paired run at 0.9889 (drift -0.33%), repeat REJECTED on a -2.13% drift gate. NOT parity (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index a09c9f95f..fd8da6f40 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -591,12 +591,19 @@ arm ~6% low), settle barriers (vLLM asserts free GPU memory does not grow during its startup profile, and GB10 releases our pages lazily, which killed every earlier paired attempt), and a host-RAM headroom guard (gpu_memory_utilization reserves HOST RAM here, which took the machine down -three times on 2026-08-13) -- measures ours at 142.534 tok/s against the -oracle's 144.130, a ratio of **0.9889** with before/after drift of -0.33%, -inside the 1% validity gate. So the gap is 1.1%, not 3.4%: the -0.9757/0.9646/0.9569 recorded earlier were measuring an unwarmed first arm as -much as the engine. Still NOT parity and the row stays open, with a repeat -owed (n=1). Two traps worth carrying: dram__bytes.sum reads n/a on GB10, so +three times on 2026-08-13) -- measures ours at 142.534 tok/s against the oracle's 144.130, a ratio +of **0.9889** with before/after drift of -0.33%, inside the 1% validity gate. So +the gap is ~1%, not 3.4%: the 0.9757/0.9646/0.9569 recorded earlier were +measuring an unwarmed first arm as much as the engine. The n=2 repeat RAN and +was REJECTED by the same gate at -2.13% drift, so its 0.9795 is not averaged +in and the standing claim is ~0.98 on ONE valid paired run, still NOT parity. +The repeat did establish two things: the oracle's bimodality is BOOT-DEPENDENT +(unimodal near 144 in one run, 10-at-148 plus 5-at-157 in the next, same +script and pin), so no harness may assume either shape; and both arms' +absolutes moved together across the reboot, ours +1.8% and oracle +2.8%, +matching the recorded 12.8% boot-to-boot clock variation. The binding +constraint on resolving 1% here is the box, which rebooted or dropped five +times on 2026-08-13. Two traps worth carrying: dram__bytes.sum reads n/a on GB10, so ncu's Memory Throughput % excludes DRAM traffic; and fuser -v $HOME/gpu.lock is the check, because nvidia-smi showing no compute apps does not mean the GPU is unreserved. Weight