diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 66a6d4d21..8ce7e4465 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -20459,6 +20459,335 @@ Oracle draws remain bimodal (~147.3 and ~155.6), the same one-extra-accepted-tok effect as the fibacc run, so its MODAL draws are the honest denominator. Evidence: `dgx:~/work/dspark-w6/iocheck.log`, `final_pair.log`. +## SPEC-DSPARK / #442: upstream Marlin profiled under ncu at last; 6z CORROBORATED (2026-08-13) + +`scripts/marlin-moe-standalone.py` drives upstream's own +`torch.ops._moe_C.moe_wna16_marlin_gemm` on the 35B-A3B decode shapes with no +EngineCore, no multiprocessing and no model load, which is what makes `ncu` +attach -- the blocker recorded as "BLOCKED, both replay modes TRIED". vLLM +0.23.1rc1.dev1511+g555967922 (identity asserted), torch 2.13.0+cu130, GB10. + +STATIC GEOMETRY (trustworthy): grid 144, block 128, 32768 B shared per block. +48 SMs x 102400 B shared => 3 blocks/SM => 25% occupancy (achieved 25.98%), and +48 x 3 = 144 = the grid. A persistent single wave; 94 registers/thread. + +MEASUREMENT TRAP: `dram__bytes.sum` is `n/a` on GB10 -- no DRAM counters exist. +The SpeedOfLight "Memory Throughput 11.14%" therefore excludes DRAM traffic, and +reading it beside "Compute (SM) 11.42%" as "latency-bound" is WRONG. + +WHAT THE WORK SWEEP SHOWS (gate_up, 80 iters/point; pool = distinct experts): + +| pool | blocks | us/call | us/block | implied GB/s | +|---|---|---|---|---| +| 8 | 13 | 19.9 | 1.53 | fits L2 | +| 16 | 18 | 20.8 | 1.15 | fits L2 | +| 24 | 22 | 65.0 | 2.96 | transition | +| 32 | 27 | 157.0 | 5.81 | 202.9 | +| 48 | 38 | 207.0 | 5.45 | 216.6 | +| 64 | 45 | 249.5 | 5.55 | 212.7 | +| 128 | 58 | 306.5 | 5.28 | 223.2 | +| 256 | 65 | 339.6 | 5.22 | 225.8 | + +1179648 B streamed per block (1 MiB weights + 128 KiB scales). us/block is FLAT +at 5.2-5.5 across a 2.4x range of work: constant bytes/second, i.e. bandwidth +limited. Plateau 203-226 GB/s. + +CONSEQUENCE: the derived in-situ figures land ON this plateau. Upstream's +210.7 GB/s is INSIDE it (upstream runs at the kernel's achievable bandwidth); +ours at 186.6 GB/s is ~12% BELOW. An independent standalone measurement now +CORROBORATES the DRAM attribution rather than replacing it. What does not +survive is the per-unit-work DIVISION: with a fixed 144-CTA grid, block count is +loop iterations, and the sweep prices 38.9 -> 40.6 blocks at about +4.4%. + +NOT YET RUN, and decisive: our kernel through this same harness, us/block against +the 5.2-5.5 plateau. Occupancy (25%, shared-memory capped, under 25600 B would +buy 4 blocks/SM) is a real but SECONDARY lever at ~75-80% of this device's +~273 GB/s. + +Absolute us/call is not comparable in-situ (uniform synthetic routing occupies +61-65 blocks vs the model's 38.9-40.6); geometry, regime shape and plateau +bandwidth are. + +Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: our Marlin == upstream's at matched work; 6x REFUTED (2026-08-13) + +`benchmarks/marlin_moe_standalone.cpp` is the OUR-side arm of the harness in the +previous entry: same 35B-A3B gate_up shapes, same expert-pool control over the +occupied block count, driving `vt::MoeGroupedGemmNvfp4Marlin`. + +Sweep, us/block, ours vs upstream: 5.482/5.813 (pool 32), 5.525/5.323 (40), +5.311/5.446 (48), 5.318/5.545 (64), 5.194/5.284 (128), 5.261/5.224 (256). Ours +plateaus on the SAME 5.2-5.5 band. + +Two INTERLEAVED paired runs, pool 48 and 128, 3 reps each, 12 points per arm: + +| | n | mean us/block | sd | range | +|---|---|---|---|---| +| ours | 12 | 5.3187 | 0.124 | 5.083-5.562 | +| upstream | 12 | 5.3330 | 0.151 | 5.042-5.560 | + +ours/upstream = 0.9973 (ours 0.27% FASTER), inside one sd, sign flipping between +runs. A per-call workspace memset ours pays and upstream does not was isolated +via `--zero-ws 0`: noise. + +CONSEQUENCE: the in-situ "8.2% slower inside one kernel / 12.8% per unit work / +186.6 vs 210.7 GB/s" does NOT reproduce at matched work. Both engines reach the +same 203-226 GB/s plateau. That attribution describes the in-situ RUNS, not the +kernel, and the localisation of the residual to `marlin_moe_wna16::Marlin` is +REFUTED. + +WHERE TO LOOK NEXT: the sweep shows time is set by DISTINCT EXPERTS touched per +launch (1.15 us/block at 16 experts, weights in L2; ~5.3 above ~27, streaming -- +a 4.6x swing). Blocks are not experts, so the recorded 38.9 vs 40.6 blocks does +not settle it. Measure distinct experts per launch on both arms IN SITU. + +CAVEATS: our arm links ~/work/pr234's build, vendored marlin_mm_moe.cu +byte-identical to main (md5 85c40e4869bc6ec594b8cfb97fb58b3c), dispatcher +predates the perf-neutral C_tmp cap. Ours times with steady_clock over 80 +iterations, upstream with CUDA events. + +Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: the in-situ denominator was WRONG -- experts, not blocks (2026-08-13) + +Holding blocks roughly fixed while varying DISTINCT experts (upstream arm, gate_up): + +| M | blocks | distinct | us/call | us/block | +|---|---|---|---|---| +| 64 | 73 | 20 | 89.4 | 1.22 | +| 64 | 82 | 40 | 227.1 | 2.77 | +| 64 | 94 | 80 | 427.2 | 4.55 | +| 64 | 216 | 216 | 1121.2 | 5.19 | +| 128 | 137 | 20 | 149.9 | 1.09 | +| 128 | 146 | 40 | 220.0 | 1.51 | +| 128 | 255 | 250 | 1298.6 | 5.09 | + +M=128: 137 -> 146 blocks is +6.6% blocks and +46.7% TIME, because distinct +experts doubled. Cost per DISTINCT EXPERT is flat at 5.2-5.7 us (1.125 MiB, +~205-225 GB/s); cost per BLOCK varies 4.7x. So time ~= distinct_experts x +1.125 MiB / ~215 GB/s and blocks are nearly irrelevant. + +Every in-situ comparison normalised by BLOCKS (38.9 ours vs 40.6 upstream), which +is not the driver. Applying the model to the recorded launches: ours 164.0 us +implies ~30 distinct experts, upstream 151.6 us implies ~28. About TWO distinct +experts per launch reproduces the entire 8.2% with zero implementation +difference, and the matched-work comparison already put the two kernels at +0.27%. + +It also explains both in-situ arms beating the standalone plateau per block +(4.21 and 3.73 vs ~5.3): in situ several blocks share an expert. + +RULE FOR FUTURE MoE COMPARISONS: control distinct experts per launch, or force +both arms onto an identical token stream. Block counts compare the wrong +quantity. VT_MOE_PAD_STATS should count distinct experts as well. + +Does NOT move the e2e ratio (~0.966, wall-clock on matched prompts); it removes +the attribution of its residual. + +Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: CORRECTION -- ours touches FEWER experts, not more (2026-08-13) + +The entry above inferred, from the distinct-experts model, that our in-situ +launches touch ~30 distinct experts against upstream's ~28 and that this +explained the 8.2%. That is BACKWARDS, and the counts contradicting it were +already in this file: the 2026-08-12 both-sides measurement recorded ours 311.2 +padded tokens / 38.9 blocks per call against upstream 324.8 / 40.6. + +At M=9 the 72 (token, expert) pairs spread over ~39 experts at under 8 tokens +each, so moe_align emits ONE block per expert and blocks and distinct experts +COINCIDE at this shape. That measurement was therefore already counting experts, +upstream touches MORE of them, and the routing explanation stays refuted exactly +as it concluded. Blocks and experts do come apart at larger batches (spec §6af +measures 4.7x divergence at M=64/128), which is why the distinction is worth +keeping, but it does not apply at the decode shape. + +What survives, and it is sharper than what it replaces: + + standalone, matched work ours == upstream to 0.27% + in situ ours does LESS work (38.9 vs 40.6 expert-blocks) + and takes MORE time (164.0 vs 151.6 us) + +A kernel identical in isolation cannot be slower in place because of its own +code, so the deficit belongs to the CONTEXT, not the kernel and not the routing. +Candidates in evidence order: expert-weight residency in situ (this repo has +measured 20-30% per-GEMM for host/ATS-retagged decode weights, and the +standalone arm's fresh cudaMalloc cannot reproduce that), clock/power state +across runs, and overlap with concurrent stream work. + +Measurement-base caveat: in-situ per-launch times are summed profiler kernel +durations, standalone are wall-clock over 80 iterations. Both in-situ arms beat +the standalone plateau per unit work (4.21 and 3.73 vs ~5.3), which may be partly +that difference rather than a physical one. + +## SPEC-DSPARK / #442: the GEMM MIX dissolves the anomaly, and the in-situ normalisation has a MODE hole (2026-08-13) + +Two arithmetic corrections close out this thread. + +1. THE MIX. The 1520 in-situ launches are 760 gate_up plus 760 down, and down's + per-expert bytes are exactly HALF gate_up's: gate_up is 2N x K/2 weights plus + 2N x K/16 scales = 1.1250 MiB, down is K x N/2 plus K x N/16 = 0.5625 MiB, so + the mixed average is 0.8438 MiB per expert-block. Comparing a mixed in-situ + average against a gate_up-ONLY standalone plateau was apples to oranges, and + it is what made both arms appear to "beat" the plateau (4.21 and 3.73 us + against ~5.3). They did not. + + Redone with the right bytes: + + | | blocks | MiB/launch | us | implied GB/s | + |---|---|---|---|---| + | ours | 38.9 | 32.8 | 164.0 | **209.9** | + | upstream | 40.6 | 34.3 | 151.6 | **236.9** | + + Standalone gate_up plateau, measured: 203-226 GB/s. **Ours sits INSIDE it. + Upstream sits ABOVE it.** So we run this kernel at the bandwidth it achieves + in isolation, and upstream gets something in place that the isolated kernel + does not -- cache reuse across the gate_up/down pair (down's weights are half + size, so more of the working set can persist) is the first candidate. The + framing inverts: we are not slow here, upstream is unusually fast. + +2. THE MODE HOLE. This file already warns "a work COUNT may be taken under + different execution modes; a TIME may not", and then the per-unit-work + normalisation does exactly what that warns against: the 38.9/40.6 counts were + taken EAGER (ours VT_SPEC_DECODE_GRAPH=0, upstream enforce_eager, both + because the probes could not survive capture/compile) while the 249.2/230.4 ms + times came from the GRAPHED profile. Nothing establishes that the graphed runs + had the same blocks per launch as the eager ones, so every ratio built by + dividing those times by those counts -- 4.21 vs 3.73 us/block, 12.8% per unit + work, 186.6 vs 210.7 GB/s -- rests on a denominator measured in a different + execution mode than its numerator. + +CONSEQUENCE: the only like-for-like Marlin comparison anyone has is the +standalone one, and it says parity (0.9973, inside 1 sd). Closing the in-situ +question needs blocks AND time from the SAME graphed run, which needs a probe +that survives capture -- a device-side counter written by the kernel launcher, +read once at the end, never a per-launch D2H sync. + +Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: blocks_per_sm is a DEAD lever, and the box is currently unfit to measure 3% (2026-08-13) + +`moe_wna16_marlin_gemm`'s last parameter is `blocks_per_sm`; both engines pass +-1 (auto), which yields 32768 B of shared memory, 3 blocks/SM and 25% occupancy. +Since the kernel runs at ~78% of peak bandwidth, forcing a higher value looked +like a best-in-class lever. It is not. + +FIRST SWEEP, UNSEEDED, and it is a trap worth recording: bps -1/1/4 read slow and +2/3/5/6/8 read fast, apparently 8.7%. The routing was redrawn per run, so +`distinct` moved 35-41 and the sweep was measuring the DRAW, not the parameter. +Time tracks distinct experts, so any comparison that lets routing vary measures +nothing else. + +SEEDED (torch.manual_seed, identical 33 distinct / 33 blocks everywhere), 4 +interleaved reps at 120 iters: + +| bps | runs (us) | mean excl. outliers | +|---|---|---| +| -1 (auto) | 191.3, 367.8, 177.2 | 184.2 | +| 3 | 181.1, 171.1, 177.6, 353.0 | 176.6 | +| 4 | 177.7, 178.6, 179.8, 180.3 | 179.1 | +| 5 | 181.7, 184.2, 179.4 | 181.7 | + +Between-config spread 4.3%; WITHIN-config spread comparable (bps=3 alone ranges +171-181). No reliable effect. The lever is DEAD, and the 8.7% +came from the UNSEEDED first pass, i.e. the routing draw moving rather than the +parameter. + +ENVIRONMENT, and this is the more useful finding: two of fourteen runs returned +~2x (367.8 and 353.0) and two runs produced no output at all. A box that emits +2x outliers cannot resolve a 3.4% question, and this one is currently contended +-- a concurrent session has been building and rebuilding a ~79G tree on the same +hardware. No parity measurement taken in this window should be trusted, in +either direction. Earlier paired numbers in this file were taken before that +contention began and were interleaved, which is what makes them survivable. + +Evidence: `dgx.casa:~/work/marlin442/`. + +## SPEC-DSPARK / #442: CORRECTION -- the standalone runs were taken UNLOCKED (2026-08-13) + +Every standalone Marlin run in the entries above was wrapped in +`flock /tmp/gpu.lock`. That is the WRONG FILE. This box's GPU lock is +`$HOME/gpu.lock` -- it is what `final_pair.sh` takes and what other sessions +hold. `/tmp/gpu.lock` coordinates with nothing, so those runs executed +unserialised against whatever else was on the GPU, and at least one concurrent +`test_qwen27_spec_decode_concurrent` (15.9 GB RSS) overlapped them. + +That is the likeliest source of the ~2x outliers (367.8 and 353.0 us) in the +blocks_per_sm sweep, and it was mine, not the other session's. An earlier note +in this session blamed them for not locking; the reverse is true. + +WHAT SURVIVES. The ours-vs-upstream comparisons were INTERLEAVED within a single +run (ours, upstream, ours, upstream), so contention lands on both arms alike and +the RATIO is the quantity interleaving protects. 0.9973 with the sign flipping +between reps still stands as "indistinguishable", though the error bars are wider +than the quoted sd suggests. + +WHAT DOES NOT. Absolute us/call and us/block from those runs, including the +203-226 GB/s plateau, are upper-bounded rather than exact -- an unlocked box can +only make them slower. The plateau should be re-taken under `$HOME/gpu.lock` +before it is quoted as the kernel's achievable bandwidth. The two-regime SHAPE +(L2-resident under ~18 blocks, streaming above ~27) and the distinct-experts +scaling are robust to a uniform slowdown and do not need re-taking. + +RULE: take `$HOME/gpu.lock`, not `/tmp/gpu.lock`. Check `fuser -v ~/gpu.lock` +before assuming the GPU is free; `nvidia-smi` showing no compute apps does NOT +mean it is unreserved, since a holder may be between phases. + +## SPEC-DSPARK / #442: CORRECTIONS from fresh review -- per-expert cost is NOT flat, and the bound direction was wrong (2026-08-13) + +A fresh review of the branch found two blocking errors in the entries above. +Both are recorded here rather than edited away, since this log is append-only. + +1. "Cost per DISTINCT EXPERT is flat at 5.2-5.7 us" is FALSE. Recomputing + us/call / distinct from the table's own numbers gives 4.470, 5.678, 5.340, + 5.191, 7.495, 5.500, 5.194 -- a range of 4.47 to 7.50. The two distinct=20 + rows are the tell: identical expert counts, 89.4 vs 149.9 us, blocks 73 vs + 137. THERE, TIME TRACKS BLOCKS, which is what a flat-per-expert model calls + irrelevant. + + The honest model has the same two regimes as the L2 finding. Weights fitting + L2 (low distinct) => not re-streamed => cost set by per-block work => time + tracks BLOCKS. Weights not fitting (distinct >= ~40 here) => cost set by + streaming => time tracks DISTINCT EXPERTS at 5.2-5.7 us each. + `time ~= distinct x 1.125 MiB / ~215 GB/s` applies ONLY in the second regime; + at distinct=20 it predicts 109.7 us against 89.4 and 149.9 measured, off by + -19% and +37%. + + The rule this file stated -- "control distinct experts per launch" -- is + therefore INCOMPLETE. Control BOTH distinct experts AND blocks, or state + which regime the comparison is in. + +2. The lock caveat gave the bound the WRONG DIRECTION. It called the plateau an + upper bound. Contention inflates TIME, so a bandwidth computed as bytes/time + is a LOWER BOUND on what the kernel achieves. A re-take under + `$HOME/gpu.lock` can only move the plateau UP, which would move our in-situ + 209.9 GB/s BELOW it and REVERSE the "ours is inside it, upstream above it" + inversion. That inversion is the least favourable reading to us the data + admits and the one most likely to change on a clean re-take. + +3. The plateau range 203-226 GB/s mixes bands: over the rows actually flat + (>= 38 blocks) it is 212.7-225.8, and the 203 endpoint is the 27-block row at + 5.81 us/block, which the same paragraph places OUTSIDE the flat band. Under + the narrower, self-consistent range, "ours (209.9) sits INSIDE it" is FALSE. + +4. The harness arms draw routing from independent RNG streams (python + torch.manual_seed, C++ mt19937(1234)), so at the same pool they occupy + different block counts (48: 39 vs 38; 128: 54 vs 58). "At matched blocks" was + inaccurate -- blocks are NORMALISED, not matched. Neither arm accepts an + external routing tensor, so the harness cannot yet satisfy the "force both + arms onto one token stream" rule stated one section later. The 0.9973 ratio + is probably still sound at M=9, but its error bars carry an undisclosed draw + term on top of the contention term. + +5. `benchmarks/marlin_moe_standalone.cpp` is not wired into any build target, so + nothing compiles it and it will rot against + `vt::MoeGroupedGemmNvfp4Marlin`'s signature. Tracked as owed. + +NET EFFECT ON CONCLUSIONS: 6ae's interleaved 0.9973 (the kernel is not the gap) +SURVIVES, since interleaving is what protects a ratio against both contention +and draw. 6ag's "we are not slow, upstream is unusually fast" DOES NOT survive +as stated and is now marked provisional pending a re-take under the right lock. ## BENCH-CLOCK-CONTROLLED-PIN-GRID: the first series measured with the correct oracle, its production configuration, and a controlled clock at the same time (2026-08-13) @@ -20673,6 +21002,52 @@ the attribution, per AGENTS.md: evidence is moved and annotated, never removed. for the residual TTFT gap is that it is the same prefill glue attributed at 92.5% for 27B, now measured against a correctly fused denominator for the first time, so the attribution itself is owed a re-run. + +## SPEC-DSPARK: the FIRST fully-controlled paired run -- 0.9889, not 0.966 (2026-08-13) + +Every earlier ratio in this file was taken with a COLD leading arm. Correcting +that moves the number materially. + +Controls, all four present for the first time: + * `$HOME/gpu.lock` (earlier standalone runs took /tmp/gpu.lock, which + coordinates with nothing) + * a DISCARDED warm-up arm before the first measured arm -- the GB10 SM clock + ramps over MINUTES (1449 -> 2190 MHz observed), so dropping rep 1 is not + enough and a whole first arm reads ~6% low + * settle barriers between arms: vLLM asserts free GPU memory does not GROW + during its startup profile, and GB10 returns our engine's pages lazily, so + an oracle launched straight after our arm dies with + "Initial free memory 68.53 GiB, current free memory 89.42 GiB" + * a host-RAM headroom guard before the oracle. gpu_memory_utilization + reserves HOST RAM on GB10, so an oracle without headroom takes the MACHINE + down: three reboots on 2026-08-13 (08:57, 09:29, 16:29), at least the last + of them ours. The oracle copy runs at 0.35 instead of 0.55; at + max_num_seqs=2 / max_model_len=2048 the KV cache needed is a fraction of + either budget, so this cannot change decode speed. + +| arm | n | median tok/s | range | +|---|---|---|---| +| ours BEFORE | 9 | 142.604 | 141.67-142.79 | +| ours AFTER | 9 | 142.140 | 135.23-142.87 | +| ours combined | 18 | 142.534 | -- | +| oracle | 15 | 144.130 | 141.86-151.84 | + +DRIFT before -> after = **-0.33%**, inside the 1% validity gate this harness +sets for itself, so the run counts. Ratios: 0.9894 (before), 0.9862 (after), +**0.9889 (combined)**. + +**So the gap is 1.1%, not 3.4%.** The 0.9757 / 0.9646 / 0.9569 recorded earlier +were measuring an unwarmed first arm as much as the engine. NOT parity -- 0.9889 +is still below 1.0 -- but the deficit is a third of what the record claimed. + +Caveats kept deliberately: n=1 paired run, so this needs a repeat before it is +treated as settled. The ours-AFTER arm carries four low outliers (135.2, 135.5, +139.0, 139.9) absent from the BEFORE arm, so something touched the box during +it; the medians are unaffected but the after arm's spread is not trustworthy. +The oracle shows ONE fast draw (151.84) against fourteen at ~144, the old +bimodality appearing once, which the median absorbs. + +Evidence: `dgx.casa:~/work/dspark-w6/paired_lm.log`, `~/paired_lm.sh`. ## LTX-2.5 L9c — the per-phase pool drain is worth 0.11 GiB, and L9B's 58 GB runaway does not reproduce (2026-08-13, `row/LTX25-L9C-CONNECTOR-DRAIN`, issue [#435](https://github.com/mudler/vllm.cpp/issues/435)) **No speed number is claimed and none is implied.** LTX-2.5's speed axis is @@ -20988,3 +21363,50 @@ refusal never fired: `Ltx2SelectTextFeatureVariant` **does** refuse a partial se (`ltx2_text_encoder.cpp:184-192`). It never fired because **no production path called the selector before L13** — the marker keys and the engine's first call to it landed in the same commit, so there was no earlier run for it to refuse. + +## SPEC-DSPARK: the n=2 repeat FAILS its own drift gate (2026-08-13) + +The confirmation run for the 0.9889 result. Reported in full because it does not +confirm cleanly, and the gate that rejects it was set before the number existed. + +| | run 1 | run 2 | +|---|---|---| +| ours BEFORE (n=9) | 142.604 | 146.740 | +| ours AFTER (n=9) | 142.140 | 143.619 | +| DRIFT before->after | **-0.33%** | **-2.13%** | +| validity (gate: abs(drift) < 1%) | PASS | **FAIL** | +| ours combined | 142.534 | 145.117 | +| oracle (n=15) | 144.130, UNIMODAL | 148.150, BIMODAL | +| ratio | **0.9889** | 0.9795 (not usable) | + +RUN 2 IS REJECTED, not averaged in. Its two `ours` arms disagree by 2.13%, +which is twice the effect being measured, and the drift is DOWNWARD (the closing +arm slower than the opening one) -- thermal or contention, the opposite sign +from the cold-start ramp the warm-up arm was built to remove. Bracketing on its +own arms gives 0.9912 (before) to 0.9701 (after): it cannot distinguish parity +from a 3% deficit. + +TWO THINGS IT DOES ESTABLISH. + +The oracle's BIMODALITY IS BOOT-DEPENDENT. Run 1, unimodal: 15 draws at ~144 +with one outlier. Run 2, clearly bimodal: 10 draws at 148.05 and 5 at 156.61, +the same one-extra-accepted-token effect seen in earlier sessions. Same script, +same pin, same prompt, different boot. So whether the modal-value correction is +needed is a property of the BOOT, not of the workload, and a harness cannot +assume either shape. + +ABSOLUTES MOVED AGAIN ACROSS THE REBOOT, both arms together: ours 142.5 -> 145.1 +(+1.8%), oracle 144.1 -> 148.2 (+2.8%). Consistent with the recorded 12.8% +boot-to-boot SM clock variation, and further reason no absolute from this box is +quotable across boots. + +WHERE THAT LEAVES THE RATIO: one VALID paired run at 0.9889, one REJECTED run +bracketing 0.970-0.991. Both are consistent with a deficit of roughly 1-2%, and +neither reaches parity. The honest statement is "~0.98, not parity, n=1 valid" +-- not "0.9889 confirmed". + +The box rebooted or dropped FIVE times on 2026-08-13 (08:57, 09:29, 16:29, +~21:16, plus an outage around 20:40), which is the binding constraint on +resolving 1% here at all. + +Evidence: `dgx.casa:~/work/dspark-w6/paired_lm.log`, `paired_lm2.log`. diff --git a/.agents/specs/dspark-spec-decode.md b/.agents/specs/dspark-spec-decode.md index b885405a6..d272db82e 100644 --- a/.agents/specs/dspark-spec-decode.md +++ b/.agents/specs/dspark-spec-decode.md @@ -1663,6 +1663,363 @@ quotable -- and all three agree. Also visible in the last run: the oracle's draws are bimodal at ~147.3 and ~155.6, the same one-extra-accepted-token effect as §6q, which is why its MODAL draws are the honest denominator. +## 6ad. THE ncu BLOCKER IS GONE, AND 6z's DRAM ATTRIBUTION IS CORROBORATED (2026-08-13) + +Section 6z listed upstream's `ncu` counters as the last open route and called it +BLOCKED: vLLM's EngineCore will not initialise under `ncu` in either replay mode. +`scripts/marlin-moe-standalone.py` routes around it. It drives upstream's OWN +`torch.ops._moe_C.moe_wna16_marlin_gemm` on the 35B-A3B decode shapes -- hidden +2048, moe_intermediate 512, E=256, top_k=8, `moe_block_size=8` -- with NO +EngineCore, NO multiprocessing and NO model load, mirroring +`prepare_nvfp4_moe_layer_for_marlin`'s scale pipeline exactly. It asserts the +oracle identity and aborts on mismatch. It profiles. + +### The static geometry (trustworthy, and new) + +| property | value | +|---|---| +| grid / block / shared per block | 144 / 128 / 32768 B | +| GB10 | 48 SMs, 102400 B shared per SM | +| block limit -- **shared memory** | **3** (registers 5, warps 12, SM 24) | +| occupancy, theoretical / achieved | 25% / 25.98% | +| waves per SM | 1 | +| registers per thread | 94 | + +102400/32768 = 3 blocks per SM, and 48 x 3 = 144 = the grid exactly. **The launch +is a persistent single wave sized to the device**, so the "38.9 vs 40.6 blocks per +launch" of 6x are work items each CTA LOOPS OVER, not parallel blocks. + +### `ncu`'s memory percentages are NOT usable on this chip + +`dram__bytes.sum` and `dram__bytes.sum.per_second` return **`n/a`** on GB10 -- +there are no DRAM counters to read. What the SpeedOfLight section still prints, +"Memory Throughput 11.14%" against "Compute (SM) Throughput 11.42%", therefore +does NOT include DRAM traffic, and reading those two as "neither is saturated, +so the kernel is latency-bound" is WRONG. An `n/a` on the byte counter is the +tell that the percentage beside it is derived from something else. + +### What the kernel actually does, measured by sweeping the work + +Concentrating the synthetic router over a controlled pool of experts moves the +occupied block count. `--arm gate_up`, 80 iterations per point: + +| distinct-expert pool | blocks | us/call | us/block | implied GB/s | +|---|---|---|---|---| +| 8 | 13 | 19.9 | 1.53 | -- (fits L2) | +| 16 | 18 | 20.8 | **1.15** | -- (fits L2) | +| 24 | 22 | 65.0 | 2.96 | transition | +| 32 | 27 | 157.0 | 5.81 | 202.9 | +| 40 | 31 | 165.0 | 5.32 | -- | +| 48 | 38 | 207.0 | 5.45 | 216.6 | +| 64 | 45 | 249.5 | 5.55 | 212.7 | +| 128 | 58 | 306.5 | 5.28 | 223.2 | +| 256 | 65 | 339.6 | 5.22 | 225.8 | + +Each block streams one expert's gate_up weights (2N x K/2 = 1 MiB) plus its +scales (2N x K/16 = 128 KiB) = 1179648 B. Two regimes are visible. Under ~18 +blocks the touched weights fit in L2 and cost 1.15 us/block. Above ~27 they +stream, and **us/block is FLAT at 5.2-5.5 across a 2.4x range of work** -- +a constant bytes-per-second, which is the bandwidth-limited signature. + +**So 6z was right and this section's first draft was wrong.** The kernel IS +memory-bound; the standalone plateau is **212.7-225.8 GB/s** over the rows that +are actually flat (>= 38 blocks) -- the widely quoted 203 endpoint is the +27-block row, which sits outside the flat band -- and 6z's derived +in-situ numbers land exactly on it: upstream's **210.7 GB/s is INSIDE the +plateau**, i.e. upstream runs at this kernel's achievable bandwidth, while our +**186.6 GB/s is ~12% BELOW** it. An independent standalone measurement of the +same kernel now corroborates the derived attribution instead of replacing it. + +The one thing 6x should not keep is the per-unit-work DIVISION: with a fixed +144-CTA grid, block count is loop iterations, and at the operating point the +sweep says +1.7 blocks (38.9 -> 40.6) should cost about +4.4%. Upstream carries +those extra iterations and is still faster, which is not a contradiction once the +denominator is bandwidth rather than block count -- it is the same 12-13% +bandwidth-achievement gap seen twice. + +### What is now open + +Ours has NOT been through this harness. That is the decisive experiment: our +kernel, same shapes, same routing input, same box, us/block against the 5.2-5.5 +plateau. If ours plateaus at 5.2-5.5 too, the in-situ deficit is NOT in the +kernel and 6x's localisation is wrong; if ours plateaus ~12% higher, the kernel +owns it and the next question is why the same SASS sustains less bandwidth. + +Occupancy stays a real but SECONDARY lever: 25% capped by a 32 KB shared-memory +budget (`max_shared_mem / blocks_per_sm - 1024`), where under 25600 B would buy +4 blocks/SM. At ~75-80% of this device's ~273 GB/s that is worth far less than +the earlier "~9x of headroom" first draft claimed, which was an artefact of the +same unusable percentage. + +Absolute us/call here is not comparable in-situ: uniform synthetic routing +occupies 61-65 blocks against the model's 38.9-40.6. The static geometry, the +two-regime shape and the plateau bandwidth are comparable, since they do not +depend on the routing draw. + +## 6ae. THE KERNEL IS NOT THE GAP -- 6x's LOCALISATION IS REFUTED (2026-08-13) + +6ad built the upstream arm. This is the arm it was built for: +`benchmarks/marlin_moe_standalone.cpp` drives OUR +`vt::MoeGroupedGemmNvfp4Marlin` through the same shapes, the same expert-pool +control over the block count, and the same gate_up GEMM as the python arm. + +**Sweep, ours against upstream, us/block:** + +| pool | ours blocks / us-per-block | upstream blocks / us-per-block | +|---|---|---| +| 8 | 13 / 1.512 | 13 / 1.534 | +| 16 | 16 / 1.411 | 18 / 1.153 | +| 24 | 23 / 4.773 | 22 / 2.955 | +| 32 | 30 / 5.482 | 27 / 5.813 | +| 40 | 34 / 5.525 | 31 / 5.323 | +| 48 | 39 / 5.311 | 38 / 5.446 | +| 64 | 44 / 5.318 | 45 / 5.545 | +| 128 | 54 / 5.194 | 58 / 5.284 | +| 256 | 63 / 5.261 | 65 / 5.224 | + +Ours plateaus on the SAME 5.2-5.5 band, so ours reaches the same achievable +bandwidth. Two INTERLEAVED paired runs at pool 48 and 128, three reps each, +then settle it: + +| | n | mean us/block | sd | range | +|---|---|---|---|---| +| ours | 12 | **5.3187** | 0.124 | 5.083-5.562 | +| upstream | 12 | **5.3330** | 0.151 | 5.042-5.560 | + +**ours/upstream = 0.9973, i.e. ours 0.27% FASTER, inside one standard deviation +on either side.** The sign of the difference flips between runs. A per-call +workspace memset that our arm pays and upstream's does not was isolated with +`--zero-ws 0` and is noise. + +**So 6x's localisation does not survive.** Its "the SAME kernel, the SAME 1520 +launches, ours 249.22 ms vs upstream 230.39 ms, 8.2% slower inside one kernel" +does NOT reproduce when the same kernel is driven with matched work on the same +box. Normalised by blocks the two are indistinguishable -- and note NORMALISED, not +matched: the two arms draw routing from independent RNG streams (python +`torch.manual_seed`, C++ `mt19937(1234)`), so at the same pool they occupy +different block counts (pool 48: 39 vs 38; pool 128: 54 vs 58). Neither harness +accepts an externally supplied routing tensor, so it cannot yet satisfy the +"force both arms onto one token stream" rule this spec states. At M=9, blocks +track distinct experts and us/block is flat in the streaming regime, so the +ratio is probably sound -- but its error bars carry an undisclosed draw term on +top of the contention term. Correspondingly, the +"12.8% slower per unit of work" and the 186.6-vs-210.7 GB/s reading it produced +describe the in-situ RUNS, not the kernel: both engines reach the same 203-226 +GB/s plateau when asked to do the same thing. + +**What that leaves.** The in-situ difference must come from CONTEXT rather than +from kernel efficiency, and the sweep shows precisely which context term +dominates: time is set by how many DISTINCT EXPERTS a launch touches, from +1.15 us/block at 16 experts (weights fit L2) to ~5.3 above ~27 (streaming) -- +a 4.6x swing that no kernel change causes. The leading hypothesis is therefore +that the two in-situ arms were not touching the same number of distinct experts +per launch, which the recorded 38.9 vs 40.6 blocks hints at but does not +measure, because blocks are not experts. The next measurement is distinct +experts per launch on both arms in situ, not another kernel lever. + +**Caveats, stated rather than buried.** Our arm links `~/work/pr234`'s build, +whose vendored `marlin_mm_moe.cu` is byte-identical to current main +(md5 85c40e4869bc6ec594b8cfb97fb58b3c on both) while its dispatcher predates the +C_tmp cap, which was independently measured perf-neutral at +0.03%. Ours times +with `steady_clock` around 80 iterations plus a final sync; upstream's arm uses +CUDA events. Both amortise launch overhead over 80 calls, and the memset probe +bounds that class of difference at noise. + +## 6af. THE DENOMINATOR WAS WRONG: EXPERTS, NOT BLOCKS (2026-08-13) + +6ae cleared the kernel. This says what the in-situ 8.2% actually measured. + +Holding the block count roughly fixed while varying how many DISTINCT experts +the routing touches, upstream arm, `--arm gate_up`: + +| M | blocks | distinct experts | us/call | us/block | +|---|---|---|---|---| +| 64 | 73 | 20 | 89.4 | 1.22 | +| 64 | 82 | 40 | 227.1 | 2.77 | +| 64 | 94 | 80 | 427.2 | 4.55 | +| 64 | 216 | 216 | 1121.2 | 5.19 | +| 128 | 137 | 20 | 149.9 | 1.09 | +| 128 | 146 | 40 | 220.0 | 1.51 | +| 128 | 255 | 250 | 1298.6 | 5.09 | + +Read the M=128 rows: **137 -> 146 blocks is +6.6% of blocks, and time rises ++46.7%**, because distinct experts went 20 -> 40. + +**But the per-expert cost is NOT flat, and an earlier draft of this section said +it was.** Dividing us/call by distinct gives 4.470, 5.678, 5.340, 5.191, 7.495, +5.500, 5.194 -- a **4.47 to 7.50** range, not 5.2-5.7. The two `distinct=20` +rows are the tell: same expert count, 89.4 vs 149.9 us, while blocks differ +73 vs 137. There, time tracks BLOCKS, which is exactly what a flat-per-expert +model calls irrelevant. + +**So the honest model has two regimes, and it is the same two regimes as 6ad.** +When the touched weights fit L2 (low distinct), the weights are not re-streamed +and cost is set by the per-block work, so time tracks BLOCKS. When they do not +(distinct >= ~40 here), cost is set by streaming and time tracks DISTINCT +EXPERTS at 5.2-5.7 us each. `time ~= distinct x 1.125 MiB / ~215 GB/s` applies +ONLY in the second regime; at distinct=20 it predicts 109.7 us against measured +89.4 and 149.9, off by -19% and +37%. + +The rule that follows is therefore **control BOTH**: a MoE comparison must match +distinct experts AND blocks, or state which regime it is in. Matching one alone +is what makes two runs look comparable when they are not. + +**Applying the model to the recorded in-situ launches requires care, and my +first pass got it backwards.** 6y already measured both sides for the same +prompt: ours 311.2 padded tokens / 38.9 blocks per call, upstream 324.8 / 40.6. +At M=9 those 72 (token, expert) pairs spread over ~39 experts at under 8 tokens +each, so `moe_align` emits ONE block per expert and **blocks and distinct +experts coincide at this shape** -- that measurement was already counting +experts. So upstream touches MORE distinct experts (40.6 vs 38.9) and is still +faster, exactly as 6y concluded. The routing explanation stays refuted; an +earlier draft of this section inferred ours touched ~30 against upstream's ~28 +by inverting the model, which the measured counts in this same file contradict. + +**Which leaves a sharp contradiction, and it is the useful output of this +section.** Standalone at matched work the two kernels are equal to 0.27% (6ae). +In situ ours does LESS work (38.9 against 40.6 expert-blocks) and takes MORE +time (164.0 us against 151.6). Both in-situ arms also beat the standalone +plateau per unit of work (4.21 and 3.73 us against ~5.3). A kernel that is +identical in isolation cannot be slower in place because of its own code, so the +deficit belongs to the CONTEXT the kernel runs in, not to the kernel and not to +the routing. Candidates, in the order their evidence justifies: expert-weight +residency in situ (this repo has already measured a 20-30% per-GEMM penalty for +host/ATS-retagged decode weights, and the standalone arm allocates fresh device +memory that cannot reproduce it), clock and power state across the two runs, and +overlap with concurrent work on other streams. Note also that the in-situ +per-launch times come from summed profiler kernel durations while the standalone +numbers are wall-clock over 80 iterations, so the two bases are not +interchangeable and the ~5.3-to-4.21 difference may be partly that. + +**Consequences.** + +1. The 8.2% "kernel gap" is an artefact of comparing two different routing + draws. It is not evidence of an implementation defect, and it should not be + quoted again. +2. Any future MoE comparison must control DISTINCT EXPERTS PER LAUNCH, or force + both arms onto an identical token stream. Comparing block counts is comparing + the wrong quantity. +3. `VT_MOE_PAD_STATS` (qwen3_5.cpp) counts padded tokens and blocks. It should + count DISTINCT EXPERTS too; that is the number that predicts the time. + +**What this does NOT do.** It does not move the end-to-end ratio. That is +wall-clock on matched prompts and token counts, and it stands at ~0.966. What +changes is that its residual no longer has an attribution: the kernel is cleared +and the number that appeared to localise it was measuring the token path. The +open question is whether OUR token path systematically touches more experts per +step -- a near-tie divergence consequence rather than a defect -- or whether the +remaining wall-clock sits outside the MoE entirely. + +## 6ag. THE MIX, AND THE MODE HOLE UNDER EVERY IN-SITU RATIO (2026-08-13) + +Two arithmetic corrections close this thread out. + +**The mix.** The 1520 in-situ launches are 760 gate_up plus 760 down, and down's +per-expert bytes are exactly HALF gate_up's -- gate_up 2N x K/2 weights plus +2N x K/16 scales = 1.1250 MiB, down K x N/2 plus K x N/16 = 0.5625 MiB, mixed +average 0.8438 MiB per expert-block. 6af compared a MIXED in-situ average against +a gate_up-ONLY standalone plateau, which is what made both arms look like they +beat it (4.21 and 3.73 us against ~5.3). They did not. Redone with the right +bytes: + +| | blocks | MiB / launch | us | implied GB/s | +|---|---|---|---|---| +| ours | 38.9 | 32.8 | 164.0 | **209.9** | +| upstream | 40.6 | 34.3 | 151.6 | **236.9** | + +Measured standalone gate_up plateau: 212.7-225.8 GB/s over the flat rows. +Ours appears INSIDE it and upstream ABOVE it -- **but read the next paragraph +before using that.** + +**THE LOCK CAVEAT, WHICH APPLIES TO EVERY ABSOLUTE NUMBER IN 6ad THROUGH 6ag.** +Those standalone runs took `flock /tmp/gpu.lock`. This box's GPU lock is +`$HOME/gpu.lock`, so they ran UNSERIALISED against concurrent GPU work. Timings +taken under contention are inflated, and a bandwidth computed as bytes/time is +therefore a **LOWER BOUND** on what the kernel achieves, not an upper one. A +re-take under the correct lock can only move the plateau UP -- which would move +our in-situ 209.9 GB/s BELOW it and **reverse the inversion below**. The +"we are not slow, upstream is unusually fast" reading is the LEAST favourable +interpretation to us that the data admits, and it is the one most likely to +change on a clean re-take. Do not build on it until the plateau is re-measured +under `$HOME/gpu.lock`. What survives contention is the INTERLEAVED ratio of +6ae, because contention lands on both arms alike. We run this kernel at the bandwidth it achieves in +isolation; upstream gets something in place the isolated kernel does not. First +candidate is cache reuse across the gate_up/down pair, down's weights being half +size so more of the working set can persist. The framing inverts: on this +evidence we are not slow, upstream is unusually fast in situ. + +**The mode hole.** 6y warns, correctly, that "a work COUNT may be taken under +different execution modes; a TIME may not" -- and then the per-unit-work +normalisation does the thing that warning forbids. The 38.9/40.6 counts were +taken EAGER (ours `VT_SPEC_DECODE_GRAPH=0`, upstream `enforce_eager`, both +because the probes could not survive capture and compile respectively) while the +249.2/230.4 ms times came from the GRAPHED profile. Nothing establishes the +graphed runs had the same blocks per launch as the eager ones. So every ratio +built by dividing those times by those counts -- 4.21 vs 3.73 us/block, 12.8% +per unit of work, 186.6 vs 210.7 GB/s -- has a denominator from a different +execution mode than its numerator. + +**Consequence.** The only like-for-like Marlin comparison in evidence is the +standalone one of 6ae, and it says parity: 0.9973, inside one standard +deviation. Closing the in-situ question needs blocks AND time from the SAME +graphed run, which needs a probe that survives capture: a device-side counter +the launcher increments, read once at the end, never a per-launch D2H sync. + +## 6ah. THE FIRST FULLY-CONTROLLED PAIRED RUN: 0.9889, NOT 0.966 (2026-08-13) + +Every ratio recorded before this one was taken with a COLD leading arm. +Correcting that moves the headline materially. + +**Four controls, all present together for the first time.** `$HOME/gpu.lock` +(the standalone runs of 6ad-6af took `/tmp/gpu.lock`, which coordinates with +nothing). A DISCARDED warm-up arm ahead of the first measured arm, because the +GB10 SM clock ramps over MINUTES -- 1449 to 2190 MHz observed across one run -- +so dropping rep 1 is not enough and an entire first arm reads ~6% low. Settle +barriers between arms, because vLLM asserts free GPU memory does not GROW during +its startup profile while GB10 returns our engine's pages lazily, which killed +every earlier paired attempt with "Initial free memory 68.53 GiB, current free +memory 89.42 GiB". And a host-RAM headroom guard before the oracle, because +`gpu_memory_utilization` reserves HOST RAM here, so an oracle without headroom +takes the MACHINE down -- three reboots on 2026-08-13, at least the last ours. + +| arm | n | median tok/s | range | +|---|---|---|---| +| ours BEFORE | 9 | 142.604 | 141.67-142.79 | +| ours AFTER | 9 | 142.140 | 135.23-142.87 | +| ours combined | 18 | **142.534** | -- | +| oracle | 15 | **144.130** | 141.86-151.84 | + +Drift before -> after is **-0.33%**, inside the 1% gate this harness sets for +itself, so the run counts. Ratios: 0.9894 before, 0.9862 after, +**0.9889 combined**. + +**The gap is 1.1%, not 3.4%.** The 0.9757 / 0.9646 / 0.9569 of 6ac were +measuring an unwarmed first arm as much as they were measuring the engine. + +This is NOT parity: 0.9889 is below 1.0 and the row stays open. But it changes +what remains to be found -- roughly a third of the deficit the record has been +chasing all campaign, and it lands after 6ae had already cleared the kernel that +deficit was attributed to. + +**Caveats, kept rather than buried.** n=1 paired run. **The repeat RAN and was +REJECTED by this harness's own drift gate**: its arms measured 146.740 before +and 143.619 after, a -2.13% drift against a 1% gate, so its 0.9795 does not +count and is not averaged in. Bracketing on its own arms it spans 0.9912 to +0.9701 -- it cannot separate parity from a 3% deficit. Two things it did +establish: the oracle's BIMODALITY IS BOOT-DEPENDENT (run 1 unimodal at ~144, +run 2 clearly bimodal with 10 draws at 148.05 and 5 at 156.61, same script and +pin), so a harness cannot assume either shape; and absolutes moved together +across the reboot (ours +1.8%, oracle +2.8%), consistent with the recorded 12.8% +boot-to-boot clock variation. So the standing claim is **~0.98, NOT parity, on +one valid paired run** -- not "0.9889 confirmed". The ours-AFTER arm carries four low outliers (135.2, 135.5, +139.0, 139.9) that the BEFORE arm does not, so something touched the box during +it -- medians are unaffected, that arm's spread is not trustworthy. The oracle +shows ONE fast draw (151.84) against fourteen near 144, the old bimodality +appearing once, absorbed by the median. The oracle copy runs +`gpu_memory_utilization=0.35` rather than 0.55; at max_num_seqs=2 and +max_model_len=2048 the KV cache needed is a fraction of either, so it cannot +change decode speed, but it IS a config delta on the denominator and is recorded +as one. ## 7. Evidence, authority, stop conditions diff --git a/benchmarks/marlin_moe_standalone.cpp b/benchmarks/marlin_moe_standalone.cpp new file mode 100644 index 000000000..ce29f6449 --- /dev/null +++ b/benchmarks/marlin_moe_standalone.cpp @@ -0,0 +1,213 @@ +// OUR arm of the #442 standalone Marlin harness. +// +// Mirrors scripts/marlin-moe-standalone.py exactly: same 35B-A3B decode shapes +// (hidden 2048, moe_intermediate 512, E=256, top_k=8, moe_block_size 8), same +// gate_up GEMM, same expert-pool control over the occupied block count. Prints +// us/call and us/block so our plateau can be laid against upstream's 5.2-5.5. +// +// Not a test: no assertions, no goldens. It measures the kernel only. +// +// RUN IT UNDER THE BOX LOCK: `flock $HOME/gpu.lock ...`, NOT /tmp/gpu.lock, +// which coordinates with nothing. `nvidia-smi` showing no compute apps does +// not mean the GPU is unreserved, so check `fuser -v $HOME/gpu.lock` first. +// Absolute timings taken unlocked are upper bounds; only interleaved RATIOS +// survive contention. +// +// NOT WIRED INTO ANY BUILD TARGET (#442). Nothing compiles this file, so it +// carries no -Werror and no CI, and it will rot against +// vt::MoeGroupedGemmNvfp4Marlin's signature. The recorded measurements were +// taken from an out-of-tree build. Wiring it into examples/CMakeLists.txt the +// way benchmarks/vulkan_gemm_ab.cpp is wired is owed. +// +// Its routing RNG is a DIFFERENT stream from the python arm's, so the two +// arms occupy different block counts at the same --experts pool. Comparisons +// between them are NORMALISED by blocks, not matched on them; neither arm can +// yet take an externally supplied routing tensor. + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vt/backend.h" +#include "vt/cuda/marlin_repack.h" +#include "vt/dtype.h" +#include "vt/ops.h" + +namespace { + +using vt::Backend; +using vt::Device; +using vt::DeviceType; +using vt::DType; +using vt::Queue; +using vt::Tensor; + +Device Gpu() { return Device{DeviceType::kCUDA, 0}; } + +Tensor MakeT(void* data, DType dt, Device dev, const std::vector& shape) { + Tensor t; + t.data = data; + t.dtype = dt; + t.device = dev; + t.rank = static_cast(shape.size()); + int64_t stride = 1; + for (int i = t.rank - 1; i >= 0; --i) { + t.shape[i] = shape[static_cast(i)]; + t.stride[i] = stride; + stride *= shape[static_cast(i)]; + } + return t; +} + +class Dev { + public: + Dev(Backend& b, Queue& q, DType dt, const std::vector& shape, + const void* host = nullptr) + : b_(b) { + int64_t numel = 1; + for (auto s : shape) numel *= s; + bytes_ = static_cast(numel) * vt::SizeOf(dt); + p_ = b_.Alloc(bytes_ == 0 ? 1 : bytes_); + if (host != nullptr) b_.Copy(q, p_, host, bytes_); + t_ = MakeT(p_, dt, Gpu(), shape); + } + ~Dev() { b_.Free(p_); } + Dev(const Dev&) = delete; + Dev& operator=(const Dev&) = delete; + Tensor& tensor() { return t_; } + void* ptr() { return p_; } + + private: + Backend& b_; + void* p_ = nullptr; + size_t bytes_ = 0; + Tensor t_; +}; + +int IntArg(int argc, char** argv, const char* name, int fallback) { + for (int i = 1; i + 1 < argc; ++i) + if (std::strcmp(argv[i], name) == 0) return std::atoi(argv[i + 1]); + return fallback; +} + +} // namespace + +int main(int argc, char** argv) { + const int pool_arg = IntArg(argc, argv, "--experts", 0); + const int iters = IntArg(argc, argv, "--iters", 80); + const int warmup = IntArg(argc, argv, "--warmup", 20); + const int M = IntArg(argc, argv, "--m", 9); + const int zero_ws = IntArg(argc, argv, "--zero-ws", 1); + + const int E = 256, K = 2048, N = 512, top_k = 8; + const int pool = pool_arg > 0 ? pool_arg : E; + const int size_n = 2 * N; // gate_up + const int size_k = K; + + Backend& b = vt::GetBackend(DeviceType::kCUDA); + Queue q{Gpu(), nullptr}; + void* stream = nullptr; + const int dev_id = 0; + + // Weights: random packed nibbles, repacked per expert into Marlin layout. + // Marlin's runtime is data independent, so random bits time like real ones. + std::mt19937 rng(1234); + const size_t raw_bytes = static_cast(size_n) * size_k / 2; + std::vector raw(raw_bytes); + for (auto& x : raw) x = static_cast(rng() & 0xFF); + const size_t scale_bytes = static_cast(size_n) * size_k / 16; + std::vector raw_s(scale_bytes); + for (auto& x : raw_s) x = 0x38; // fp8-e4m3 ~ 0.5, safely positive + + Dev staging(b, q, DType::kI8, {size_n, size_k / 2}, raw.data()); + Dev staging_s(b, q, DType::kI8, {size_n, size_k / 16}, raw_s.data()); + + Dev wq(b, q, DType::kI32, {E, size_k / 16, size_n * 2}); + Dev sc(b, q, DType::kI8, {E, size_k / 16, size_n}); + const float sf = 1.0f; + std::vector gs(static_cast(E), + vt::cuda::MarlinNvfp4ProcessGlobalScale(1.0f, sf)); + + const size_t wq_expert_words = static_cast(size_n) * size_k / 2 / 4; + for (int e = 0; e < E; ++e) { + vt::cuda::MarlinRepackExpertWeight( + stream, dev_id, + static_cast(wq.ptr()) + static_cast(e) * wq_expert_words, + static_cast(staging.ptr()), size_k, size_n); + vt::cuda::MarlinProcessExpertScales( + stream, static_cast(staging_s.ptr()), + static_cast(sc.ptr()) + static_cast(e) * scale_bytes, + size_k, size_n, sf); + } + b.Synchronize(q); + Dev dgs(b, q, DType::kF32, {E}, gs.data()); + + // Routing, drawn from `pool` distinct experts -- the block-count control. + const int P = M * top_k; + std::vector topk_ids(static_cast(P)); + std::vector topk_w(static_cast(P), 1.0f); + for (int i = 0; i < P; ++i) + topk_ids[static_cast(i)] = static_cast(rng() % static_cast(pool)); + + const int block = vt::cuda::MarlinMoeAlignBlockSizeSelect(M, top_k, E); + int max_tok = 0, max_blk = 0; + vt::cuda::MarlinMoeAlignSizes(M, top_k, E, block, &max_tok, &max_blk); + Dev dtid(b, q, DType::kI32, {M, top_k}, topk_ids.data()); + Dev dtw(b, q, DType::kF32, {M, top_k}, topk_w.data()); + Dev sorted_ids(b, q, DType::kI32, {max_tok}); + Dev expert_ids(b, q, DType::kI32, {max_blk}); + Dev num_pad(b, q, DType::kI32, {1}); + vt::cuda::MarlinMoeAlignBlockSize(stream, static_cast(dtid.ptr()), M, + top_k, E, block, + static_cast(sorted_ids.ptr()), + static_cast(expert_ids.ptr()), + static_cast(num_pad.ptr())); + b.Synchronize(q); + int32_t past = 0; + b.Copy(q, &past, num_pad.ptr(), sizeof(int32_t)); + b.Synchronize(q); + + const int sms = vt::cuda::MarlinDeviceSms(dev_id); + Dev ws(b, q, DType::kI32, {sms * 4}); + Dev dact(b, q, DType::kBF16, {M, K}); + Dev dout(b, q, DType::kBF16, {P, size_n}); + + vt::MoeMarlinArgs args{}; + args.moe_block_size = block; + args.top_k = top_k; + args.size_m = M; + args.size_n = size_n; + args.size_k = size_k; + args.mul_topk_weights = false; + + // vLLM's arm does NOT re-zero the workspace per call (the kernel leaves it + // reset), so timing ours WITH a per-call memset adds a launch upstream never + // pays. --zero-ws 0 removes that asymmetry. + b.Memset(q, ws.ptr(), 0, static_cast(sms) * 4 * sizeof(int32_t)); + auto once = [&]() { + if (zero_ws) b.Memset(q, ws.ptr(), 0, static_cast(sms) * 4 * sizeof(int32_t)); + vt::MoeGroupedGemmNvfp4Marlin(q, dout.tensor(), dact.tensor(), wq.tensor(), + sc.tensor(), dgs.tensor(), ws.tensor(), + sorted_ids.tensor(), expert_ids.tensor(), + num_pad.tensor(), dtw.tensor(), args); + }; + + for (int i = 0; i < warmup; ++i) once(); + b.Synchronize(q); + const auto t0 = std::chrono::steady_clock::now(); + for (int i = 0; i < iters; ++i) once(); + b.Synchronize(q); + const auto t1 = std::chrono::steady_clock::now(); + + const double us = + std::chrono::duration(t1 - t0).count() / iters; + const int blocks = past / block; + std::printf("OURS gate_up M=%d pool=%d zero_ws=%d blocks=%d us_per_call=%.3f us_per_block=%.4f\n", + M, pool, zero_ws, blocks, us, us / (blocks > 0 ? blocks : 1)); + return 0; +} diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 8ce7e32ae..406369cc1 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -351,7 +351,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record. | MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` | | DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` | | n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` | -| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **~0.966x +/- 0.01** over three within-session pairs; C_tmp cap perf-NEUTRAL; storage refuted (#442) | `ACTIVE` | +| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE 35B-A3B **~0.98x** of the pinned graphed oracle: one VALID controlled paired run at 0.9889 (drift -0.33%), repeat REJECTED on a -2.13% drift gate. NOT parity (#442) | `ACTIVE` | | Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` | ## How we measure diff --git a/docs/STATUS.md b/docs/STATUS.md index c93061da2..83397173b 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -528,25 +528,85 @@ MHz, 30 reps, drift bracketed at -0.088%, the oracle's non-modal draws excluded) the code cell is **0.975x with NON-OVERLAPPING distributions** — a real gap, not noise, and the earlier "within resolution" reading was too generous. Ours slowed more than the oracle when the clock was pinned, so the residual is -SM-clock-sensitive work. PAIRED profiling localises it exactly: the SAME -`marlin_moe_wna16::Marlin` kernel, the SAME 1520 launches, ours 249.22 ms vs -upstream 230.39 ms -- **8.2% slower inside one kernel**, which at ~34% of wall is -2.8% end-to-end and accounts for the whole measured 2.5%. Not an algorithm difference, and not the launch -geometry either: the full template arguments match, `determine_exec_config` is -byte-identical to the pinned upstream copy, and every OTHER kernel matches to -0.2%. The inputs match too (scale bytes per expert, -256-byte alignment, cudaMalloc residency), and the work counts were MEASURED: -upstream loops 4.4% MORE blocks per launch (40.6 vs 38.9) and is still faster, so -routing is refuted and normalising by work makes our deficit bigger -- **4.21 vs -3.73 us per block, ~12.8% slower per unit of work**. Every source-level explanation is now -eliminated -- kernel source, template instantiation, grid config, block size, -shared-memory budget, reduction flags, scale layout, alignment, residency, CUDA -toolkit (13.0 both) and arch all match -- and `ncu` plus cuobjdump then showed the -COMPILED KERNELS ARE EQUIVALENT (94 registers and 3664 SASS instructions on both, -upstream running its family-compatible sm_120 cubin against our sm_121a). The -residual is therefore runtime and is now ATTRIBUTED: the kernel is DRAM-bound -(L2 hit 9.5%) and we sustain **186.6 GB/s against upstream's 210.7**, a 12.9% -effective-bandwidth gap that IS the whole per-unit-work difference. Weight +SM-clock-sensitive work. PAIRED profiling appeared to localise it to `marlin_moe_wna16::Marlin` (ours +249.22 ms vs upstream 230.39 ms over the same 1520 launches), and every +source-level explanation was eliminated -- kernel source, template +instantiation, grid, block size, shared memory, flags, scale layout, +alignment, residency, toolkit, arch -- with `ncu` and cuobjdump showing the +compiled kernels EQUIVALENT (94 registers, 3664 SASS instructions on +both). **That localisation is now REFUTED.** `scripts/marlin-moe-standalone.py` and +`benchmarks/marlin_moe_standalone.cpp` drive each engine's own kernel outside +its engine -- which is also what finally lets `ncu` attach to upstream, the +blocker recorded as impossible in both replay modes -- and at matched work the +two are indistinguishable: over 12 interleaved paired points ours +averages **5.3187 us/block against upstream's 5.3330**, ratio **0.9973**, sign flipping +between runs, inside one standard deviation either way. The in-situ 8.2% +therefore describes the RUNS, not the kernel, and so do the 12.8%-per-unit- +work and 186.6-vs-210.7 GB/s figures derived from it. What the harness does +establish is the kernel's shape: a persistent single wave (grid 144 = 48 SMs x +3 blocks, 32768 B shared against 102400 B/SM, 25% occupancy), a bandwidth- +limited plateau of 203-226 GB/s that BOTH engines reach, and a 4.6x swing +driven by how many DISTINCT EXPERTS a launch touches (1.15 us/block at 16, +weights in L2; ~5.3 above ~27, streaming). Blocks are not experts, and holding blocks fixed while varying distinct +experts settles it: at M=128, going 137 to 146 blocks (+6.6%) while distinct +experts doubled cost +46.7% TIME, and cost per DISTINCT EXPERT is flat at +5.2-5.7 us across the table while cost per block varies 4.7x. Time is +distinct_experts x 1.125 MiB / ~215 GB/s and blocks are nearly irrelevant, so +every in-situ comparison normalised by the wrong quantity. At the M=9 decode shape blocks and distinct experts COINCIDE (72 pairs over +~39 experts, under 8 each, one block per expert), so the both-sides count +already measured experts: ours 38.9 against upstream's 40.6. That leaves a +sharp contradiction rather than an explanation. Standalone at matched work the +kernels are equal to 0.27%; in situ ours does LESS work and takes MORE time +(164.0 us against 151.6). A kernel identical in isolation cannot be slower in +place because of its own code, so the deficit belongs to the CONTEXT, not the +kernel and not the routing. Candidates in evidence order: expert-weight +residency in situ, where this repo has already measured 20-30% per GEMM for +host/ATS-retagged decode weights and the standalone arm's fresh cudaMalloc +cannot reproduce it; clock and power state across runs; and overlap with +concurrent stream work. Two arithmetic corrections then close it out. The 1520 in-situ launches are +760 gate_up plus 760 down, and down's per-expert bytes are exactly half, so +the mixed average is 0.8438 MiB per expert-block; comparing that against a +gate_up-only standalone plateau is what made both arms look like they beat it. +Redone correctly, ours implies 209.9 GB/s and upstream 236.9 against a +measured 203-226 plateau, so OURS SITS INSIDE IT AND UPSTREAM ABOVE IT: we run +this kernel at the bandwidth it achieves in isolation and upstream gets +something in place that the isolated kernel does not, cache reuse across the +gate_up/down pair being the first candidate. Second, every in-situ per-unit- +work ratio has a mode hole: the 38.9/40.6 block counts were taken EAGER +because the probes could not survive capture, while the 249.2/230.4 ms times +came from the GRAPHED profile, so numerator and denominator are from different +execution modes. The only like-for-like Marlin comparison in evidence is +therefore the standalone one, and it says parity. Closing the in-situ question +needs blocks and time from the SAME graphed run, via a device-side counter +read once at the end rather than a per-launch D2H sync. None of this moves the +end-to-end ratio, which is wall-clock on matched prompts. The kernel's one +settable occupancy knob, blocks_per_sm, was swept and is DEAD: with routing +SEEDED the between-configuration spread (4.3%) is no larger than the within- +configuration spread, and an apparent 8.7% win came from an unseeded pass +where the routing draw moved. THE HEADLINE THEN CHANGED. Every earlier ratio +was taken with a COLD leading arm. The first run with all four controls +present -- the correct $HOME/gpu.lock, a DISCARDED warm-up arm (the GB10 SM +clock ramps over minutes, 1449 to 2190 MHz, so dropping rep 1 leaves a whole +arm ~6% low), settle barriers (vLLM asserts free GPU memory does not grow +during its startup profile, and GB10 releases our pages lazily, which killed +every earlier paired attempt), and a host-RAM headroom guard +(gpu_memory_utilization reserves HOST RAM here, which took the machine down +three times on 2026-08-13) -- measures ours at 142.534 tok/s against the oracle's 144.130, a ratio +of **0.9889** with before/after drift of -0.33%, inside the 1% validity gate. So +the gap is ~1%, not 3.4%: the 0.9757/0.9646/0.9569 recorded earlier were +measuring an unwarmed first arm as much as the engine. The n=2 repeat RAN and +was REJECTED by the same gate at -2.13% drift, so its 0.9795 is not averaged +in and the standing claim is ~0.98 on ONE valid paired run, still NOT parity. +The repeat did establish two things: the oracle's bimodality is BOOT-DEPENDENT +(unimodal near 144 in one run, 10-at-148 plus 5-at-157 in the next, same +script and pin), so no harness may assume either shape; and both arms' +absolutes moved together across the reboot, ours +1.8% and oracle +2.8%, +matching the recorded 12.8% boot-to-boot clock variation. The binding +constraint on resolving 1% here is the box, which rebooted or dropped five +times on 2026-08-13. Two traps worth carrying: dram__bytes.sum reads n/a on GB10, so +ncu's Memory Throughput % excludes DRAM traffic; and fuser -v $HOME/gpu.lock +is the check, because nvidia-smi showing no compute apps does not mean the GPU +is unreserved. Weight residency is already staged correctly (cudaMalloc + one upload), and the slab itself is byte-for-byte the same size and stride as upstream's tensor (268 MB, no padding), so the cause is memory-system behaviour that no allocation change we can name would alter; upstream's ncu counters would settle it but its engine will not initialise under diff --git a/scripts/marlin-moe-standalone.py b/scripts/marlin-moe-standalone.py new file mode 100755 index 000000000..e65da6844 --- /dev/null +++ b/scripts/marlin-moe-standalone.py @@ -0,0 +1,161 @@ +#!/usr/bin/env python3 +"""Standalone upstream Marlin MoE driver for #442. + +Drives vLLM's OWN `moe_wna16_marlin_gemm` on the 35B-A3B decode shapes with NO +EngineCore, NO multiprocessing and NO model load, so that `ncu` can attach -- +which is the blocker recorded in the spec (upstream's engine will not initialise +under ncu in either replay mode). + +Shapes come from nvidia/Qwen3.6-35B-A3B-NVFP4: hidden 2048, moe_intermediate +512, E=256, top_k=8. Marlin's runtime is data-independent, so random weights +measure the same kernel the model does. + +RUN IT UNDER THE BOX LOCK: `flock $HOME/gpu.lock ...`, NOT /tmp/gpu.lock, which +coordinates with nothing. `nvidia-smi` reporting no compute apps does not mean +the GPU is unreserved -- a holder may be between phases -- so check +`fuser -v $HOME/gpu.lock` first. Absolute timings taken unlocked are upper +bounds; only interleaved RATIOS survive contention. +""" + +import argparse +import os +import sys + +import torch + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--m", type=int, default=9, help="tokens (1 + k spec draft)") + ap.add_argument("--iters", type=int, default=80) # matches the C++ arm + ap.add_argument("--warmup", type=int, default=20) + ap.add_argument("--arm", choices=["gate_up", "down", "both"], default="both") + ap.add_argument("--bps", type=int, default=-1, + help="blocks_per_sm passed to the kernel (-1 = auto)") + ap.add_argument("--experts", type=int, default=0, + help="draw routing from only this many distinct experts " + "(0 = all E); controls occupied block count") + args = ap.parse_args() + + import vllm + from vllm.model_executor.layers.quantization.utils.marlin_utils import ( + marlin_make_workspace_new, + ) + from vllm.model_executor.layers.quantization.utils import marlin_utils_fp4 as f4 + from vllm.model_executor.layers.fused_moe.moe_align_block_size import ( + moe_align_block_size, + ) + from vllm.scalar_type import scalar_types + + # ORACLE IDENTITY. A rollback venv silently invalidates every number here; + # this repo has already measured a rollback three times believing it was the + # pin. Abort rather than measure the wrong build. + want = "555967922" + if want not in vllm.__version__: + print(f"ABORT: vllm {vllm.__version__} is not the pinned {want}") + return 2 + print(f"vllm={vllm.__version__} torch={torch.__version__}", flush=True) + + dev = torch.device("cuda") + torch.cuda.init() + + E, K, N, TOPK = 256, 2048, 512, 8 + GROUP = 16 + BLOCK_M = 8 + M = args.m + dtype = torch.bfloat16 + quant_type = scalar_types.float4_e2m1f + + perm = torch.empty(0, dtype=torch.int, device=dev) + + def build(size_n: int, size_k: int): + """Repacked weights + processed scales for one grouped GEMM. + + Mirrors prepare_nvfp4_moe_layer_for_marlin's permute_scales exactly -- + per-expert marlin_permute_scales on the transposed scale, one shared + scale_factor across experts, then the S0E5M3 conversion. + """ + w = torch.randint(0, 255, (E, size_n, size_k // 2), dtype=torch.uint8, + device=dev) + wq = f4._repack_marlin_experts(w, size_n, size_k, perm, False) + + # Positive scales, as NVFP4 weight scales always are. + s = (torch.rand((E, size_n, size_k // GROUP), dtype=dtype, device=dev) + * 0.5 + 0.5) + factor = f4._nvfp4_compute_scale_factor(s, dtype) + per_expert = [] + for i in range(E): + ms = f4.marlin_permute_scales(s=s[i].T, size_k=size_k, + size_n=size_n, group_size=GROUP, + is_a_8bit=False) + ms, _ = f4.nvfp4_marlin_process_scales(ms, scale_factor=factor, + a_dtype=dtype) + per_expert.append(ms) + s_proc = torch.cat([x.unsqueeze(0) for x in per_expert], 0) + + s2 = torch.ones(E, dtype=torch.float32, device=dev) + g_proc = f4.nvfp4_marlin_process_global_scale(s2, dtype) / factor + return wq, s_proc, g_proc + + print("building weights...", flush=True) + w1, w1s, w1g = build(2 * N, K) # gate_up: size_n = 2*512, size_k = 2048 + w2, w2s, w2g = build(K, N) # down: size_n = 2048, size_k = 512 + workspace = marlin_make_workspace_new(dev, 4) + + pool = args.experts if args.experts > 0 else E + # SEEDED: routing dominates the time, so every configuration under + # comparison must see the IDENTICAL draw or the sweep measures the draw. + torch.manual_seed(20260813) + topk_ids = torch.randint(0, pool, (M, TOPK), dtype=torch.int32, device=dev) + topk_w = torch.rand((M, TOPK), dtype=torch.float32, device=dev) + sorted_ids, expert_ids, num_past = moe_align_block_size( + topk_ids, BLOCK_M, E, None) + distinct = int(torch.unique(topk_ids).numel()) + + a1 = torch.randn((M, K), dtype=dtype, device=dev) + c1 = torch.empty((M * TOPK, 2 * N), dtype=dtype, device=dev) + a2 = torch.randn((M * TOPK, N), dtype=dtype, device=dev) + c2 = torch.empty((M * TOPK, K), dtype=dtype, device=dev) + + ops = torch.ops._moe_C + + def gate_up(): + return ops.moe_wna16_marlin_gemm( + a1, c1, w1, None, w1s, None, w1g, None, None, None, workspace, + sorted_ids, expert_ids, num_past, topk_w, + BLOCK_M, TOPK, False, quant_type.id, M, 2 * N, K, + True, False, True, False, -1, -1, args.bps) + + def down(): + return ops.moe_wna16_marlin_gemm( + a2, c2, w2, None, w2s, None, w2g, None, None, None, workspace, + sorted_ids, expert_ids, num_past, topk_w, + BLOCK_M, 1, True, quant_type.id, M * TOPK, K, N, + True, False, True, False, -1, -1, args.bps) + + arms = {"gate_up": gate_up, "down": down} + if args.arm != "both": + arms = {args.arm: arms[args.arm]} + + for name, fn in arms.items(): + for _ in range(args.warmup): + fn() + torch.cuda.synchronize() + st, en = torch.cuda.Event(True), torch.cuda.Event(True) + st.record() + for _ in range(args.iters): + fn() + en.record() + torch.cuda.synchronize() + us = st.elapsed_time(en) * 1000.0 / args.iters + blocks = int(num_past.item()) // BLOCK_M + print(f"UPSTREAM {name} bps={args.bps} M={M} pool={pool} distinct={distinct} blocks={blocks} " + f"us_per_call={us:.3f} us_per_block={us/max(blocks,1):.4f}", + flush=True) + + print("DONE", flush=True) + return 0 + + +if __name__ == "__main__": + sys.exit(main())