perf(dspark): unblock upstream ncu; the DRAM attribution is CORROBORATED (#442) - #562
Open
localai-bot wants to merge 15 commits into
Open
perf(dspark): unblock upstream ncu; the DRAM attribution is CORROBORATED (#442)#562localai-bot wants to merge 15 commits into
localai-bot wants to merge 15 commits into
Conversation
…ions (#442) The developer asked whether the weights sitting on NAS, or not being fully resident, could be distorting these measurements. Tested and refuted: the weights are on local NVMe (no NAS mount exists on the box), a run reads 22.06 GB once at load, decode-time RSS is 4.8 GB because the mapping is released after upload, and 8 warm reps hold a 0.5% spread. File-backed weights could not produce that stability, and on GB10 the failure mode would be loud anyway -- host pages reach the GPU through ATS at a measured 20-30% per-GEMM penalty. Recorded alongside it: that NVMe is 98% full (76 GB free). This repo has already lost a gate run to ENOSPC reporting green over work that never ran. Also records a third WITHIN-session paired ratio, ours -> oracle -> ours at free clocks with drift bracketed at -0.89%: 140.98 vs a modal 147.32, RATIO 0.9569. With the two pinned-clock pairs (0.9757, 0.9646) that puts the gap at ~0.966 +/- 0.01, consistently below 1.0, while the ABSOLUTE numbers move up to 5% between sessions for the same binary because GB10's memory clock cannot be pinned. Only the within-session ratio is quotable, and all three agree. The oracle's draws remain bimodal (~147.3 vs ~155.6) from the same one-extra- accepted-token effect, so its modal draws stay the honest denominator. No code change; this is the record catching up with the measurements. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Picks up the hf_snapshot.h build repair (#556) so this record-only PR builds. Its CPU, sanitizer and Windows-compile jobs were red for that reason alone -- the branch changes four markdown files and cannot affect a build. No overlap: #556 touches tests/parity/hf_snapshot.h, this branch touches the DSpark spec, the benchmark record, STATUS and BENCHMARKS. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…tion (#442) Section 6z listed upstream's ncu counters as the last open route and recorded it BLOCKED: vLLM's EngineCore will not initialise under ncu in either replay mode, so its DRAM efficiency stayed DERIVED rather than counted, and the spec named a standalone harness as the only remaining way in. scripts/marlin-moe-standalone.py is that harness. It drives upstream's own torch.ops._moe_C.moe_wna16_marlin_gemm on the 35B-A3B decode shapes -- hidden 2048, moe_intermediate 512, E=256, top_k=8, moe_block_size 8 -- with no EngineCore, no multiprocessing and no model load, mirroring prepare_nvfp4_moe_layer_for_marlin's scale pipeline exactly. It asserts the oracle identity and aborts on mismatch, because this repo has three times measured a rollback venv believing it was the pin. It profiles. And the counters refute what 6z concluded from them: Memory Throughput 11.14% of peak Compute (SM) 11.42% of peak Occupancy 25% theoretical / 25.98% achieved Block limit shared memory = 3 Waves per SM 1 grid / block / shmem 144 / 128 / 32768 B Both throughputs at ~11% is the LATENCY-bound signature. A kernel at 11% of memory peak is not against a bandwidth wall, so "we sustain 186.6 GB/s against upstream's 210.7, and that IS the per-unit-work difference" does not hold -- those numbers were time x analytic bytes, so they restated the time gap instead of explaining it. The geometry shows the real limiter. GB10 has 48 SMs with 102400 B of shared memory each; the kernel takes 32768 B per block, giving 3 blocks/SM and 25% occupancy, and 48 x 3 = 144 is exactly the grid. It is a persistent single wave. So the per-unit-work normalisation in 6x is also invalid: 38.9 vs 40.6 "blocks per launch" are loop iterations inside a FIXED 144-CTA grid, not parallel work, and cost is set by the CTA with the most work. ncu flags exactly that, plus uncoalesced access at 20.2 of 32 bytes per load sector. What does NOT change: the 3.4% gap itself. Our kernel through this same harness on identical routing is not yet run, and no claim about the gap may rest on an upstream-only measurement. What changes is its attribution. What this opens: ~9x of absolute headroom that BOTH engines leave on the table, since occupancy is capped purely by a shared-memory budget Marlin computes as max_shared_mem / blocks_per_sm - 1024. Under 25600 B would allow 4 blocks/SM. Not a parity lever -- upstream pays it identically -- a best-in-class one. The synthetic router draws uniformly over 256 experts and so occupies 61 blocks against the model's 38.9-40.6; absolute us/call from this harness is therefore NOT comparable in-situ, while the static geometry and throughput percentages are. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…tion stands (#442) The previous commit on this branch read ncu's SpeedOfLight line -- "Memory Throughput 11.14%" beside "Compute (SM) Throughput 11.42%" -- as neither resource being saturated, and concluded the Marlin kernel is latency-bound and 6z's DRAM attribution refuted. That is wrong, and the tell was in the same output: dram__bytes.sum reads n/a on GB10. The chip exposes no DRAM counters, so that memory percentage EXCLUDES DRAM traffic and cannot support any claim about being far from a bandwidth wall. Sweeping the occupied block count settles it without those counters. Under ~18 blocks the touched expert weights fit in L2 and cost 1.15 us/block; above ~27 they stream and us/block is FLAT at 5.2-5.5 across a 2.4x range of work. Constant bytes per second across varying work is the bandwidth-limited signature. At 1179648 B per block (1 MiB of weights + 128 KiB of scales) that plateau is 203-226 GB/s. 6z's derived in-situ numbers land ON it. Upstream's 210.7 GB/s is INSIDE the plateau, so upstream runs at this kernel's achievable bandwidth; ours at 186.6 GB/s is ~12% below. The standalone harness CORROBORATES the attribution it was built to test rather than overturning it. Two things do change. 6x's per-unit-work division does not survive a fixed 144-CTA persistent grid, where block count is loop iterations and the sweep prices 38.9 -> 40.6 blocks at about +4.4%. And the "~9x of absolute headroom" claimed by the retracted commit was an artefact of the same unusable percentage: at ~75-80% of this device's ~273 GB/s, occupancy is a secondary lever. The geometry findings are unaffected -- they are static properties, not counter-derived: grid 144 = 48 SMs x 3 blocks, 32768 B shared per block against 102400 B per SM, 25% occupancy, one persistent wave. Corrected in place rather than force-pushed, so the branch keeps both the wrong reading and its retraction. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
changed the title
…EFUTED (#442) 6ad built the upstream arm of the standalone harness. This is the arm it was built for. benchmarks/marlin_moe_standalone.cpp drives OUR vt::MoeGroupedGemmNvfp4Marlin through the same 35B-A3B gate_up shapes, the same expert-pool control over the occupied block count, and the same GEMM as the python arm, so the two kernels can be asked to do the same work on the same box. Ours plateaus on the SAME 5.2-5.5 us/block band as upstream. Two INTERLEAVED paired runs at pool 48 and 128, three reps each: n mean us/block sd range ours 12 5.3187 0.124 5.083-5.562 upstream 12 5.3330 0.151 5.042-5.560 ours/upstream = 0.9973, ours 0.27% FASTER, inside one standard deviation on either side, with the sign of the difference flipping between runs. A per-call workspace memset our arm pays and upstream's does not was isolated behind --zero-ws 0 and is noise. So 6x does not survive. Its "the SAME kernel, the SAME 1520 launches, ours 249.22 ms vs upstream 230.39 ms, 8.2% slower inside one kernel" does NOT reproduce when both kernels are driven with matched work. The "12.8% slower per unit of work" and the 186.6-vs-210.7 GB/s reading derived from it therefore describe the in-situ RUNS, not the kernel: both engines reach the same 203-226 GB/s plateau when asked to do the same thing. What the harness does establish, and it points somewhere new: time is set by how many DISTINCT EXPERTS a launch touches, 1.15 us/block at 16 experts where the weights fit L2, rising to ~5.3 above ~27 where they stream. That is a 4.6x swing no kernel change causes. Blocks are not experts, so the recorded 38.9 vs 40.6 blocks per launch never settled which arm did more work. Measuring distinct experts per launch on both arms IN SITU is the next step, not another kernel lever. docs/STATUS.md's DSpark narrative is collapsed onto the binding result rather than extended, which is what the ratchet asks for when a finding supersedes the story that preceded it. Caveats, stated rather than buried: our arm links ~/work/pr234's build, whose vendored marlin_mm_moe.cu is byte-identical to current main (md5 85c40e4869bc6ec594b8cfb97fb58b3c) while its dispatcher predates the C_tmp cap, independently measured perf-neutral at +0.03%. Ours times with steady_clock over 80 iterations plus a final sync, upstream's with CUDA events; both amortise launch overhead, and the memset probe bounds that class of difference at noise. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
#442) 6ae cleared the kernel by showing ours and upstream's Marlin are the same to 0.27% at matched work. This says what the in-situ 8.2% was actually measuring. Holding the block count roughly fixed while varying how many DISTINCT experts the routing touches: M blocks distinct us/call us/block 64 73 20 89.4 1.22 64 82 40 227.1 2.77 64 94 80 427.2 4.55 64 216 216 1121.2 5.19 128 137 20 149.9 1.09 128 146 40 220.0 1.51 128 255 250 1298.6 5.09 Read the M=128 rows: 137 -> 146 blocks is +6.6% of blocks and +46.7% of TIME, because distinct experts went 20 -> 40. Cost per DISTINCT EXPERT is flat at 5.2-5.7 us across the whole table (1.125 MiB each, ~205-225 GB/s, the same plateau as 6ad); cost per BLOCK varies 4.7x over the same rows. So time is distinct_experts x 1.125 MiB / ~215 GB/s, and block count is nearly irrelevant. Every in-situ comparison in 6x and 6y normalised by BLOCKS -- 38.9 ours against 40.6 upstream -- which is not the driver. Blocks are what num_tokens_past_padded reports; experts are what the kernel streams. With block_size_m = 8 and correlated spec-decode tokens one expert routinely spans several blocks, so the two come apart. Apply the model to the recorded in-situ launches: ours 164.0 us implies ~30 distinct experts, upstream 151.6 us implies ~28. About TWO distinct experts per launch reproduces the entire 8.2% with ZERO implementation difference. It also explains why both in-situ arms beat the standalone plateau per block (4.21 and 3.73 against ~5.3) -- in situ several blocks share an expert, so fewer bytes are streamed per block. That was never a sign of anything being wrong. Consequences. The 8.2% "kernel gap" is an artefact of comparing two different routing draws and should not be quoted again. Any future MoE comparison must control distinct experts per launch or force both arms onto an identical token stream, because block counts compare the wrong quantity. VT_MOE_PAD_STATS counts padded tokens and blocks; it should count distinct experts too, since that is the number that predicts the time. What this does NOT do: it does not move the end-to-end ratio, which is wall-clock on matched prompts and token counts and stands at ~0.966. It removes the ATTRIBUTION of that residual. The open question is whether our token path systematically touches more experts per step -- a near-tie divergence consequence rather than a defect -- or whether the remaining wall-clock sits outside the MoE entirely. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…was inverted (#442) The previous commit inferred from the distinct-experts model that our in-situ launches touch ~30 distinct experts against upstream's ~28, and that this reproduced the 8.2%. That is BACKWARDS, and the counts contradicting it were already in .agents/benchmark-record.md: the 2026-08-12 both-sides measurement recorded ours at 311.2 padded tokens / 38.9 blocks per call against upstream's 324.8 / 40.6. I inferred where the file already had a measurement. At M=9 the 72 (token, expert) pairs spread over ~39 experts at under 8 tokens each, so moe_align emits ONE block per expert and blocks and distinct experts COINCIDE at the decode shape. That measurement was therefore already counting experts, upstream touches MORE of them, and the routing explanation stays refuted exactly as 6y concluded. The distinction is still worth keeping, because blocks and experts come apart 4.7x at M=64/128 -- it simply does not bite at M=9. What survives is sharper than what it replaces: standalone, matched work ours == upstream to 0.27% in situ ours does LESS work (38.9 vs 40.6 expert-blocks) and takes MORE time (164.0 vs 151.6 us) A kernel identical in isolation cannot be slower in place because of its own code, so the deficit belongs to the CONTEXT, not the kernel and not the routing. Candidates in the order their evidence justifies: expert-weight residency in situ, where this repo has already measured 20-30% per GEMM for host/ATS-retagged decode weights and the standalone arm's fresh cudaMalloc cannot reproduce it; clock and power state across the two runs; and overlap with concurrent stream work. Measurement-base caveat now stated: in-situ per-launch times are summed profiler kernel durations, standalone are wall-clock over 80 iterations, and both in-situ arms beat the standalone plateau per unit work (4.21 and 3.73 against ~5.3), which may be partly that rather than physical. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
#442) Two arithmetic corrections close this thread. THE MIX. The 1520 in-situ launches are 760 gate_up plus 760 down, and down's per-expert bytes are exactly HALF gate_up's: gate_up is 2N x K/2 weights plus 2N x K/16 scales = 1.1250 MiB, down is K x N/2 plus K x N/16 = 0.5625 MiB, mixed average 0.8438 MiB per expert-block. 6af compared a MIXED in-situ average against a gate_up-ONLY standalone plateau, which is the whole reason both arms appeared to beat it at 4.21 and 3.73 us against ~5.3. They did not. Redone with the right bytes: blocks MiB/launch us implied GB/s ours 38.9 32.8 164.0 209.9 upstream 40.6 34.3 151.6 236.9 Measured standalone gate_up plateau: 203-226 GB/s. Ours sits INSIDE it, upstream ABOVE it. We run this kernel at the bandwidth it achieves in isolation, and upstream gets something in place that the isolated kernel does not -- cache reuse across the gate_up/down pair is the first candidate, down's weights being half size so more of the working set persists. The framing inverts: on this evidence we are not slow here, upstream is unusually fast in situ. THE MODE HOLE. 6y warns, correctly, that "a work COUNT may be taken under different execution modes; a TIME may not" -- and the per-unit-work normalisation then does exactly what that forbids. The 38.9/40.6 counts were taken EAGER (ours VT_SPEC_DECODE_GRAPH=0, upstream enforce_eager, because neither probe survived capture and compile) while the 249.2/230.4 ms times came from the GRAPHED profile. Nothing establishes the graphed runs had the same blocks per launch as the eager ones, so every ratio dividing those times by those counts -- 4.21 vs 3.73 us/block, 12.8% per unit of work, 186.6 vs 210.7 GB/s -- has a denominator from a different execution mode than its numerator. CONSEQUENCE: the only like-for-like Marlin comparison in evidence is the standalone one of 6ae, and it says parity at 0.9973, inside one standard deviation. Closing the in-situ question needs blocks AND time from the SAME graphed run, which needs a probe that survives capture: a device-side counter the launcher increments, read once at the end, never a per-launch D2H sync. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…ntly resolve 3% (#442) moe_wna16_marlin_gemm's last parameter is blocks_per_sm, and both engines pass -1 (auto), which yields 32768 B of shared memory, 3 blocks/SM and 25% occupancy. With the kernel running at ~78% of peak bandwidth, forcing a higher value was the one settable occupancy lever left. It does not work. A TRAP FIRST, because it nearly produced a false result. The unseeded sweep read bps -1/1/4 slow and 2/3/5/6/8 fast, an apparent 8.7% win. The routing was redrawn per run, so `distinct` moved between 35 and 41, and since time tracks distinct experts (spec 6af) the sweep was measuring the DRAW. Any MoE comparison that lets routing vary between arms measures the draw and not the change. The harness now seeds torch explicitly for that reason. SEEDED, identical 33 distinct / 33 blocks everywhere, 4 interleaved reps at 120 iterations: bps runs (us) mean excl. outliers -1 auto 191.3, 367.8, 177.2 184.2 3 181.1, 171.1, 177.6, 353.0 176.6 4 177.7, 178.6, 179.8, 180.3 179.1 5 181.7, 184.2, 179.4 181.7 Between-configuration spread is 4.3%; the WITHIN-configuration spread is comparable (bps=3 alone ranges 171-181). No reliable effect. The lever is dead. THE MORE USEFUL FINDING is the environment. Two of fourteen runs returned ~2x (367.8 and 353.0) and two produced no output at all. A box emitting 2x outliers cannot resolve a 3.4% question, and this one is currently contended: a concurrent session has been building and rebuilding a ~79G tree on the same hardware. No parity measurement taken in this window should be trusted in either direction. The paired numbers earlier in the record predate that contention and were interleaved, which is what makes them survivable. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
#550 landed squashed, so its content arrives on main as a new commit while this branch already carried it plus everything since. All three conflicts are that shape -- HEAD is the superset in every hunk -- so HEAD is taken wholesale, which is what the record-conflict rule asks for. Verified rather than assumed: all 317 `## ` entries on main's .agents/benchmark-record.md are present here, so the union lost nothing. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…lock file (#442) Every standalone run in this branch was wrapped in `flock /tmp/gpu.lock`. That is the wrong file. This box's GPU lock is `$HOME/gpu.lock` -- it is what final_pair.sh takes and what other sessions hold -- so /tmp/gpu.lock coordinated with nothing and those runs executed unserialised against whatever else was on the GPU. At least one concurrent test_qwen27_spec_decode_concurrent (15.9 GB RSS) overlapped them. That is the likeliest source of the ~2x outliers (367.8 and 353.0 us) in the blocks_per_sm sweep, and it was MINE. An earlier note in this session blamed a concurrent session for not locking; the reverse is true -- they took the correct lock and I did not. WHAT SURVIVES: the ours-vs-upstream comparisons were INTERLEAVED within a single run, so contention lands on both arms alike and the RATIO is exactly what interleaving protects. 0.9973 with the sign flipping between reps still reads as indistinguishable, with wider error bars than the quoted sd suggests. WHAT DOES NOT: absolute us/call and us/block, including the 203-226 GB/s plateau, are upper bounds rather than exact, since an unlocked box can only make them slower. That plateau needs re-taking under $HOME/gpu.lock before it is quoted as the kernel's achievable bandwidth. The two-regime SHAPE (L2-resident under ~18 blocks, streaming above ~27) and the distinct-experts scaling are robust to a uniform slowdown and do not need re-taking. Both harness arms now carry the rule in their header, including that nvidia-smi showing no compute apps does NOT mean the GPU is unreserved -- a holder may be between phases -- so `fuser -v $HOME/gpu.lock` is the check. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
localai-bot
force-pushed
the
row/SPEC-DSPARK-MARLIN-NCU
branch
from
August 13, 2026 02:43
5ebbe53 to
69e9e7a
Compare
…the bound direction was inverted (#442) A fresh scoped review returned FAIL on this branch with two blocking findings. Both are real, both are repaired here, and neither needed the GPU -- which is just as well, since dgx.casa is down. 1. "Cost per DISTINCT EXPERT is flat at 5.2-5.7 us" is FALSE. Recomputing us/call / distinct from the section's OWN table gives 4.470, 5.678, 5.340, 5.191, 7.495, 5.500, 5.194 -- a 4.47 to 7.50 range. The two distinct=20 rows are the tell: identical expert counts, 89.4 vs 149.9 us, blocks 73 vs 137. There, time tracks BLOCKS -- precisely what a flat-per-expert model calls irrelevant. The honest model has the same two regimes as the L2 finding one section earlier. Weights fitting L2 (low distinct) are not re-streamed, so cost is per-block work and time tracks BLOCKS. Weights not fitting (distinct >= ~40 here) make cost streaming-bound, so time tracks DISTINCT EXPERTS at 5.2-5.7 us each. time ~= distinct x 1.125 MiB / ~215 GB/s holds ONLY in the second regime: at distinct=20 it predicts 109.7 us against 89.4 and 149.9 measured, off by -19% and +37%. So the rule this spec promulgated -- "control distinct experts per launch" -- was INCOMPLETE. Control BOTH distinct experts and blocks, or state the regime. 2. The lock caveat reached the record and STATUS but NOT the spec, which is the file a fresh implementer is told to read, and it gave the bound the WRONG DIRECTION. Contention inflates TIME, so a bandwidth computed as bytes/time is a LOWER bound on what the kernel achieves, not an upper one. A re-take under $HOME/gpu.lock can only move the plateau UP, which would put our in-situ 209.9 GB/s BELOW it and REVERSE 6ag's inversion. "We are not slow, upstream is unusually fast" is the least favourable reading to us that the data admits and the one most likely to change on a clean re-take; it is now marked provisional in the spec itself. Also repaired: the plateau range mixed bands (over the rows actually flat it is 212.7-225.8 GB/s, and the 203 endpoint is the 27-block row the same paragraph excludes -- under the self-consistent range "ours sits INSIDE it" is FALSE); "at matched blocks" overstated the harness, whose two arms draw routing from independent RNG streams and therefore occupy different block counts at the same pool, so blocks are NORMALISED rather than matched and neither arm can yet take an external routing tensor; the blocks_per_sm entry credited the 8.7% to the seeded pass when it came from the unseeded one; and --iters defaults differed between the arms (python 200, C++ 80) while the prose asserted 80 for both. benchmarks/marlin_moe_standalone.cpp is not wired into any build target, so nothing compiles it and it will rot against vt::MoeGroupedGemmNvfp4Marlin's signature. Recorded as owed in the file header and the record rather than quietly left. WHAT SURVIVES: 6ae's interleaved 0.9973 -- the kernel is not the gap -- because interleaving is exactly what protects a ratio against contention AND routing draw. What does not survive as stated is 6ag's inversion. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
Brings in #583, which repairs the Invoke-Checked empty-argument defect that was red on every open PR including this one, plus the Nemotron and intake records. No overlap with this branch, which touches the DSpark spec, the benchmark record, STATUS, BENCHMARKS and two standalone harnesses. Worth noting for this row specifically: BENCH-ASSERT-CLOCK-STATE (51ec6be) records that the SM clock moves 12.8% between boots. That is the same class of effect this branch records as "absolute numbers move up to 5% between sessions for the same binary", and it independently reinforces why only WITHIN-session, interleaved ratios are quotable here. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…not 0.966 (#442) Every ratio this row has recorded was taken with a COLD leading arm. Correcting that moves the headline by two thirds of the deficit. Four controls, together for the first time: * $HOME/gpu.lock. The standalone runs of 6ad-6af took /tmp/gpu.lock, which coordinates with nothing. * A DISCARDED warm-up arm ahead of the first measured arm. The GB10 SM clock ramps over MINUTES -- 1449 to 2190 MHz observed across one run -- so dropping rep 1 is not enough and an entire first arm reads ~6% low. * Settle barriers between arms. vLLM asserts free GPU memory does not GROW during its startup profile, and GB10 returns our engine's pages lazily, so an oracle launched straight after our arm dies with "Initial free memory 68.53 GiB, current free memory 89.42 GiB". That killed every earlier paired attempt today. * A host-RAM headroom guard before the oracle. gpu_memory_utilization reserves HOST RAM here, so an oracle without headroom takes the MACHINE down: three reboots on 2026-08-13 (08:57, 09:29, 16:29), at least the last of them mine. arm n median tok/s range ours BEFORE 9 142.604 141.67-142.79 ours AFTER 9 142.140 135.23-142.87 ours combined 18 142.534 oracle 15 144.130 141.86-151.84 Drift before -> after is -0.33%, inside the 1% gate this harness sets for itself, so the run counts. Ratios: 0.9894 before, 0.9862 after, 0.9889 combined. THE GAP IS 1.1%, NOT 3.4%. The 0.9757 / 0.9646 / 0.9569 of 6ac were measuring an unwarmed first arm as much as the engine. This is NOT parity. 0.9889 is below 1.0 and the row stays open. What it changes is the size of what remains -- and it lands after 6ae had already cleared the kernel the deficit was attributed to, so the two findings agree: there was less to explain than the record claimed, and it was not in Marlin. Caveats kept rather than buried. n=1 paired run; a repeat is running and is owed before this is settled. The ours-AFTER arm carries four low outliers (135.2, 135.5, 139.0, 139.9) the BEFORE arm does not, so something touched the box during it -- medians unaffected, that arm's spread is not trustworthy. The oracle shows ONE fast draw (151.84) against fourteen near 144, the old bimodality appearing once, absorbed by the median. The oracle copy runs gpu_memory_utilization=0.35 rather than 0.55; at max_num_seqs=2 and max_model_len=2048 the KV cache needed is a fraction of either budget, so it cannot change decode speed, but it IS a config delta on the denominator and is recorded as one. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
One conflict, in .agents/benchmark-record.md, which is the append-only log -- union-merged, both sides kept in order. Verified rather than assumed: all 322 `## ` entries from main are present in the result, and this branch's new entry survives alongside them. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #550 (same row, same issue).
§6z recorded upstream's
ncucounters as the last open route and marked it BLOCKED — vLLM's EngineCore will not initialise underncuin either replay mode — so its DRAM efficiency stayed derived rather than counted, and the spec named a standalone harness as the only way in.scripts/marlin-moe-standalone.pyis that harness: it drives upstream's owntorch.ops._moe_C.moe_wna16_marlin_gemmon the 35B-A3B decode shapes (hidden 2048, moe_intermediate 512, E=256, top_k=8,moe_block_size8) with no EngineCore, no multiprocessing and no model load, mirroringprepare_nvfp4_moe_layer_for_marlin's scale pipeline exactly. It asserts the oracle identity and aborts on mismatch. It profiles.Static geometry (new, and trustworthy — these are static properties, not counters)
grid 144, block 128, 32768 B shared per block. GB10 has 48 SMs with 102400 B of shared memory each → 3 blocks/SM → 25% occupancy (achieved 25.98%), and 48 × 3 = 144 is the grid. The launch is a persistent single wave, so the "38.9 vs 40.6 blocks per launch" of §6x are work items each CTA loops over, not parallel blocks.
A measurement trap worth the commit on its own
dram__bytes.sumreadsn/aon GB10 — the chip exposes no DRAM counters. SpeedOfLight still prints "Memory Throughput 11.14%" beside "Compute (SM) 11.42%", and reading those as "neither saturated, so latency-bound" is wrong: that memory figure excludes DRAM traffic entirely. Ann/aon the byte counter is the tell that the percentage beside it comes from somewhere else. The second commit on this branch retracts exactly that misreading — the branch deliberately keeps both the wrong reading and its retraction rather than being force-pushed.What actually answers it: sweep the work
Each block streams 1179648 B (1 MiB of weights + 128 KiB of scales). Two regimes: under ~18 blocks the weights fit in L2; above ~27 they stream and µs/block is flat at 5.2–5.5 across a 2.4× range of work. Constant bytes-per-second across varying work is the bandwidth-limited signature. Plateau: 203–226 GB/s.
So §6z was right
Its derived in-situ numbers land on that plateau. Upstream's 210.7 GB/s is inside it — upstream runs at this kernel's achievable bandwidth — while ours at 186.6 GB/s is ~12% below. The harness built to test the attribution corroborates it instead of overturning it.
What does not survive is §6x's per-unit-work division: with a fixed 144-CTA grid, block count is loop iterations, and the sweep prices 38.9 → 40.6 blocks at about +4.4%.
Open, and decisive
Ours has not been through this harness. That is the next measurement: our kernel, same shapes, same routing, µs/block against the 5.2–5.5 plateau. If ours plateaus there, §6x's localisation to this kernel is wrong; if ~12% higher, the kernel owns the gap. Occupancy stays a real but secondary lever at ~75–80% of this device's ~273 GB/s.
Absolute µs/call here is not comparable in-situ (uniform synthetic routing occupies 61–65 blocks vs the model's 38.9–40.6); the geometry, regime shape and plateau bandwidth are.
Gate:
scripts/agent-preflight.sh --staged— All gates green.FOLLOWING_AGENTS_PROTOCOL