Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions .agents/benchmark-record.md
Original file line number Diff line number Diff line change
Expand Up @@ -20353,3 +20353,34 @@ per stream), NOT as a perf fix; C_tmp size is ELIMINATED as an explanation of th
Method: third time drift has fooled a before/after here. Pairing caught the
first, pinned clocks the second, and only an in-process toggle catches this one.
Future perf claims on this row need the toggle, not two runs.

## SPEC-DSPARK: storage ruled out; ratio stable at ~0.966 across three sessions (2026-08-12)

Question raised: are the weights on NAS, or not fully resident, distorting the
measurements?

Weights are on LOCAL NVMe (/dev/nvme0n1p2 ext4); no NAS mount exists on the box.
A run reads 22.06 GB total = one full model read at load. Process RSS during
decode is 4.8 GB, so weights are uploaded and the mapping released, not held.
Decode is stable to 0.5% across 8 warm reps (146.0-147.6), which file-backed
weights could not be. Storage is NOT a factor.

Operational: that NVMe is 98% full (76 GB free), and this repo has already lost a
gate run to ENOSPC reporting green over work that never ran.

Within-session ratios, three independent measurements:

| session | ours | oracle (modal) | ratio |
|---|---|---|---|
| pinned clocks, pre-C_tmp | 135.98 | 139.36 | 0.9757 |
| pinned clocks, post-C_tmp | 139.20 | 144.32 | 0.9646 |
| free clocks, ours->oracle->ours | 140.98 | 147.32 | 0.9569 |

~0.966 +/- 0.01, consistently below 1.0. Absolute numbers move up to 5% between
sessions for the SAME binary because GB10's memory clock cannot be pinned, so
only the within-session ratio is quotable -- and all three agree.

Oracle draws remain bimodal (~147.3 and ~155.6), the same one-extra-accepted-token
effect as the fibacc run, so its MODAL draws are the honest denominator.

Evidence: `dgx:~/work/dspark-w6/iocheck.log`, `final_pair.log`.
41 changes: 41 additions & 0 deletions .agents/specs/dspark-spec-decode.md
Original file line number Diff line number Diff line change
Expand Up @@ -1623,6 +1623,47 @@ in-process A/B switch, because on a machine whose memory clock cannot be pinned,
even a pinned-clock before/after across two runs is not a controlled experiment.
Any future perf claim on this row needs the toggle, not two runs.

## 6ac. STORAGE RULED OUT, and the ratio is stable at ~0.966 (2026-08-12)

Developer question: could the weights being on NAS, or not fully resident in
device memory, be distorting these measurements?

**Tested, and no.**

| check | result |
|---|---|
| weight location | local NVMe (`/dev/nvme0n1p2`, ext4) -- **no NAS mount exists on this box** |
| total disk read for a run | 22.06 GB, i.e. ONE full model read at load |
| process RSS during decode | 4.8 GB -- the weights are NOT held in host RSS; they are uploaded and the mapping released |
| decode stability | 8 warm reps at 146.0 / 147.6 / 147.2 / 146.9 / 147.2 / 147.3 / 147.4 -- **0.5% spread** |

If weights were still file-backed, decode would fault pages from NVMe and the
per-rep numbers would be erratic; a 0.5% spread says they are resident and decode
touches no storage. Worth keeping in mind that on GB10 the failure mode would be
severe if it ever regressed -- host pages reach the GPU through ATS, which this
repo measured at a 20-30% per-GEMM penalty.

**Operational finding worth acting on separately:** that NVMe is **98% full**
(3.4T of 3.6T, 76 GB free). This repo has already lost a gate run to ENOSPC
producing a green-looking report over work that never executed.

**And the ratio is now measured three times, each WITHIN one session:**

| session | ours | oracle (modal) | ratio |
|---|---|---|---|
| pinned clocks, pre-C_tmp | 135.98 | 139.36 | 0.9757 |
| pinned clocks, post-C_tmp | 139.20 | 144.32 | 0.9646 |
| free clocks, ours->oracle->ours | 140.98 | 147.32 | **0.9569** |

**~0.966 +/- 0.01, consistently below 1.0.** The absolute numbers move a lot
between sessions (135.98 to 142.09 for the same binary) because GB10's memory
clock cannot be pinned, which is exactly why only the within-session ratio is
quotable -- and all three agree.

Also visible in the last run: the oracle's draws are bimodal at ~147.3 and
~155.6, the same one-extra-accepted-token effect as §6q, which is why its MODAL
draws are the honest denominator.

## 7. Evidence, authority, stop conditions

- Evidence root: `dgx:~/work/vllm.cpp-dspark-<slice>/`, one `flock`, named tmux.
Expand Down
2 changes: 1 addition & 1 deletion docs/BENCHMARKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -344,7 +344,7 @@ in the tree, default-OFF, for reproducibility; detail in the benchmark record.
| MTP | Qwen3.6-27B NVFP4 | token-identical to vLLM MTP, **~4% faster at c1**; on-par at c2-c8 | `DONE` |
| DFlash | Qwen3.6-27B NVFP4 | **2.9x over spec-off** (10.16 → 29.32 tok/s), at/above vLLM DFlash-on (**1.003x**, non-overlapping bands) | `DONE` |
| n-gram | Qwen3.6-27B NVFP4 | draft-free (`SPEC-NGRAM`); 27B 5/5 STRICT our-ngram-ON == vLLM-ngram-ON, 180/180 drafts accepted (correctness only, no speed row yet) | `DONE` |
| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **0.965x-0.976x** within-session; C_tmp cap landed but is perf-NEUTRAL (+0.03%) (#442) | `ACTIVE` |
| DSpark | 27B NVFP4 dense k=15; 35B-A3B MoE k=8 | MoE **0.975x** code / **1.012x** prose vs the pinned graphed oracle (PINNED CLOCKS, non-overlapping). NOT parity: **~0.966x +/- 0.01** over three within-session pairs; C_tmp cap perf-NEUTRAL; storage refuted (#442) | `ACTIVE` |
| Breadth (EAGLE1/3, suffix, ngram-gpu, dynamic-k, ...) | n/a | enumerated from vLLM source + `INVENTORIED` 2026-08-06 (`.agents/specs/spec-decode-inventory.md`), unmeasured | `INVENTORIED` |

## How we measure
Expand Down
11 changes: 10 additions & 1 deletion docs/STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -529,7 +529,16 @@ memory-system behaviour that no allocation change we can name would alter; upstr
ncu in either replay mode. A C_tmp over-allocation (15-30 MB vs upstream's
3.15 MB) was found and fixed, but an in-session A/B shows it is perf-NEUTRAL
(+0.03%) -- an apparent +2.9% was machine drift, since GB10 cannot lock memory
clocks. Editing
clocks. The ratio has now been measured WITHIN a single session three times --
0.9757, 0.9646 and (ours->oracle->ours at free clocks, drift -0.89%) 0.9569 --
so it is **~0.966 +/- 0.01, consistently below 1.0**, while the absolute numbers
move up to 5% BETWEEN sessions for the same binary. Storage was raised as a
possible distortion and is refuted: the weights are on local NVMe (no NAS mount
exists on the box), a run reads 22.06 GB once at load, decode-time RSS is 4.8 GB
because the mapping is released after upload, and 8 warm reps hold a 0.5%
spread -- decode touches no storage. (That NVMe is 98% full, 76 GB free, which
is its own operational risk given ENOSPC has previously produced a green report
over a gate that never ran.) Editing
the kernel, its launch config, layout or flags is NOT indicated: all are proven
identical. (The repack kernels that appear to take 40% of a long run are
LOAD-TIME.) NOT parity. The Gemma4 `1 + N` layout is coded and unit-tested but has
Expand Down
Loading