diff --git a/.agents/NOW.md b/.agents/NOW.md index 477157d4d..338964767 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -1,6 +1,6 @@ # NOW — the one-Read resume surface - + Read this FIRST, every session. A SNAPSHOT, rewritten in place: what is live, the gate being chased, what to do next. Never a log — evidence lives in the @@ -26,7 +26,7 @@ Work: exact-chunks on main `1ce0d662b`; sm_120 measured at `3d2581551`. | CPU levers (`QUANT-GGUF-CIQ-GEMM`) | Profile DONE: decode **47% threadpool sync**, prefill **~39% paged attn**. **G5 not next** | Parakeet encoder; attn dtype hoist | | Supported-models list | **LANDED**: FEATURES arch table CI-bound (33 archs) | — | | `/v1/videos` OpenAI shape | **MERGED** (#71): Sora `model`/`size`/`seconds` + `GET /{id}/content` | `row/SERVE-VIDEOS-REFS` PR open: reference conditioning | -| Vulkan 27B lm_head | **column-blocked -1.07 ms/tok, 6/6**; roof is 230 not 273 | 20x bimodality REAL on an IDLE box; reclaim REFUTED | +| Vulkan 27B decode | **4.285 vs llama.cpp 4.35 (98.5%)** BINDING, 8/8 clean legs @`81ea01f0`; GPU 240.3->227.7 ms/tok; GEMV lever CLOSED | 3.1 ms short. Open: 20x lm_head bimodality REAL on an IDLE box (reclaim REFUTED, mechanism unknown) | | `BACKEND-ROCM` | **(b) fix in; #140 gfx1201 hipBLAS + Gemma-4 MoE landed (contributor, authorship-preserved); W0 green 4 archs** | compile + M2 ([spec](specs/rocm-unified-memory-b.md)) | | TP spike #287 (PR #143) | **TP-W1 LANDED**: rank-group table + TP handle (6/6); DSR leak FIXED (unblocks #127/#154/#155) | TP-W2 (linears + loader) | | Release | SPIKE; 30/30 | #129 | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 8ea0e8083..e4e89f090 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -16542,3 +16542,65 @@ column blocking suppresses it. It is NOT "a lever bigger than every other Vulkan optimization" on the evidence available -- at 1 leg in 16 it costs roughly 6% of mean decode time, against the 8.5% that column blocking takes off this kernel deterministically. + +## BACKEND-VULKAN combined: the three levers measured TOGETHER on merged main (2026-08-09, GB10, `81ea01f0`) + +Each of the three Vulkan levers was measured against `93852c28` in isolation, so +none of their numbers described a tree carrying all three. `docs/STATUS.md` was +deliberately left at the CONSERVATIVE 4.24 (the largest single lever) rather than +a sum. This entry is the binding combined measurement that replaces it. + +**METHOD.** `git archive` of the merged tree to dgx, `vt_matmul.comp` md5 verified +identical on both sides. Fresh Release configure. 8 wall-clock legs with the page +cache dropped before each and `flock ~/gpu.lock` held; then a two-length +GPU-timestamp diff (output-len 36 minus 4, over 32 decode tokens) so prefill and +one-time costs cancel. Qwen3.6-27B bf16, 1 prompt, 32-in, c1. + +**A FIRST ATTEMPT WAS INVALID AND IS RECORDED BECAUSE THE FAILURE IS REUSABLE.** +It read 0.75-1.27 tok/s. `VLLM_CPP_VULKAN` defaults to `AUTO`, which resolves to +`OFF` (Vulkan is opt-in so it cannot register into gate builds), and the fresh +configure omitted `-DVLLM_CPP_VULKAN=ON`. The options had been copied from the +reference build's `CMakeCache.txt` through `grep | head -15`, and the +alphabetically-sorted list ended at `TRITON_TARGET`, exactly one line before +`VULKAN`. Three independent tells were present in the output and are the cheap +check: NO `[vt vulkan]` lines despite `VT_VULKAN_DISPATCH_STATS=1`, no +`[vt reference-tier]` lines, and "Asynchronous scheduling is ENABLED" where every +valid Vulkan run reports it disabled. Same family as this campaign's stale-binary +false greens, in the opposite direction: a false catastrophe rather than a false +pass. Re-run after confirming `VLLM_CPP_VULKAN:STRING=ON` in the cache AND that +the run emits Vulkan lines BEFORE trusting it. + +**MEASURED, 8 legs, ALL CLEAN.** TPOT 232.18, 232.42, 232.50, 233.34, 233.42, +233.47, 233.80, 234.07 ms; decode 4.27-4.31 tok/s. **Median 4.285 tok/s**, spread +0.8%. Zero bimodal collapses in 8 legs (the ~1.8x pathology hit 1 leg in 16 in the +lm_head row's block, so its absence here is consistent, not evidence against it). + +**MEASURED, per decode token, two-length diff:** + +| shader | before (`93852c28`) | merged | delta | +|---|---:|---:|---:| +| `vt_matmul_vec` | 214.1 | **210.1** | -4.0 | +| `vt_matmul` (lm_head) | 12.43 | **11.57** | -0.86 | +| `vt_rms_norm` -> `vt_rms_norm_wide` | 7.97 | **1.57** | -6.40 | +| `vt_paged_attn` | 2.02 | 2.00 | - | +| `vt_gdn_decode` | 1.40 | 1.40 | - | +| **total GPU** | **240.3** | **227.7** | **-12.6** | + +Wall 233.0 ms, so host is **5.3 ms/token**. Each lever reproduced its own claim on +the merged tree: `vt_rms_norm_wide` at 0.0123 ms/call is exactly the 0.0123 its row +reported, and the GEMV and lm_head deltas match theirs to within the leg spread. + +**WHERE THE REMAINING 3.1 ms IS.** llama.cpp Vulkan is 4.35 tok/s = 229.9 ms/token +on the same 50.89 GiB weights on this box. `vt_matmul_vec` is now 92% of our GPU +time at 248.0 GB/s over 52.1 GB/token. The two named residuals are both small: +lm_head at 219.8 GB/s is 95.4% of the 230.3 GB/s ceiling MEASURED for its own shape +(worth ~1.3 ms if it could reach the layer GEMVs' 248), and host at 5.3 ms. + +**A CORRECTION TO THIS CAMPAIGN'S ROOF ARITHMETIC.** The lm_head row established +that GB10 does not reach its theoretical 273 GB/s, by running a known-good +streaming kernel on the identical byte count: 230.3 GB/s. That correctly retires +the 9.3 ms lm_head floor. It does NOT invalidate the layer-GEMV percentages, which +MEASURE 243-248 GB/s -- above 230.3 -- so 230.3 is a ceiling for THAT SHAPE +(k=5120, n=248320, one 2.54 GB buffer), not a device ceiling. Why one shape's +ceiling sits ~7% below the same kernel's on layer weights is UNEXPLAINED, and is +the same lone buffer the 20x bimodal collapse attaches to. diff --git a/.agents/state.md b/.agents/state.md index b8c38281f..48588f1f9 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -43483,3 +43483,61 @@ Settling it needs ~60 more legs. It is the named next experiment, not a claim. The coordinator's "bigger than every other Vulkan lever" framing is NOT supported: at 1 leg in 16 the collapse costs roughly 6% of mean decode time, against the 8.5% column blocking takes off this kernel deterministically. + + + +## Vulkan 27B decode: three levers landed and MEASURED TOGETHER; 4.285 vs llama.cpp 4.35 + +IMMUTABLE HEAD: `81ea01f0` on `main` (PRs #183, #184, #185, #186 all merged). + +WHAT LANDED, each verified by the operator re-running the gate rather than taking +the row's report: +- #183 `kAttnQkNormRopeGate` native: reference-tier drains during decode 16 -> 0. +- #184 GEMV wide loads: 90.0 -> 91.1% of roof. llama.cpp's own rows-per-workgroup + trick MEASURED 0.966x, a LOSS here, and ships off. +- #185 `vt_rms_norm` was dispatching ONE WORKGROUP PER ROW, so batch-1 decode ran + 128 lanes on a single SM: 7.85 -> 1.59 ms/token. +- #186 lm_head column blocking: -1.07 ms/token, and the 9.3 ms floor the + coordinator briefed does NOT exist. + +BINDING COMBINED MEASUREMENT (the combination no row tested), GB10, 8 legs, page +cache dropped before each, `flock $HOME/gpu.lock`: median decode **4.285 tok/s**, +TPOT 232.18-234.07 ms, all 8 legs clean. GPU 240.3 -> 227.7 ms/token; host 5.3 ms. +Full detail and the per-shader table are in the benchmark record. + +GATES RUN: on the merged tree, `test_vulkan_backend` 32/32 (1905 assertions), +`test_backend_cross_device` 11/11, `test_opt_paged_engine` on Vulkan 6/6 +token-exact (96/96) with 0 declines, `gen-vulkan-spirv.py --check` byte-clean under +pinned glslang 16.5.0. Per-branch GB10 gates are in each row's own entry. + +BLOCKER / WHY NOT DONE: the goal is llama.cpp Vulkan's 4.35 tok/s = 229.9 +ms/token. We are 3.1 ms short. `vt_matmul_vec` is now 92% of GPU time at 248.0 +GB/s and its lever is CLOSED with the binding constraint identified as DRAM +bandwidth on a ~50 GB working set streamed once per token. The named residuals are +small and both are open questions rather than known work: lm_head at 219.8 GB/s is +95.4% of the 230.3 GB/s ceiling measured for ITS OWN SHAPE, and why that shape's +ceiling sits ~7% below the same kernel's on layer weights is UNEXPLAINED. + +PROHIBITIONS: do not run two 27B processes on dgx at once; that box has +OOM-rebooted. Take `flock $HOME/gpu.lock`. `local-ai-worker` must be stopped +before GPU work and restored to `--restart=always` after. + +OPEN, NOT CLAIMS: +- The ~20x `vt_matmul` bimodal collapse is REAL on an IDLE box with 119 GB + MemAvailable before every leg; co-residency, page reclaim and load effects are + all REFUTED. Mechanism UNKNOWN. The one surviving correlate is near-ceiling + OCCUPANCY (106 GB RSS of 119); the ballast test that would settle it was not run + because that has OOM-rebooted this box. Zero collapses in the 8 combined legs. +- Suggestive only (p about 0.07, needs ~60 more legs): collapses landed only on + the unblocked lm_head arm. + +THREE DOC GATES BLOCKED FACTUALLY-CORRECT CHANGES and were NOT gamed: `check-pr-size` +counts generated `vulkan_spirv.cpp` as reviewable product code (every Vulkan shader +PR trips it); `docs/STATUS.md` sits over a shrink-only ratchet that only a +33,211-char foreign table cell can clear; and `README.md:310` still says "24 native +ops" (now 25) because the README gate demands an unrelated landing-page edit. These +are checker gaps, and they are for the developer to rule on. + +FIRST RESUME COMMAND: + git log --oneline -1 origin/main # expect 81ea01f0 or later + ssh dgx.casa 'grep -E "^VLLM_CPP_VULKAN" ~/vkcomb/build-vk/CMakeCache.txt' # must be ON diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 1e44c4d2d..33cf598a7 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -361,7 +361,7 @@ built on it rather than keeping the flattering one. | Memory footprint vs declared workload (`ROAD-V1-MEM`, #83) | **Never measured, and not measurable today**: there is no auto-sizing to compare against, because the KV pool is a hand-typed `--num-blocks`, so "what the run actually needed" has no number | Once M1's `MemoryBudget` lands: predicted-vs-actual bytes per allocation class, then peak footprint ours-auto vs vLLM at its 0.9 default on the same model and config | | Startup latency (cold to first `/health`) | **36.51 s vs vLLM 0.25.0's 221.51 s = 6.07x** (medians of 3, 27B-NVFP4, GB10). PROVISIONAL: 3 of 6 legs contended, repeat killed by a host reboot. [Detail](../.agents/benchmark-record.md) | Uncontended 3-rep re-run on a quiet box | | Speculation depth (`ROAD-V1-D3-SPEC-K`, #81) | **Never measured, MTP is k=1** (our port covers vLLM's k=1 branch only), so no acceptance-vs-depth curve exists | k=2..4 three-way greedy gate, then the c1/c>1 A/B + the per-workload (prose vs code) acceptance-vs-depth curve any dynamic or adaptive depth policy needs | -| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN); 62 host-tier. **27B prefill 21.5x**, decode 4.13->**4.24** vs 4.35. **GEMV CLOSED at 91.1% of roof**; RMSNorm 1 wg/row; lm_head **-1.07ms/tok**. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | +| Vulkan vs llama.cpp Vulkan (`BENCH-VK-LLAMA`) | 25 NATIVE (+8 GDN); 62 host-tier. **27B prefill 21.5x**, decode **4.28**/4.35 BINDING (8/8 clean legs, merged tree). GPU 240.3->227.7 ms/tok. [Detail](../.agents/specs/vulkan-full-support.md) | `VK-C` coopmat A/B on Thor (`VT_VULKAN_COOPMAT=0` A/Bs it): **11.1x-32.9x** vs our UNTILED scalar kernel, not vs a competent GEMM. `VK-E`: llama.cpp `-DGGML_VULKAN=ON` at `237ad9b96` on dgx, same GGUF, three columns | | ROCm (`BACKEND-GATE-ROCM-VLLM` / `-SGLANG`) | **NOT APPLICABLE: no number measured, claimed or owed.** W0 ctest-green on 4 gfx archs (#41); gfx1201 hipBLAS + Gemma-4 MoE (#140, contributor) ran M0/M1 on 2× R9700, our side CPU-link-verified only. No AMD HW here | The approach-(b) fix (PENDING community) unblocks the first APU model run (M2); the gate becomes a same-box vLLM-ROCm oracle once a model runs ([#41](https://github.com/mudler/vllm.cpp/issues/41)); floor: vLLM | | SGLang floor arms | Never ran | Both arms of the SGLang comparison | | Embeddings on the ONE surface (ROW 6, `LlamaModel` + `vllm_embed` + `/v1/embeddings`) | **NO number measured, claimed or owed.** Correctness-gated only, CPU: the 2026-08-08 fold (engine path == direct registry path, f64 LAST+normalize reference on the committed fixture) is plumbing, no speed claim | A REAL embedding checkpoint (e5-mistral class) + a same-box `vllm.LLM(task="embed")` oracle; only then does an embed-throughput bar exist | diff --git a/docs/STATUS.md b/docs/STATUS.md index c633b1efb..b68dfcfe6 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -422,7 +422,7 @@ Parakeet ASR (2026-08-07): *CPU-correct, ON THE ONE SURFACE (ROW 1)*. Ids exact LoRA (W1 CPU runtime brick landed; not yet usable end-to-end), multi-GPU, Vulkan (opt-125m exact; 25 native +8 GDN, both recurrences + fused attn preamble; 27B prefill 21.5x, decode -4.24/4.35 llama.cpp, RMSNorm+GEMV+lm_head; #125 VERIFIED +4.28/4.35 llama.cpp, 3 levers combined; #125 VERIFIED [campaign](../.agents/specs/vulkan-full-support.md)), ROCm (W0 community-green on 4 gfx archs (#41); the ratified (b) APU unified-memory fix is in (**blind-written, unverified**); M2 needs verification; gfx1201 hipBLAS +