From 298e78f6f0e5401b8705af8bd5fa677a06c93227 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 08:25:42 +0000 Subject: [PATCH] =?UTF-8?q?perf(kimi=5Flinear):=20W7-speed=20STRICT=20leve?= =?UTF-8?q?r=20=E2=80=94=20bf16=20regime=20MEASURED=20106=E2=86=92120/128,?= =?UTF-8?q?=20plateaus;=20device=20islands=20remain=20the=20residual?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The recorded STRICT path ("device islands + bf16 residual stream") is implemented as three env-gated numeric knobs in kimi_linear_device.cpp (default OFF -> the f32 correctness vehicle is byte-identical; CPU gate test_kimi_linear_forward 13/13·656 in the CUDA binary) and MEASURED on GB10 (full 48.9B model, the §12 128-token gate vs the STRICT deterministic golden; both flock locks + reclaim-waits, min-avail >=115 GiB, no reboot): VT_KIMI_BF16_RESIDUAL bf16 residual stream (vLLM fused_add_rms_norm order: residual stored bf16, block outputs bf16, RMSNorm variance over the f32 pre-store sum) via in-place f32<->bf16 rounds, keeping f32 storage so the islands still read f32. VT_KIMI_BF16_ISLANDS round the host-fallback island INPUTS (KDA q/k/v/g1/beta, NoPE-MLA q/kv/kpe) to bf16 (RNE) before the recurrence/softmax. VT_KIMI_ISLAND_F32ACC f32 (not f64) island accumulation -- MEASURED NEGATIVE, kept as a documented-negative A/B knob. RESULT (token match /128): control 106 (reproduces §13); residual-alone 106 (net-zero, shuffles flips, BREAKS p3 into a repeat loop); islands-alone 106 (fixes p2, destabilizes p3); residual+islands = 120/128 BEST (p0-p6 all 16/16; only p7 pos-8 flips: golden 18705 vs ours 58084, a single near-tie that cascades). The two levers INTERACT -- island input-rounding fixes p2 but repeats p3; the bf16 residual re-stabilizes p3. Further precision-matching regresses (island-output bf16 -> 90, reverted; f32 accum -> 91-106): host-precision-matching PLATEAUS at 120/128 because it is not vLLM's ACTUAL GDN-Triton / FA2 kernel arithmetic. VERDICT: NO arm STRICT (K=3-deterministic golden -> STRICT required, not distributional); default STAYS OFF (parity-enablers). Named residual (= also the speed lever): the device islands -- a NEW per-channel-decay GDN kernel (g[T,H,D]; vt::GdnDecode/GdnPrefill carry only per-head g[T,Hv], ops.h:1797/1846 -- NOT a drop-in) + the paged FA2 MLA. Speed: 1.30 tok/s (O(n^2) full-recompute + host islands, invariant to the knobs); vLLM can't serve Kimi-Linear-48B at bf16 on ONE GB10 (oracle capture needed util 0.82 for a single-seq eager run) -> HW-forced-indirect. Records: spec §14, STATUS/BENCHMARKS/ FEATURES/ENVIRONMENT keyed rows, NOW, benchmark-record, anchored state entry. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/benchmark-record.md | 19 +++ .agents/specs/kimi-linear.md | 78 ++++++++++ .agents/state.md | 23 +++ docs/BENCHMARKS.md | 2 +- docs/ENVIRONMENT.md | 3 + docs/FEATURES.md | 4 +- docs/STATUS.md | 2 +- .../models/kimi_linear_device.cpp | 134 ++++++++++++++++-- 9 files changed, 249 insertions(+), 18 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index b4bc37a83..342823425 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -18,7 +18,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool | | Invocation-parity prevention | CI guard + checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx | | MiniMax-H3 lane | **fl2va COHERENT; ref2va NVFP4 grid DIAGNOSED (#95): NO loader bug** | weights/islands/RoPE all quant-noise-close to coherent GGUF; residual = community-NVFP4 quant fidelity §8.12 | -| Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | **e2e RUNS** (bf16-resident §13): 13/13·656. Token gate **NEAR-TIE 106/128** | device GDN/MLA islands; 1.59 tok/s; default OFF | +| Kimi-Linear-48B (KDA+NoPE-MLA+MoE) | bf16 knobs **106→120/128**, NOT STRICT (§14, `row/KIMI-LINEAR-STRICT-SPEED`); default OFF | residual = device islands; 1.30 tok/s | | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 8a3ddc91a..5fb4be229 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -14535,3 +14535,22 @@ Ran #94's prescribed next diagnostic — a layer-by-layer activation diff of the **Render A/B re-confirmed in the same byte-inert build (256×256/22f/12steps, `pe.f32`, `VT_H3_ACT_DUMP` unset).** NVFP4 t2va = pale multicolour PATCH GRID every frame (0/10/21, per-patch uniform = degenerate latent); FL2VA-GGUF t2va = a COHERENT photorealistic orange cat on a windowsill. The diagnostic hooks are byte-inert (reproduce #94's outputs exactly). **CONCLUSION.** The mission's "2nd load-path defect" does not exist — our loader materializes the community NVFP4 file's weights faithfully (byte-verified). The residual is the **community NVFP4 checkpoint's quantization fidelity** (metadata `converted_by: "Star Ultimate Model Converter Pro"` — same dubious-converter lineage as the #94 nibble bug; corr to the coherent Q3_K only 0.85-0.94, well below the >0.99 a clean 4-bit quant gives) interacting with the DiT's massive-activation sensitivity. Definitively separating "poor community quant" from "inherent t2va-OOD sensitivity of these fl2va/ref2va finetunes" needs a clean reference — a bf16 ground truth (132 GiB host-f32 = OOM on one GB10) or a same-finetune REF2VA-GGUF control (disk-blocked, 23 GiB free) — both currently blocked; the path forward is an official modelopt-NVFP4 checkpoint, not a loader change. The #94 nibble fix stands as the objectively-correct dequant. The fp4-resident Marlin arm's separate grid stays a distinct, wiring-gated-only residual (untouched, per the mission). Artifacts `~/h3fp4/diff/{nvfp4_t2va,gguf_t2va}.mp4` + `{gguf,nvfp4}{2,3,4}.txt` fingerprints. + +--- + +## 2026-08-07 — Kimi-Linear-48B W7-speed STRICT lever: bf16 regime 106→120/128, plateaus (device islands the residual) + +Full 48.9B model on GB10 (sm_121a clean CUDA build, cutlass-4.5.0, 14 GDN AOT symbols; CPU gate `test_kimi_linear_forward` 13/13·656 in the CUDA binary), §12 8-prompt×16-token gate vs the STRICT deterministic golden. Three env-gated numeric knobs added to `kimi_linear_device.cpp` (default OFF, byte-identical). Both flock locks, reclaim-wait ≥90 GiB between every reload, min-avail ≥115 GiB, no reboot. + +| Config | env | TOKEN MATCH /128 | tok/s | note | +|---|---|---|---|---| +| control | (none) | 106 | 1.31 | reproduces §13 baseline exactly (build integrity) | +| residual only | `VT_KIMI_BF16_RESIDUAL` | 106 | 1.61 | net-zero: shuffles flips, BREAKS p3 into `163586×` repeat | +| islands only | `VT_KIMI_BF16_ISLANDS` | 106 | 1.30 | fixes p2, destabilizes p3 — net-zero | +| **residual + islands** | both | **120** | 1.30 | **BEST**: p0–p6 all 16/16; only p7 pos-8 flips (`18705`→`58084`) | +| + island-output bf16 | (islands, out-round) | 90 | 1.32 | REGRESSION — reverted | +| + f32 accumulation | `…VT_KIMI_ISLAND_F32ACC` | 91–106 | 0.84 | NEGATIVE — kept as documented A/B | + +FLIP LEDGER (deterministic golden arbitrates): the two levers interact — island bf16-input rounding fixes p2 but repeats p3; the bf16 residual stream (vLLM `fused_add_rms_norm` order) re-stabilizes p3. At 120/128 the SOLE divergence is p7 position 8 (golden deterministically `18705`, ours `58084`), a single near-tie that cascades to 8/16 on p7. Further precision-"matching" (output bf16, f32 accum) is a coin-flip that regresses — it is not vLLM's actual GDN-Triton/FA2 kernel arithmetic. Host-precision-matching PLATEAUS at 120/128. + +VERDICT: NO arm STRICT (K=3-deterministic golden → STRICT required, not distributional); default STAYS OFF (parity-enablers). NAMED residual (= also the speed lever): the device islands — a NEW per-channel-decay GDN kernel (`g[T,H,D]`; `vt::GdnDecode`/`GdnPrefill` carry only per-head `g[T,Hv]`, ops.h:1797/1846 — NOT a drop-in) + paged `mla::ForwardMlaAttentionBlock` (FA2). Speed HW-forced-indirect: 1.30 tok/s (O(n²) recompute + host islands, invariant to the numeric knobs); vLLM cannot serve Kimi-Linear-48B at bf16 on one GB10 (oracle capture needed util 0.82 for a single-seq eager run) so a direct `vllm bench throughput` arm is infeasible. Row STAYS ACTIVE. diff --git a/.agents/specs/kimi-linear.md b/.agents/specs/kimi-linear.md index 51ae0d2ba..32e0d82b5 100644 --- a/.agents/specs/kimi-linear.md +++ b/.agents/specs/kimi-linear.md @@ -752,6 +752,84 @@ is not token-exact). Row STAYS `ACTIVE`. --- +## 14. W7-speed STRICT lever MEASURED — bf16 regime recovers 106→120/128, plateaus; device islands remain the residual (2026-08-07, `row/KIMI-LINEAR-STRICT-SPEED`) + +The recorded path to STRICT ("device islands + bf16 residual stream = the same work as +speed") was implemented as three env-gated numeric knobs in `kimi_linear_device.cpp` +(default OFF → the f32 vehicle is byte-identical, CPU gate `test_kimi_linear_forward` +**13/13·656** in the CUDA binary) and MEASURED on GB10 (the full 48.9B model, the §12 +128-token gate vs the STRICT deterministic golden). Clean-from-`origin/main` CUDA build +(`-DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=ON -DVLLM_CPP_CUDA_ARCHITECTURES=121a -DVLLM_CPP_ +CUTLASS_DIR=…cutlass-4.5.0`, nvcc 13.0.88, Release, 14 GDN AOT symbols linked). Memory +safe throughout every reload (min-avail ≥ 115 GiB, both flock locks, reclaim-wait, no +reboot). + +### Knobs (`kimi_linear_device.cpp`) +- `VT_KIMI_BF16_RESIDUAL` — carry the residual stream in bf16 like vLLM's + `fused_add_rms_norm` (residual stored bf16, block outputs bf16, RMSNorm variance over + the f32 pre-store sum). Implemented as in-place `CastBf16`→`CastF32` rounds at exactly + vLLM's rounding points (embed out, each block out, residual after each add) keeping f32 + STORAGE so the islands still read f32 — byte-matching vLLM's fused-add order. +- `VT_KIMI_BF16_ISLANDS` — round the host-fallback island INPUTS (KDA q/k/v/g1/beta, + NoPE-MLA q/kv/kpe) to bf16 (RNE) before the recurrence/softmax. +- `VT_KIMI_ISLAND_F32ACC` — f32 (not f64) accumulation in the islands. **MEASURED NEGATIVE**, + kept as a documented-negative A/B knob. + +### Measurement (token match /128 vs the deterministic golden; STRICT required) +| Config | env | /128 | verdict | +|---|---|---|---| +| control | (none) | **106** | reproduces §13 baseline exactly | +| residual only | `BF16_RESIDUAL` | 106 | net-zero — SHUFFLES flips (fixes p2, BREAKS p3 into a `163586×` repeat loop) | +| islands only | `BF16_ISLANDS` | 106 | fixes p2 but destabilizes p3 (repeat loop) — net-zero | +| **residual + islands** | `BF16_RESIDUAL BF16_ISLANDS` | **120** | **BEST** — p0–p6 all 16/16 exact; only p7 flips | +| + island-output bf16 | `…BF16_ISLANDS(out)` | 90 | **REGRESSION** (reverted) | +| + f32 accumulation | `…ISLAND_F32ACC` | 91–106 | **NEGATIVE** (reverted from the ISLANDS path) | + +### Flip ledger / razor verdict (the deterministic golden arbitrates every flip) +- The two levers INTERACT: island bf16-input rounding fixes p2 but destabilizes p3 into a + degenerate repeat (`163586×`); the bf16 residual stream then RE-stabilizes p3 (kills the + repeat). Together they make **p0–p6 all 16/16 token-exact** (was 6/8 → 7/8 fully exact). +- The SOLE remaining divergence at 120/128 is **p7 position 8**: the golden deterministically + emits `18705`, our island emits `58084` (a genuine near-tie), then the greedy path cascades + (8/16 on p7). A single near-tie flip across the whole 8-prompt battery. +- Further precision-"matching" (rounding the island OUTPUT to bf16 → 90; f32 accumulation → + 91–106) is a COIN-FLIP that regresses, because it is not vLLM's ACTUAL GDN-Triton / FA2 + kernel arithmetic — it just perturbs which near-ties flip. Host-precision-matching PLATEAUS + at 120/128. + +### Verdict + default +**NO arm reaches STRICT** (best 120/128 is a DIVERGENCE; the golden is K=3 deterministic so +STRICT — not the distributional gate — is required). Per parity-enablers, `VT_KIMI_DEVICE_ +COMPUTE` and all three knobs STAY **OFF** (a near-tie is not token-exact). Row STAYS `ACTIVE`. + +### The named residual — the device islands (why host-matching cannot close p7) +The one principled path to STRICT is routing the two islands through vLLM's ACTUAL device +kernels, but it is NOT a drop-in (the mission's own assessment, now proven by measurement): +- **KDA:** vLLM's decay is **per-k-channel** `g[T,H,D]` (`kimi_gdn_linear_attn.py` + + `third_party/flash_linear_attention/ops/kda.py`), but our `vt::GdnDecode`/`GdnPrefill` + carry only a **per-HEAD scalar** decay `g/beta[T,Hv]` (`include/vt/ops.h:1797,1846`). So the + vendored GDN Triton-AOT cubins CANNOT express KDA — a NEW per-channel-decay GDN kernel + (`g[T,H,D]` + the `-exp(A_log)*softplus(f_b(f_a(x))+dt_bias)` gate) is required. +- **NoPE-MLA:** needs the paged `mla::ForwardMlaAttentionBlock` (FA2) over the runner's het-KV + (the born-on-runner residual), not a host softmax. +These are ALSO the speed levers — the correctness vehicle re-computes the WHOLE sequence every +decode step (O(n²)) with a host Download/upload per KDA/MLA layer per step. That is why the +measured tok/s is invariant to the numeric knobs. + +### Speed (HW-forced-indirect — vLLM cannot serve this model on one GB10 at bf16) +Steady tok/s over 127 steps (single-load, medians, cold first-leg discarded): control **1.31**, +islands **1.30**, resIsl **1.30** (first-step ~0.62 s, steady ~0.77 s/step). The extra bf16 +rounding casts cost ~0.15 s/step vs the §13 1.59 baseline; ALL configs are the same O(n²) +full-recompute + host-island rate. **Honest bar:** the §12 oracle golden capture itself needed +`gpu_memory_utilization=0.82` (~97.6 GiB) with only 15 GiB min-avail for a SINGLE-seq eager +run — vLLM cannot SERVE Kimi-Linear-48B at bf16 on ONE GB10 with any KV headroom, so a direct +`vllm bench throughput` arm is HW-infeasible; the comparison is recorded as HW-forced-indirect +(our absolute 1.30 tok/s + the per-step GPU-active anchor). No isolated host-tail lever (grouped +MoE seam, on-GPU sampling) moves the needle without the device-island + paged-incremental-decode +rewrite, which is the SAME W7-speed residual. Scoped as the named follow-up, not forced. + +--- + ## Structured contract (machine-readable — mirrors deepseek-v4-flash.md) ## Scope diff --git a/.agents/state.md b/.agents/state.md index 2d7eafdac..48395ec16 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -40773,3 +40773,26 @@ sensitivity. Clean-reference disambiguation blocked (bf16 132GiB = OOM on 1 GB10 = 23G-disk-blocked); path forward = an official modelopt-NVFP4 ckpt. fp4-resident Marlin arm's separate grid untouched (wiring-gated-only). Diagnostic hooks landed byte-inert. Records: spec §8.12 + §8.2 row, STATUS/BENCHMARKS/FEATURES H3 rows, benchmark-record, NOW. Box left clean. + +## 2026-08-07T09:45 — Kimi-Linear-48B W7-speed STRICT lever MEASURED: bf16 regime 106→120/128, plateaus; device islands remain the residual + +KIMI-LINEAR-STRICT-SPEED (`row/KIMI-LINEAR-STRICT-SPEED`, helper) — implemented + MEASURED the recorded +STRICT path ("device islands + bf16 residual stream") as three env-gated numeric knobs in +`kimi_linear_device.cpp` (default OFF → f32 vehicle byte-identical, CPU gate `test_kimi_linear_forward` +13/13·656 in the CUDA binary). Clean CUDA build on GB10 (sm_121a, cutlass-4.5.0, 14 GDN AOT symbols), +full 48.9B model, §12 128-token gate vs the STRICT deterministic golden, both flock locks + reclaim-waits, +min-avail ≥115 GiB, no reboot. RESULT (token match /128): control 106; `VT_KIMI_BF16_RESIDUAL` alone 106 +(net-zero — shuffles flips, BREAKS p3 into a `163586×` repeat); `VT_KIMI_BF16_ISLANDS` alone 106 (fixes p2 +but destabilizes p3); **residual + islands = 120/128 BEST** (p0–p6 all 16/16; only p7 pos-8 flips: golden +`18705` vs ours `58084`, a single near-tie that cascades). The two levers INTERACT — island input-rounding +fixes p2 but repeats p3; the bf16 residual re-stabilizes p3. Further precision-matching MEASURED NEGATIVE +(island-output bf16 → 90, reverted; f32 accumulation → 91–106, kept as documented-negative A/B knob): +host-precision-matching PLATEAUS at 120/128 because it is not vLLM's ACTUAL GDN-Triton/FA2 arithmetic. +VERDICT: NO arm STRICT (golden is K=3-deterministic → STRICT, not distributional); default STAYS OFF +(parity-enablers). NAMED residual (also the speed lever): the device islands — a NEW per-channel-decay GDN +kernel (`g[T,H,D]` per `kda.py`; `vt::GdnDecode`/`GdnPrefill` carry only per-head `g[T,Hv]`, ops.h:1797,1846 +— NOT a drop-in) + the paged `mla::ForwardMlaAttentionBlock` (FA2). Speed: 1.30 tok/s (O(n²) full-recompute ++ host islands; invariant to the numeric knobs, confirming the cost is the recompute structure not the +arithmetic); vLLM can't SERVE Kimi-Linear-48B at bf16 on ONE GB10 (oracle capture needed util 0.82 for a +single-seq eager run) → HW-forced-indirect. SHAs e048f4ee/3d6f81d1/(revert) on the row branch; PR pending. +Records: spec §14, STATUS/BENCHMARKS/FEATURES Kimi rows, benchmark-record, NOW. Box left clean. diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 025f86802..e2bddf892 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -309,7 +309,7 @@ built on it rather than keeping the flattering one. | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids | | `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (request aliases, the MP4 content route, and reference conditioning wiring), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | -| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident, RSS peak 1.7 GiB, min-avail 21 GiB, no OOM). Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle). 1.59 tok/s. Detail: spec §13 | +| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | e2e RUNS (bf16-resident §13); bf16-regime knobs 106→120/128 (7/8 exact), NOT STRICT; default OFF | bf16 residual+island-inputs → 120/128 best (control/each-alone 106; output-bf16 & f32-accum NEGATIVE); 1 near-tie left. 1.30 tok/s (O(n²)); vLLM HW-can't-serve bf16 on 1 GB10. Residual = device islands. §14 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. Detail: benchmark-record + spec §8 | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index e4245f5eb..9bbc87d8e 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -64,6 +64,9 @@ portable/reference path. In normal operation leave them unset. | `VT_MARLIN_DENSE` | off (opt-in) | `=1` routes the E=1 dense NVFP4/MXFP4 projections (`dense_nvfp4_gemm.h` `MatmulNvfp4MarlinD`/`GateUpFusedMarlinD`) through vLLM's OWN dense marlin GEMM (`vt::MarlinDenseGemm`) instead of the single-expert MoE-marlin route. The dense kernel is direct-A + tile-per-CTA with vLLM's own dense fp32-C_tmp reduce, so at M≤8 it runs the sms-wide (48-CTA) grid `VT_MARLIN_E1_PAR1` targets but WITHOUT that flag's par-regroup ULP — the byte-preserving fix for the strict-32B token flip (row `KERNEL-MARLIN-DENSE-PORT`). Reuses the same marlin resident + workspace (shared `marlin_permute` repack). **DEFAULT OFF** until the strict token battery proves oracle byte-match and the binding beats the MoE route; then flipped ON per the parity-enabler policy. CUDA-only (needs `VT_MARLIN_NVFP4`) | | `VT_MM_DECODE_EAGER` | off (graph on) | Set to `1` to force the eager per-step multimodal (Qwen3.6-27B image/video) decode instead of routing it through the captured dense decode graph. Rollback / A-B knob; the graphed path is token-exact with the eager path | | `VT_KIMI_DEVICE_COMPUTE` | off (opt-in) | `=1` routes the Kimi-Linear-48B-A3B runner path (`KimiLinearModel::ForwardDevice`) through the W7 DBuf-resident device COMPUTE (`ForwardDeviceCompute`, the whole KDA/NoPE-MLA + MoE hybrid over pooled DBufs via the shared vt:: ops) instead of the default W6 host-reference compose. Default OFF keeps the CPU-verified host-ref-compose seam as production until the device compute is GPU-verified against the SACRED oracle; the device compute is CPU-gated (`test_kimi_linear_forward`, device==W2 reference within f32-accumulation tolerance, greedy-token-identical) but its GPU numerics are a NAMED pending. The flag exists so the device path CAN be exercised as the runner path for that verification | +| `VT_KIMI_BF16_RESIDUAL` | off (opt-in) | `=1` carries the Kimi-Linear device-compute residual stream in bf16 like vLLM's `fused_add_rms_norm` (residual/block-outputs bf16, RMSNorm variance over the f32 pre-store sum), via in-place f32→bf16→f32 rounds. W7-speed STRICT-lever A/B (spec §14). Default OFF → byte-identical. MEASURED: alone net-zero; WITH `VT_KIMI_BF16_ISLANDS` → 120/128 (best, still a near-tie, NOT STRICT) | +| `VT_KIMI_BF16_ISLANDS` | off (opt-in) | `=1` rounds the Kimi-Linear host-fallback island INPUTS (KDA q/k/v/g1/beta, NoPE-MLA q/kv/kpe) to bf16 (RNE) before the recurrence/softmax, toward vLLM's GDN-Triton/FA2 kernel precision. W7-speed STRICT-lever A/B (spec §14). Default OFF → byte-identical. MEASURED best config paired with `VT_KIMI_BF16_RESIDUAL` (106→120/128) | +| `VT_KIMI_ISLAND_F32ACC` | off (opt-in) | `=1` computes the Kimi-Linear island recurrence/softmax in f32 accumulation (not f64). W7-speed A/B knob, **MEASURED NEGATIVE** (91–106/128; kept as a documented-negative A/B, spec §14). Default OFF → byte-identical | | `VT_WHISPER_ENC_EAGER` | off (flash-tiled attention on) | Set to `1` to force the naive per-key block-reduction attention in the Voxtral/Whisper audio encoder instead of the default flash-tiled kernel. Rollback / A-B knob; token-identical to the default path | | `VT_WHISPER_ENC_WARP` | off (flash-tiled attention on) | Set to `1` to force the warp-scoped online-softmax attention (`vt::AttentionDenseFast`, the pre-flash default) in the Voxtral/Whisper audio encoder instead of the default flash-tiled kernel (`vt::AttentionDenseFlash`). Rollback / A-B knob; the flash-tiled path is bit-identical to the warp path (encoder self-attention ~1.82x faster) | | `VT_WHISPER_ENC_REMARSHAL` | off (encoder weights resident) | Set to `1` to disable device-resident encoder weights and re-marshal (host f32->bf16 convert + H2D upload) all Whisper/Voxtral encoder weights on EVERY forward, restoring the pre-residency behavior. Rollback / A-B knob; byte-identical output (moves data only). Default residency uploads each encoder weight once and reuses it, removing ~648 ms of per-call host marshalling from the encoder forward | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index afbbef082..c188e198e 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -119,7 +119,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on | -| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | e2e runs bf16-resident, near-tie 106/128 (numerics) | 1.59 tok/s, default off | +| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | e2e runs bf16-resident; bf16-regime knobs 106→120/128 (7/8 exact), not STRICT | 1.30 tok/s, default off; residual = device islands (§14) | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | | `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run | @@ -250,7 +250,7 @@ abstraction, and `world_size == 1` stays byte-identical. | Gap | State | Detail | |---|---|---| -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128, pool math CLOSES; default OFF | Full model RUNS on GB10 (bf16-resident): host RSS 1.7 GiB, min-avail 21 GiB, no OOM. Token NEAR-TIE 106/128 (6/8 prompts exact, numerics vs deterministic oracle); STRICT path = device islands + bf16 stream | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | e2e RUNS (bf16-resident §13); bf16-regime knobs 106→120/128 (7/8 exact), NOT STRICT; default OFF | bf16 residual+island-inputs → 120/128 best (§14); 1 near-tie left. STRICT+speed residual = device islands (per-channel GDN kernel + paged FA2). 1.30 tok/s; vLLM HW-can't-serve bf16 on 1 GB10 | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Architecturally blocked | Vision tower lives outside the registered engine forward | diff --git a/docs/STATUS.md b/docs/STATUS.md index aa22dcab1..7afcd4684 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -74,7 +74,7 @@ token-for-token correctness against the pinned oracle. | DeepSeek-V2 MLA | Correctness-complete, speed-pending | Token-exact 8/8 (DeepSeek-V2-Lite); 0.86-0.95x output rate, TTFT faster at c4/c8. A2+A5 MLA norm-rope fold default-ON (`VT_MLA_FUSED_NORM_ROPE`, bit-exact rollback, SACRED 8/8 unchanged; forensics in benchmark-record) — kimi_k3/kimi-linear inherit it | | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | Full-model GB10 e2e RUNS (bf16-resident §13), NEAR-TIE 106/128 (6/8 prompts token-exact), pool math CLOSES; default OFF | CPU+CUDA 13/13·656. Full model RUNS on GB10: host RSS 1.7 GiB, min-avail 21 GiB, no OOM. Token NEAR-TIE 106/128 (numerics vs deterministic oracle, not a bug); STRICT path = device islands + bf16 stream; 1.59 tok/s | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | e2e RUNS (bf16-resident §13); bf16-regime knobs 106→120/128 (7/8 exact), NOT STRICT; default OFF | CPU+CUDA 13/13·656. bf16 residual+island-inputs → 120/128 (§14); 1 near-tie left (p7). STRICT+speed residual = device islands (per-channel GDN kernel + paged FA2). 1.30 tok/s; vLLM HW-can't-serve bf16 on 1 GB10 | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** | diff --git a/src/vllm/model_executor/models/kimi_linear_device.cpp b/src/vllm/model_executor/models/kimi_linear_device.cpp index 7b518ae37..662400ced 100644 --- a/src/vllm/model_executor/models/kimi_linear_device.cpp +++ b/src/vllm/model_executor/models/kimi_linear_device.cpp @@ -83,6 +83,97 @@ bool FusedGlue() { inline double Sigmoid(double x) { return 1.0 / (1.0 + std::exp(-x)); } +// ─── STRICT-path numerics knobs (W7-speed lane) ──────────────────────────────── +// The correctness vehicle is MORE precise than vLLM (f32 residual stream + f64 host +// islands), which flips near-tie argmaxes where vLLM's deterministic bf16 top-1 has +// a small margin (spec §13 root cause). These two knobs mirror vLLM's bf16 compute +// regime so the near-ties become token-exact. Both default OFF → the f32 vehicle is +// byte-identical (the CPU tiny-config gate stays 13/13·656); flip ON only for the +// full-model on-box token gate vs the STRICT golden. +// +// (1) VT_KIMI_BF16_RESIDUAL — carry the residual stream in bf16 like vLLM. vLLM's +// fused_add_rms_norm stores `residual` as bf16 and `hidden_states`/block-outputs are +// bf16, while it computes the RMSNorm variance over the f32 pre-store sum +// (layernorm.py / fused_add_rms_norm.cu). We keep f32 STORAGE (so the two host-fallback +// islands still consume f32) but round the VALUE to bf16 precision at exactly vLLM's +// rounding points: the embed output, each block output before it re-enters the add, +// and the residual AFTER each add (so the norm still sees the f32 sum — byte-matching +// vLLM's fused-add-rms-norm order). 54 bf16 roundings across 27 layers × 2 norms that +// vLLM does and our f32 vehicle does not. +bool Bf16Residual() { + static const bool on = [] { + const char* e = std::getenv("VT_KIMI_BF16_RESIDUAL"); + return e != nullptr && e[0] == '1'; + }(); + return on; +} + +// (2) VT_KIMI_BF16_ISLANDS — round the host-fallback island INPUTS (KDA recurrence +// q/k/v/g1/beta, MLA softmax q/kv/kpe) to bf16 precision before the f64 recurrence. +// vLLM feeds bf16 activations into the GDN Triton-AOT / FA2 kernels; our islands +// download f32 (bf16-precision projection outputs stored to f32) and recompute in f64. +// Rounding the inputs to bf16 moves the island toward vLLM's kernel precision without a +// new device kernel (the true fix — the device GDN per-channel-decay recurrence + the +// paged FA2 MLA — is the named W7-speed residual, spec §13). +bool Bf16Islands() { + static const bool on = [] { + const char* e = std::getenv("VT_KIMI_BF16_ISLANDS"); + return e != nullptr && e[0] == '1'; + }(); + return on; +} + +// Round an f32 value to bf16 precision (round-to-nearest-even), matching torch/vLLM's +// bf16 cast (and vt::CastBf16). Truncate-with-RNE-bias on the top 16 bits; qNaN-safe. +inline float ToBf16Rne(float f) { + uint32_t u; + std::memcpy(&u, &f, sizeof(u)); + if ((u & 0x7fffffffu) > 0x7f800000u) { + u |= 0x00400000u; // qNaN + } else { + u += 0x00007fffu + ((u >> 16) & 1u); // RNE rounding bias + } + u &= 0xffff0000u; + float r; + std::memcpy(&r, &u, sizeof(r)); + return r; +} +inline void RoundHostBf16(std::vector& v) { + if (!Bf16Islands()) return; + for (float& x : v) x = ToBf16Rne(x); +} + +// (3) VT_KIMI_ISLAND_F32ACC — compute the host-fallback island recurrence/softmax in +// f32 accumulation (not f64), matching vLLM's GDN Triton / FA2 kernels (bf16 I/O, f32 +// accumulation). Our island defaults to f64 (MORE precise than vLLM); rounding each +// accumulation step to f32 mirrors the device kernel's rounding. Combined with +// VT_KIMI_BF16_ISLANDS (bf16 I/O), this is the closest host approximation of vLLM's +// actual kernel numerics without a new device kernel (the named W7-speed residual). +bool IslandF32Acc() { + static const bool on = [] { + const char* e = std::getenv("VT_KIMI_ISLAND_F32ACC"); + return e != nullptr && e[0] == '1'; + }(); + return on; +} +// Round a running f64 accumulator to f32 precision when the knob is on (identity else). +inline double AccR(double x) { + static const bool f32 = IslandF32Acc(); + return f32 ? static_cast(static_cast(x)) : x; +} + +// In-place round an f32 device buffer to bf16 precision (f32→bf16→f32, on-device, no +// download). The VALUE becomes bf16-exact so the RMSNorm variance and the next residual +// add see the same bf16 numbers vLLM does; the STORAGE stays f32 (the islands read f32). +void RoundDevBf16(const Dev& d, DBuf& x) { + Tensor xt = x.t(); + std::vector shape(xt.shape, xt.shape + xt.rank); + DBuf b(d, DType::kBF16, shape); + vt::CastBf16(d.q, b.t(), xt); + Tensor out = x.t(); + vt::CastF32(d.q, out, b.t()); +} + // Device-resident weight view. On CPU this ALIASES the host f32 bytes exactly as // dense_attn::ResidentWeight does for a CPU device (host-pointer aliasing is a CPU // property); the CUDA staging over materialized OwnedTensors is the born-on-runner @@ -197,6 +288,13 @@ DBuf KdaRecurrenceIsland(const Dev& d, DBuf& qn, DBuf& kn, DBuf& vc, DBuf& g1, vc.Download(d, hv.data()); g1.Download(d, hg1.data()); braw.Download(d, hbraw.data()); + // VT_KIMI_BF16_ISLANDS: feed bf16-precision inputs to the recurrence (like vLLM's + // GDN kernel), keeping the f64 accumulation. No-op when the knob is off. + RoundHostBf16(hqn); + RoundHostBf16(hkn); + RoundHostBf16(hv); + RoundHostBf16(hg1); + RoundHostBf16(hbraw); const std::vector g = kimi_kda::KdaDecayGate(hg1, a_log, dt_bias, T, nh, hd); // [T,nh,hd] @@ -215,26 +313,26 @@ DBuf KdaRecurrenceIsland(const Dev& d, DBuf& qn, DBuf& kn, DBuf& vc, DBuf& g1, double* Sp = &S[static_cast(h) * hd * hd]; for (int64_t vd = 0; vd < hd; ++vd) { double* Sr = &Sp[vd * hd]; - for (int64_t k = 0; k < hd; ++k) Sr[k] *= std::exp(static_cast(gh[k])); + for (int64_t k = 0; k < hd; ++k) Sr[k] = AccR(Sr[k] * std::exp(static_cast(gh[k]))); } for (int64_t vd = 0; vd < hd; ++vd) { const double* Sr = &Sp[vd * hd]; double pred = 0.0; - for (int64_t k = 0; k < hd; ++k) pred += Sr[k] * knp[k]; - u[static_cast(vd)] = (static_cast(vvp[vd]) - pred) * b; + for (int64_t k = 0; k < hd; ++k) pred = AccR(pred + Sr[k] * knp[k]); + u[static_cast(vd)] = AccR((static_cast(vvp[vd]) - pred) * b); } for (int64_t vd = 0; vd < hd; ++vd) { double* Sr = &Sp[vd * hd]; const double uv = u[static_cast(vd)]; - for (int64_t k = 0; k < hd; ++k) Sr[k] += uv * knp[k]; + for (int64_t k = 0; k < hd; ++k) Sr[k] = AccR(Sr[k] + uv * knp[k]); } float* cr = &core[static_cast(base)]; for (int64_t vd = 0; vd < hd; ++vd) { const double* Sr = &Sp[vd * hd]; double o = 0.0; for (int64_t k = 0; k < hd; ++k) - o += Sr[k] * (static_cast(qnp[k]) * scale); - cr[vd] = static_cast(o); + o = AccR(o + Sr[k] * (static_cast(qnp[k]) * scale)); + cr[vd] = static_cast(o); // f32 output (bf16-rounding the output MEASURED -14, reverted) } } } @@ -258,6 +356,10 @@ DBuf MlaSoftmaxIsland(const Dev& d, DBuf& dq, DBuf& dkv, DBuf& dkpe, dq.Download(d, hq.data()); dkv.Download(d, hkv.data()); dkpe.Download(d, hkpe.data()); + // VT_KIMI_BF16_ISLANDS: bf16-precision inputs to the softmax core (like vLLM's FA2). + RoundHostBf16(hq); + RoundHostBf16(hkv); + RoundHostBf16(hkpe); const double scale = std::pow(static_cast(qk), -0.5); std::vector out(static_cast(T) * nah * vh, 0.0f); std::vector sc(static_cast(T)); @@ -271,27 +373,27 @@ DBuf MlaSoftmaxIsland(const Dev& d, DBuf& dq, DBuf& dkv, DBuf& dkpe, const float* kpe = &hkpe[static_cast(s * qr)]; double dot = 0.0; for (int64_t dd = 0; dd < qn; ++dd) - dot += static_cast(q_nope[dd]) * k_nope[dd]; + dot = AccR(dot + static_cast(q_nope[dd]) * k_nope[dd]); for (int64_t dd = 0; dd < qr; ++dd) - dot += static_cast(q_pe[dd]) * kpe[dd]; - dot *= scale; + dot = AccR(dot + static_cast(q_pe[dd]) * kpe[dd]); + dot = AccR(dot * scale); sc[static_cast(s)] = dot; mx = std::max(mx, dot); } double sum = 0.0; for (int64_t s = 0; s <= t; ++s) { - const double e = std::exp(sc[static_cast(s)] - mx); + const double e = AccR(std::exp(sc[static_cast(s)] - mx)); sc[static_cast(s)] = e; - sum += e; + sum = AccR(sum + e); } float* ot = &out[static_cast(t * nah * vh + h * vh)]; for (int64_t dd = 0; dd < vh; ++dd) { double acc = 0.0; for (int64_t s = 0; s <= t; ++s) { const float* vs = &hkv[static_cast(s * kvw + h * (qn + vh) + qn)]; - acc += (sc[static_cast(s)] / sum) * static_cast(vs[dd]); + acc = AccR(acc + (sc[static_cast(s)] / sum) * static_cast(vs[dd])); } - ot[dd] = static_cast(acc); + ot[dd] = static_cast(acc); // f32 output (bf16-rounding the output MEASURED -14, reverted) } } } @@ -807,6 +909,8 @@ DBuf DeviceForwardBodyBf16(const Dev& d, const KimiLinearWeights& weights, Tensor hh = hidden.t(); vt::Embedding(d.q, hh, htab, dids.t()); } + const bool bf16_res = Bf16Residual(); + if (bf16_res) RoundDevBf16(d, hidden); // vLLM embed output is bf16 DBuf res(d, DType::kF32, {T, H}); res.Zero(d); Tensor hcur = hidden.t(); @@ -816,12 +920,16 @@ DBuf DeviceForwardBodyBf16(const Dev& d, const KimiLinearWeights& weights, const KimiLinearLayerResidentWeights& lw = rw.layers[static_cast(l)]; DBuf dhn(d, DType::kF32, {T, H}); AddRmsNorm(d, dhn, hcur, WF32(d, lw.input_layernorm, {H}), res, eps); + if (bf16_res) RoundDevBf16(d, res); // vLLM stores residual bf16 (variance saw f32 sum) DBuf attn = lw.is_kda ? KdaLayerDeviceBf16(d, lw.kda, dhn.t(), p, T) : MlaLayerDeviceBf16(d, lw.mla, dhn.t(), p, T); + if (bf16_res) RoundDevBf16(d, attn); // vLLM attn_output is bf16 DBuf dh2(d, DType::kF32, {T, H}); AddRmsNorm(d, dh2, attn.t(), WF32(d, lw.post_attention_layernorm, {H}), res, eps); + if (bf16_res) RoundDevBf16(d, res); DBuf mlp = lw.is_moe ? MoeBlockDeviceBf16(d, lw.moe, dh2.t(), p, T) : DenseMlpDeviceBf16(d, lw.dense, dh2.t(), p, T); + if (bf16_res) RoundDevBf16(d, mlp); // vLLM mlp output is bf16 auto* held = new DBuf(std::move(mlp)); hcur = held->t(); hold = std::shared_ptr(held, [](void* q) { delete static_cast(q); });