From 6a368df7b7926910239f8bd1185bbcef5f6a089b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 10:40:27 +0000 Subject: [PATCH] =?UTF-8?q?feat(kimi-linear):=20device=20NoPE-MLA=20attent?= =?UTF-8?q?ion=20lever=20(VT=5FKIMI=5FDEVICE=5FMLA)=20=E2=80=94=20MEASURED?= =?UTF-8?q?-NEGATIVE=20on=20GB10,=20STRICT=20owes=20the=20actual=20FA2/chu?= =?UTF-8?q?nk=5Fkda=20kernels=20(#107)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The spec §15 residual (d) — the 7 NoPE-MLA layers still run a host f64 softmax island that is MORE precise than vLLM's FA2 and coin-flips near-ties — was attempted in its device-COMPUTE form (the §15 device-KDA pattern applied to the MLA half) and MEASURED NEGATIVE on the full 48.9B GB10 gate. Implementation (additive, default OFF): `VT_KIMI_DEVICE_MLA` + `MlaAttnCoreDevice` (kimi_linear_device.cpp) route the NoPE causal softmax over per-head [k_nope|k_pe(shared)]/v through the shared device op vt::Attention (f32 online max-subtracted softmax = vLLM's FA2 regime) instead of the f64 host MlaSoftmaxIsland. vt::Attention carries one head-dim; MLA is asymmetric (qk=qk_nope+qk_rope=192, v=128), so value is zero-PADDED to qk — the weighted sum over the 0 tail is 0, so out[:,:,:v] is byte-exact. Wired into the f32 and bf16 paths. MLA dims VERIFIED from the real 48.9B config.json (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA) — not the K3 nums. Unit gate (RED-first, CPU) GREEN: test_kimi_linear_forward 14/14·825 (was 13/13·656) — NEW case (g2) KimiMlaAttnCoreDevice (pad-V + vt::Attention) == a from-first-principles f64 causal-softmax reference (rtol 3e-3); a perturbed scale fails 108 assertions. Env-gated whole-forward green (VT_KIMI_DEVICE_MLA=1 alone and with VT_KIMI_DEVICE_KDA=1). Same on the GB10 CUDA binary (210 GDN + 23 KDA syms linked, CUTLASS-NVFP4 + FA2 + Triton-AOT). Full 48.9B GB10 gate vs the §12 STRICT golden (single-load per config, flock gpu.lock, min-avail 21 GiB, no reboot): control device-KDA reproduces 122/128 + 4.24 tok/s EXACTLY; +device-MLA REGRESSES to 109/128 AND 3.89 tok/s. Why (the §14 razor, re-proven): device-KDA works because its recurrence is the SAME algorithm as vLLM's decode kernel (f32-on-bf16); but vLLM's MLA prefill uses FA2 (a specific reduction ORDER) and vt::Attention's plain online-softmax is the right MATH in a DIFFERENT order, so it coin-flips near-ties (breaks p3 16→3 into the §14 163586x repeat) while p7 stays diverged, and the per-(t,h) build + pad-V waste slow the O(n^2) recompute. An approximation of vLLM's kernel is not enough. VERDICT: VT_KIMI_DEVICE_MLA STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (parity-lever precedent: §14 ISLAND_F32ACC). device-KDA (122/128, 4.24) remains the best config, itself default OFF. The one-brick STRICT-close did NOT land. STRICT residual, sharpened: needs vLLM's ACTUAL kernels — (c) chunk_kda prefill family (Triton-AOT regen for sm_121a) + (d) paged FA2 mla::ForwardMlaAttentionBlock (NOT this vt::Attention approximation) + (e) paged-incremental decode (needs a decode/paged-attn op, query_len != key_len). Records: spec §16, benchmark-record entry, state.md anchored entry, NOW.md + model-matrix row + docs/STATUS/BENCHMARKS/FEATURES one-liners. All doc/record gates green. Box left clean (build tree/markers removed, worker restored, golden preserved, no reboot). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-opus-4-8 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/benchmark-record.md | 17 +++ .agents/model-matrix.md | 2 +- .agents/specs/kimi-linear.md | 59 ++++++++ .agents/state.md | 33 +++++ docs/BENCHMARKS.md | 2 +- docs/FEATURES.md | 4 +- docs/STATUS.md | 2 +- .../vllm/model_executor/models/kimi_linear.h | 7 + .../models/kimi_linear_device.cpp | 132 +++++++++++++++++- .../vllm/models/test_kimi_linear_forward.cpp | 62 ++++++++ 11 files changed, 315 insertions(+), 7 deletions(-) diff --git a/.agents/NOW.md b/.agents/NOW.md index 5f4368281..1fc0dd599 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -18,7 +18,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool | | Invocation-parity prevention | CI guard + checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx | | MiniMax-H3 lane | **fl2va COHERENT; ref2va grid DIAGNOSED (#95): NO loader bug; bf16 13-shard DiT INDEXES** | residual = community-NVFP4 quant fidelity §8.12; no bf16 render yet | -| Kimi-Linear-48B (KDA/NoPE-MLA/MoE) | **`vt::KdaGatedDeltaRule` LANDED + GB10-MEASURED** (§15, `row/KIMI-KDA-DEVICE-KERNEL`): **106→122/128 + 1.35→4.24 tok/s (3.1×)**, beats §14's 120. OFF (122≠STRICT) | close p7: chunked-prefill/paged-MLA/incremental | +| Kimi-Linear-48B (KDA/NoPE-MLA/MoE) | device-KDA **122/128 + 4.24 tok/s** best (§15); device NoPE-MLA `VT_KIMI_DEVICE_MLA` **MEASURED-NEGATIVE 122→109** (§16, #107 — f32-softmax≠FA2, coin-flips). Both OFF ≠STRICT | STRICT owes ACTUAL kernels: chunk_kda + paged FA2 MLA + incremental | | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index e98b25126..300e2a835 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -14609,3 +14609,20 @@ The §14 named residual ("a NEW per-channel-decay GDN kernel `g[T,H,D]`; `vt::Gd RESULT (the §14 thesis CONFIRMED): the device recurrence — vLLM's ACTUAL f32-on-bf16 arithmetic — moves **106→122/128** (prompts 0-6 all 16/16; only p7 diverges at pos-6, got `387` vs golden `11`, a comma near-tie) AND is **3.1× FASTER (1.35→4.24 tok/s)**. It beats BOTH the control (106) AND §14's host-precision best (120, which needed both bf16 knobs), and FIXES the p2 divergence the f64 host path had — because it runs the right arithmetic, not a coin-flip. The §14 bf16 knobs are now SUPERSEDED + COUNTERPRODUCTIVE: device-KDA + bf16 REGRESSES 122→90 (reintroducing p3's `163586×` repeat loop) — they were tuned to compensate for the f64 host island's over-precision. The speed win = the device recurrence kills the per-step host Download/f64-recompute/upload round-trip and runs the O(T²) recurrence in parallel on the GPU. VERDICT: device-KDA (122/128, 4.24 tok/s) is the NEW BEST on BOTH axes but STILL a DIVERGENCE (STRICT required, K=3-deterministic golden) → `VT_KIMI_DEVICE_KDA` STAYS OFF (parity-enablers). The residual is now a SINGLE near-tie (p7 pos-6). NAMED next brick to STRICT (+ more speed): the KDA chunked-prefill kernel family (vLLM processes the PROMPT with `chunk_kda`, we still run the recurrent form — regen a Triton-AOT cubin for sm_121a via `scripts/regen-triton-aot.sh`, or a native `chunk_kda` port) + paged `mla::ForwardMlaAttentionBlock` for the 7 NoPE-MLA layers + paged-incremental decode (persistent KDA state + MLA-KV, kills the remaining O(n²)). Row STAYS ACTIVE. + +## 2026-08-07 — Kimi-Linear-48B: device NoPE-MLA attention lever `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE (122→109/128 AND slower); STRICT still owes vLLM's ACTUAL FA2/chunk_kda kernels (`row/KIMI-STRICT-CLOSE`, #107) + +The §15 residual (d) — "the 7 NoPE-MLA layers still use a host f64 softmax island; closing p7 needs paged `mla::ForwardMlaAttentionBlock`" — was attempted in its device-COMPUTE form (the §15 device-KDA pattern applied to the MLA half) and MEASURED NEGATIVE on GB10. Additive knob `VT_KIMI_DEVICE_MLA` + `MlaAttnCoreDevice` (`kimi_linear_device.cpp`): the NoPE causal softmax over per-head `[k_nope|k_pe(shared)]`/`v` runs through the shared device op `vt::Attention` (f32 online max-subtracted softmax — vLLM's FA2 accumulation regime) instead of the f64 host `MlaSoftmaxIsland`. `vt::Attention` carries ONE head-dim for q/k/v while MLA is asymmetric (`qk = qk_nope+qk_rope = 192`, `v = 128`), so the value is zero-PADDED to `qk` — the weighted sum over the zero tail is 0, so `out[:, :, :v]` is byte-exact (softmax weights depend only on `q·k`). MLA dims VERIFIED from the real 48.9B `config.json` (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn `[4,8,12,16,20,24,27]`, 20 KDA) — not the K3 numbers. + +Unit gate (RED-first, CPU) GREEN: `test_kimi_linear_forward` **14/14·825** (was 13/13·656) — NEW case (g2) `KimiMlaAttnCoreDevice` (pad-V + `vt::Attention`) == a from-first-principles f64 causal-softmax reference at the Kimi MLA geometry (rtol 3e-3); RED-first verified (a perturbed scale fails 108 assertions). Env-gated whole-forward runs green (`VT_KIMI_DEVICE_MLA=1` alone and with `VT_KIMI_DEVICE_KDA=1`). Same on the GB10 CUDA binary (clean Release, CUTLASS-NVFP4 + FA2 + Triton-AOT, 210 GDN + 23 KDA syms linked). + +Full 48.9B GB10 gate vs the §12 STRICT `greedy_ids.npy` (single-load per config, `flock $HOME/gpu.lock`, `drop_caches` before wall-clock, min-avail **21 GiB**, no reboot, worker parked+restored): + +| Config | env (all `VT_KIMI_DEVICE_COMPUTE=1`) | /128 | tok/s | verdict | +|---|---|---|---|---| +| control (device-KDA) | `DEVICE_KDA=1` | 122 | 4.24 | reproduces §15 EXACTLY (p0-p6 16/16, p7 10/16) | +| **+ device-MLA** | `DEVICE_KDA=1 DEVICE_MLA=1` | **109** | **3.89** | **REGRESSION on BOTH axes** | + +WHY NEGATIVE (the §14 razor, re-proven). device-KDA WORKS (106→122) because the recurrence is the SAME algorithm as vLLM's decode kernel, only f32-on-bf16 — it MATCHES. But vLLM's MLA prefill uses FA2 (a specific flash tiling/reduction ORDER); `vt::Attention`'s plain f32 online-softmax is the right MATH but a DIFFERENT reduction order, so — exactly like §14's host-precision plateau — it COIN-FLIPS near-ties: it BREAKS p3 16/16→3/16 (got `220,41938,382,1810,…163586,163586` — the same `163586×` degenerate repeat §14's bf16 knobs caused) while p7 stays diverged at 10/16. And it is SLOWER (4.24→3.89): the per-`(t,h)` key/value build copies + the 192-dim pad-V waste add overhead to the O(n²) recompute path. An approximation of vLLM's kernel is not enough — only the ACTUAL kernel matches. + +VERDICT: `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (parity-lever precedent: §14's `ISLAND_F32ACC`). device-KDA (122/128, 4.24 tok/s) remains the best config, itself default OFF (122 ≠ STRICT). The one-brick STRICT-close did NOT land. STRICT residual, SHARPENED: needs vLLM's ACTUAL kernels, not a device approximation — (c) the **chunk_kda** prefill family (`chunk_kda_scaled_dot_kkt`+`recompute_w_u`+`chunk_gla_fwd_o_gk`+`fused_kda_gate_chunk_cumsum`, FLA `ops/kda.py`) via a Triton-AOT regen for sm_121a (`scripts/regen-triton-aot.sh` + new `triton_kernels/*.py`), the named prime suspect; (d) the paged FA2 `mla::ForwardMlaAttentionBlock` (NOT the `vt::Attention` approximation tried here); (e) paged-incremental decode (needs a decode/paged-attn op with `query_len≠key_len`, which `vt::Attention` cannot express; kills the O(n²)). Each is a substantial multi-kernel brick, recorded as the named follow-on. Row STAYS ACTIVE. HONEST bar stays HW-forced-indirect (vLLM cannot serve this bf16 on one GB10 with KV headroom — §14). diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 3380c571d..c0a792ad8 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -79,7 +79,7 @@ Engaged architectures (the 45 non-`INVENTORIED` rows): | 🚧 | `Glm4ForCausalLM` | GLM-4-9B-0414 (dense, sandwich norms + partial interleaved rope) | SACRED gate 16/16 vs vLLM 0.25.0 (STRICT token-exact 13/16 + near-tie-band 3/16, max gap 0 nats, 0 forward-divergent; vLLM K=5 self-deterministic → STRICT bar); speed pending | `MODEL-TEXT-glm4-glm4-for-causal-lm` | | 📋 | `Glm4MoeForCausalLM` | GLM-4 MoE | scoped in the GLM/DSA spike, not implemented | `MODEL-TEXT-glm4-moe-glm4-moe-for-causal-lm` | | ✅ | `Glm4MoeLiteForCausalLM` | GLM-4.7-Flash (31.2B MLA + GLM MoE) | SACRED gate 8/8 vs vLLM 0.25.0 (STRICT token-exact 1/8 + near-tie-band 7/8, 69/128 tokens strictly exact, max teacher-forced gap 0.0 nats, 0 forward-divergent; vLLM K=5 self-deterministic → STRICT bar); FIRST e2e coverage of the q_lora query branch AND the noaux_tc sigmoid router (closes the MLA campaign's two gaps, C2); speed pending | `MODEL-TEXT-glm4-moe-lite-glm4-moe-lite-for-causal-lm` | -| 🚧 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B | **FULL-MODEL GB10 e2e RUNS — NEAR-TIE 106/128 (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the bf16-resident path CLEARS the f32-loader block — the full 48.9B model now runs e2e on one GB10. dgx CUDA build (`-Werror` clean, 14 GDN AOT symbols nm-linked, `test_kimi_linear_forward` 13/13·656 in the CUDA binary); `kimi-linear-gen --gpu` greedy-decodes the §12 8-prompt battery x16 vs `greedy_ids.npy`. MEMORY: load 117.6s, host RSS PEAK **1.7 GiB** (stage-then-ReleaseHost), device peak 98.5 GiB, min-avail **21.6 GiB** (above the 15 GiB floor, matches the ~25 GiB pool-math headroom), NO OOM/reboot. TOKEN gate **NEAR-TIE 106/128 (82.8%)** — prompts 0,1,3,4,5,6 are 16/16 token-exact, p2/p7 diverge at punctuation/word near-ties; 96 consecutive exact tokens across 6 prompts prove the WIRING (a wiring bug can't). Root cause (honest): the f32 residual stream + host-f64 islands are MORE precise than vLLM's bf16 device kernels, so they flip the argmax where vLLM's deterministic bf16 top-1 has a small margin. STRICT path = the named W7-speed residuals (device GDN/MLA islands -> bf16 stream matching vLLM's rounding). 1.59 tok/s (recompute+island rate). `VT_KIMI_DEVICE_COMPUTE` STAYS OFF (parity-enablers: near-tie != token-exact). Row STAYS 🚧. **bf16-RESIDENT loader/forward IMPLEMENTED + CPU-gated (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the §13 design is coded — `LoadKimiLinearResidentBf16Weights`/`StageKimiResidentBf16`/`BuildKimiResidentFromHost` (`kimi_linear_weights.cpp`; `LoadBf16Direct` -> `OwnedTensor`, per-tensor stage-to-`d_dev` + `ReleaseHost`, tiny vectors host f32), `KimiLinearResidentWeights` (`kimi_linear.h`), bf16 device forward `DeviceForwardBodyBf16` + `Gemm Bf16` cast-act at ~20 GEMM sites with the two host-fallback islands EXTRACTED+shared (`kimi_linear_device.cpp`), `ForwardDevice` resident-path dispatch (`kimi_linear.cpp`), and the `kimi-linear-gen` e2e harness. CPU **13/13·656** (12/12·614 f32 path UNTOUCHED + NEW tiny-config bf16-vs-f32 gate). PENDING: dgx CUDA build + full-model GB10 e2e vs the STRICT golden. Row STAYS 🚧. **bf16-RESIDENT brick POOL-MATH+DESIGN (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** pool math CLOSES (91.5 GiB bf16 device-resident + ~2.4 GiB act/norms/ctx ≈ 94 GiB, ~25 GiB headroom); design grounded §13 (Laguna `GemmBf16` cast-act + `OwnedTensor::d_dev`, `LoadBf16Direct`, f32 `MaterializeHost` kept for the unit gate). Impl (loader/forward rewrite + gate + e2e) pending. Row STAYS 🚧. **§8 GOLDEN CAPTURED — STRICT (2026-08-06, `row/MODEL-KIMI-LINEAR-E2E`):** the §8 SACRED oracle golden is captured on GB10 (0.25.0-stage, util 0.82, moe=triton, min 15 GiB avail, NO reboot), **8/8 prompts DETERMINISTIC over K=3 → STRICT gate**, committed at `tests/parity/goldens/kimi_linear_greedy/`. Full our-engine e2e BLOCKED on OUR f32 loader (materializes ~183 GiB > 119 pool), the bf16-residency residual; row STAYS 🚧. **W7 GPU-VERIFY (2026-08-06, branch `row/MODEL-KIMI-LINEAR-GPU`):** the device compute runs **12/12·614 GREEN on GB10 sm_121a CUDA build**, BOTH arms (`VT_KIMI_DEVICE_COMPUTE=1` + host-ref); prod stack (CUTLASS-NVFP4 GEMM + FA2 ENABLED + Triton-AOT GDN, 14 cubins nm-verified); f32 device==W2 ref, no divergence, no DeepSeek-class trap. Oracle gateability re-confirmed (0.25.0-stage registers `KimiLinearForCausalLM`). e2e §8 SACRED golden STILL disk-blocked (91.5 GiB checkpoint absent, dgx root 100% full, 34G free). Row STAYS 🚧. **W7 DBuf-resident device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`): the real device compute (`ForwardDeviceCompute`, `kimi_linear_device.cpp`) composes the whole 27-layer KDA/NoPE-MLA + 256-expert-MoE hybrid over pooled f32 `DBuf`s through the SHARED `vt::` ops (embed/`FusedChain` add+RMSNorm/`MatmulBT` projections/`CausalConv1dFwd` convs/`L2Norm`/`RmsNormGated`/`MoeRouterTopK` sigmoid-`noaux_tc`/`MoeSiluMul`/`MoeCombine`/lm_head), returning DEVICE-RESIDENT logits; 2 documented HOST-FALLBACK islands (the KDA per-k-channel gated-delta recurrence + its exp/softplus decay gate — `vt::GdnDecode` carries only a per-HEAD scalar decay; the NoPE-MLA softmax core — the paged `mla::ForwardMlaAttentionBlock` device path is born-on-runner) are the W7-speed residuals. CPU-gated vs the W2 host reference (the CPU backend runs the SAME `vt::` dispatch): `test_kimi_linear_forward` **12/12·614** (per-op KDA/NoPE-MLA/MoE/dense device==ref within f32-accumulation tolerance; the whole `ForwardDeviceCompute` == ref logits + greedy-token-identical + device-resident). Runner opt-in via `VT_KIMI_DEVICE_COMPUTE=1` (default OFF keeps the CPU-verified W6 host-ref compose). GPU numerics (bf16 activations, GDN Triton-AOT cubins, paged het-KV, grouped-MoE slabs) + the e2e SACRED golden stay a NAMED pending (box down) — row STAYS 🚧. ON TOP OF **W6 DEVICE forward SEAM** (`CLAIM-KIMI-LINEAR-W6`): the born-on-the-runner `ForwardDevice` (the DEFAULT `gather_logits` runner path) no longer refuses — it composes the `[rows,vocab]` logits via the CPU reference and hands them back DEVICE-RESIDENT (a pooled `DBuf`, wrapped like deepseek_v2 `WrapDeviceLogits`; `on_device()==true` on CPU+CUDA) so the on-GPU sampler consumes them with NO host download. Kimi-Linear now ROUTES device-resident (`check-runner-routing-consistency` reclassifies it, refuse-skipped stubs 2→1, NO allowlist; `check-fusion-consistency` green); `test_kimi_linear_forward` **7/7·300** (adds the `ForwardDevice`==host-ref device-resident gate). The DBuf-resident device COMPUTE (KDA via the GDN family, NoPE-MLA via `mla::ForwardMlaAttentionBlock`, DeepSeek-V2 grouped-MoE over the paged het-KV; full plan in `kimi_linear.cpp`) is the GPU-verify-pending W7 residual. ON TOP OF **W2-W6 CPU REFERENCE forward** (`CLAIM-KIMI-LINEAR-W2`): the real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives (KDA layer via `vllm::kimi_kda` refs + the gated-delta recurrence; NoPE-MLA materialized-MHA ref; sigmoid `noaux_tc` MoE + shared expert; dense SwiGLU); loader now materializes host float weights; `test_kimi_linear_forward` 6/6·246 (per-op gates + finite whole forward + greedy decode). ON TOP OF **W1 scaffolding** (registry + `ParseKimiLinearParams` 20 KDA + 7 NoPE-MLA + index-verified name-map + het-KV spec). e2e-gateable (FITS one GB10, 0.77× pool). RESIDUAL = the DEVICE born-on-runner forward (KDA kernel/absorbed-MLA/grouped-MoE slabs) + the W0/W7 e2e SACRED golden. Row → `ACTIVE` (device SEAM wired; the DBuf device compute + e2e SACRED golden pending) | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` | +| 🚧 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B | **DEVICE-KDA GB10 122/128 + 4.24 tok/s (§15, #104); device NoPE-MLA lever MEASURED-NEGATIVE (2026-08-07, §16, `row/KIMI-STRICT-CLOSE` #107):** the per-channel-decay device recurrence `vt::KdaGatedDeltaRule` moves 106→**122/128** (p0-p6 16/16; sole p7 pos-6 comma near-tie) AND **1.35→4.24 tok/s (3.1×)** — vLLM's ACTUAL f32-on-bf16 arithmetic, beats §14's host-precision 120. The §15 residual (d) was attempted in device-COMPUTE form: `VT_KIMI_DEVICE_MLA` routes the 7 NoPE-MLA layers' softmax core through `vt::Attention` (pad-V: value zero-padded qk_nope+qk_rope=192 vs v=128, `out[:,:,:v]` byte-exact). CPU RED-first gate GREEN (`test_kimi_linear_forward` **14/14·825**, pad-V==f64 ref rtol 3e-3; perturbation fails 108). Full 48.9B GB10 gate (single-load, flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+device-MLA REGRESSES to 109/128 AND 3.89 tok/s** — `vt::Attention`'s f32 online-softmax is the right math but a DIFFERENT reduction order than vLLM's FA2, so it coin-flips near-ties (breaks p3 16→3 into §14's `163586×` repeat) and the per-(t,h) build slows the O(n²) recompute. `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (§14 `ISLAND_F32ACC` precedent). MLA dims VERIFIED from the real config (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA). Both device knobs default OFF (122 ≠ STRICT, K=3-deterministic golden). STRICT residual, sharpened: needs vLLM's ACTUAL kernels — (c) chunk_kda prefill family (Triton-AOT regen for sm_121a) + (d) paged FA2 `mla::ForwardMlaAttentionBlock` (NOT the vt::Attention approximation) + (e) paged-incremental decode (needs a decode/paged-attn op, query_len≠key_len; kills the O(n²)) — each a substantial multi-kernel brick (§16). Row STAYS 🚧. **FULL-MODEL GB10 e2e RUNS — NEAR-TIE 106/128 (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the bf16-resident path CLEARS the f32-loader block — the full 48.9B model now runs e2e on one GB10. dgx CUDA build (`-Werror` clean, 14 GDN AOT symbols nm-linked, `test_kimi_linear_forward` 13/13·656 in the CUDA binary); `kimi-linear-gen --gpu` greedy-decodes the §12 8-prompt battery x16 vs `greedy_ids.npy`. MEMORY: load 117.6s, host RSS PEAK **1.7 GiB** (stage-then-ReleaseHost), device peak 98.5 GiB, min-avail **21.6 GiB** (above the 15 GiB floor, matches the ~25 GiB pool-math headroom), NO OOM/reboot. TOKEN gate **NEAR-TIE 106/128 (82.8%)** — prompts 0,1,3,4,5,6 are 16/16 token-exact, p2/p7 diverge at punctuation/word near-ties; 96 consecutive exact tokens across 6 prompts prove the WIRING (a wiring bug can't). Root cause (honest): the f32 residual stream + host-f64 islands are MORE precise than vLLM's bf16 device kernels, so they flip the argmax where vLLM's deterministic bf16 top-1 has a small margin. STRICT path = the named W7-speed residuals (device GDN/MLA islands -> bf16 stream matching vLLM's rounding). 1.59 tok/s (recompute+island rate). `VT_KIMI_DEVICE_COMPUTE` STAYS OFF (parity-enablers: near-tie != token-exact). Row STAYS 🚧. **bf16-RESIDENT loader/forward IMPLEMENTED + CPU-gated (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the §13 design is coded — `LoadKimiLinearResidentBf16Weights`/`StageKimiResidentBf16`/`BuildKimiResidentFromHost` (`kimi_linear_weights.cpp`; `LoadBf16Direct` -> `OwnedTensor`, per-tensor stage-to-`d_dev` + `ReleaseHost`, tiny vectors host f32), `KimiLinearResidentWeights` (`kimi_linear.h`), bf16 device forward `DeviceForwardBodyBf16` + `Gemm Bf16` cast-act at ~20 GEMM sites with the two host-fallback islands EXTRACTED+shared (`kimi_linear_device.cpp`), `ForwardDevice` resident-path dispatch (`kimi_linear.cpp`), and the `kimi-linear-gen` e2e harness. CPU **13/13·656** (12/12·614 f32 path UNTOUCHED + NEW tiny-config bf16-vs-f32 gate). PENDING: dgx CUDA build + full-model GB10 e2e vs the STRICT golden. Row STAYS 🚧. **bf16-RESIDENT brick POOL-MATH+DESIGN (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** pool math CLOSES (91.5 GiB bf16 device-resident + ~2.4 GiB act/norms/ctx ≈ 94 GiB, ~25 GiB headroom); design grounded §13 (Laguna `GemmBf16` cast-act + `OwnedTensor::d_dev`, `LoadBf16Direct`, f32 `MaterializeHost` kept for the unit gate). Impl (loader/forward rewrite + gate + e2e) pending. Row STAYS 🚧. **§8 GOLDEN CAPTURED — STRICT (2026-08-06, `row/MODEL-KIMI-LINEAR-E2E`):** the §8 SACRED oracle golden is captured on GB10 (0.25.0-stage, util 0.82, moe=triton, min 15 GiB avail, NO reboot), **8/8 prompts DETERMINISTIC over K=3 → STRICT gate**, committed at `tests/parity/goldens/kimi_linear_greedy/`. Full our-engine e2e BLOCKED on OUR f32 loader (materializes ~183 GiB > 119 pool), the bf16-residency residual; row STAYS 🚧. **W7 GPU-VERIFY (2026-08-06, branch `row/MODEL-KIMI-LINEAR-GPU`):** the device compute runs **12/12·614 GREEN on GB10 sm_121a CUDA build**, BOTH arms (`VT_KIMI_DEVICE_COMPUTE=1` + host-ref); prod stack (CUTLASS-NVFP4 GEMM + FA2 ENABLED + Triton-AOT GDN, 14 cubins nm-verified); f32 device==W2 ref, no divergence, no DeepSeek-class trap. Oracle gateability re-confirmed (0.25.0-stage registers `KimiLinearForCausalLM`). e2e §8 SACRED golden STILL disk-blocked (91.5 GiB checkpoint absent, dgx root 100% full, 34G free). Row STAYS 🚧. **W7 DBuf-resident device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`): the real device compute (`ForwardDeviceCompute`, `kimi_linear_device.cpp`) composes the whole 27-layer KDA/NoPE-MLA + 256-expert-MoE hybrid over pooled f32 `DBuf`s through the SHARED `vt::` ops (embed/`FusedChain` add+RMSNorm/`MatmulBT` projections/`CausalConv1dFwd` convs/`L2Norm`/`RmsNormGated`/`MoeRouterTopK` sigmoid-`noaux_tc`/`MoeSiluMul`/`MoeCombine`/lm_head), returning DEVICE-RESIDENT logits; 2 documented HOST-FALLBACK islands (the KDA per-k-channel gated-delta recurrence + its exp/softplus decay gate — `vt::GdnDecode` carries only a per-HEAD scalar decay; the NoPE-MLA softmax core — the paged `mla::ForwardMlaAttentionBlock` device path is born-on-runner) are the W7-speed residuals. CPU-gated vs the W2 host reference (the CPU backend runs the SAME `vt::` dispatch): `test_kimi_linear_forward` **12/12·614** (per-op KDA/NoPE-MLA/MoE/dense device==ref within f32-accumulation tolerance; the whole `ForwardDeviceCompute` == ref logits + greedy-token-identical + device-resident). Runner opt-in via `VT_KIMI_DEVICE_COMPUTE=1` (default OFF keeps the CPU-verified W6 host-ref compose). GPU numerics (bf16 activations, GDN Triton-AOT cubins, paged het-KV, grouped-MoE slabs) + the e2e SACRED golden stay a NAMED pending (box down) — row STAYS 🚧. ON TOP OF **W6 DEVICE forward SEAM** (`CLAIM-KIMI-LINEAR-W6`): the born-on-the-runner `ForwardDevice` (the DEFAULT `gather_logits` runner path) no longer refuses — it composes the `[rows,vocab]` logits via the CPU reference and hands them back DEVICE-RESIDENT (a pooled `DBuf`, wrapped like deepseek_v2 `WrapDeviceLogits`; `on_device()==true` on CPU+CUDA) so the on-GPU sampler consumes them with NO host download. Kimi-Linear now ROUTES device-resident (`check-runner-routing-consistency` reclassifies it, refuse-skipped stubs 2→1, NO allowlist; `check-fusion-consistency` green); `test_kimi_linear_forward` **7/7·300** (adds the `ForwardDevice`==host-ref device-resident gate). The DBuf-resident device COMPUTE (KDA via the GDN family, NoPE-MLA via `mla::ForwardMlaAttentionBlock`, DeepSeek-V2 grouped-MoE over the paged het-KV; full plan in `kimi_linear.cpp`) is the GPU-verify-pending W7 residual. ON TOP OF **W2-W6 CPU REFERENCE forward** (`CLAIM-KIMI-LINEAR-W2`): the real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives (KDA layer via `vllm::kimi_kda` refs + the gated-delta recurrence; NoPE-MLA materialized-MHA ref; sigmoid `noaux_tc` MoE + shared expert; dense SwiGLU); loader now materializes host float weights; `test_kimi_linear_forward` 6/6·246 (per-op gates + finite whole forward + greedy decode). ON TOP OF **W1 scaffolding** (registry + `ParseKimiLinearParams` 20 KDA + 7 NoPE-MLA + index-verified name-map + het-KV spec). e2e-gateable (FITS one GB10, 0.77× pool). RESIDUAL = the DEVICE born-on-runner forward (KDA kernel/absorbed-MLA/grouped-MoE slabs) + the W0/W7 e2e SACRED golden. Row → `ACTIVE` (device SEAM wired; the DBuf device compute + e2e SACRED golden pending) | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` | | 📋 | `KimiK3ForConditionalGeneration` | Kimi K3 (2.8T MoE + MoonViT-V2, DERIVE-AND-SHIP) | **W2/W5 CPU scaffolding landed** (registry stub + nested text/vision/quant config descent + text-backbone structural name-map + REFUSE-by-name forward + MXFP4-refuse loader; clean CPU build, scaffold gate 6/6). text backbone IS `KimiLinearForCausalLM` (KDA+MLA+MoE hybrid, HEAVY reuse); **does NOT fit GB10 (~1.56 TB MXFP4, ~12×)** and NOT in the pinned oracle ⇒ no on-box golden — DERIVED, proxy-gated on Kimi-Linear-48B; forward + MXFP4 + KDA delta + MoonViT-V2 not implemented (NOT-YET-BUILDABLE) | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` | | 🚧 | `MiniMaxH3DiTModel` | MiniMax-H3 (33.1B omni-modal video+audio DiT, DERIVE-AND-SHIP) | **W1/W2 landed**: packed layout (fl2va + ref2va, fp64 position grid BIT-EXACT), latent packing, euler-ancestral eta0 scheduler, and the full DiT forward all parity-gated against the UPSTREAM vLLM-Omni modules executed at reduced dimensions (**max abs diff 1.6e-7**, 10/10 cases / 2539 assertions). NOT autoregressive (no KV cache, no sampler, no logits) and **e2e HW-BLOCKED** (~354 GB checkpoint, ~133 GB/rank on 4x B300 vs 119 GiB unified); bf16 production stream + request planning + the ComfyUI-GGUF arm also landed (535 REAL tensors resolve onto our contract, geometry from shapes alone). **HW verdict CORRECTED: quantized arms FIT (~41 GB in 119 GiB)**, so e2e + speed are reachable; encoder/VAEs/audio VAE DONE (4.2e-9 vs the checkpoint's remote code); NVFP4 layout GATED as identical to ours (speed path is loader wiring); BOTH VAE DECODERS done (audio 4.2e-9, video ViT3D 8.9e-8); video tiling + 3D-CNN encoder (conditioning only) pending; encoder TEXT tower done (1.2e-7); **serving `/v1/videos` DONE and the DEVICE-RESIDENT forward (W2b, f32) LANDED + GPU-VERIFIED on Thor sm_110 at video 1.49e-7 / audio 8.94e-8**; bf16 stream + fusion folds + the FP4 path (needs sm_121a) + a real-checkpoint run pending. **2026-08-05: the AUDIO-VAE ENCODER is ported** (DAC analysis stack + `pre_block` AttnProjection + `mean_proj`, gated stage by stage vs the checkpoint's own remote code at 2.98e-8 / 1.64e-7 / 1.86e-8) with its own checkpoint loader gated on the real 1087-tensor manifest — so **ref2va AUDIO and VIDEO+AUDIO references are now WIRED** (audio rows move by 0.51 / 0.71; a different waveform still moves them by 7.1e-4). Both VAEs are now complete in both directions. **bf16 13-SHARD RELEASE INDEXES 2026-08-07 (`row/H3-BF16-SHARDED-DIT`)**: `MiniMaxH3ShardedCheckpoint` resolves the ORIGINAL 66.3 GB release through its own `model.safetensors.index.json` (a tensor named in the index but missing from its shard throws BY NAME), `EnumerateMiniMaxH3ShardedTensors` feeds the shared shapes-only geometry parser, and `LoadMiniMaxH3DitFromShards` is the host-f32 reference loader. Gated CPU-only at 72/72/54497 (post-rebase): every tensor resolves to the shard the index named AND to the bytes written there, the derived geometry equals the single-file path field for field, and a SPARSE 13-shard release with the REAL 535 tensors at REAL shapes (66.3 GB declared, 144 KB on disk) derives the SHIPPED geometry (50/5376/56/128/14336/24/32/1x2x2/5120). The DEVICE streamer is the stacked follow-up `row/H3-BF16-SHARDED-STREAM`. This UNBLOCKS the quantization-quality question; no bf16-vs-quant render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the device DiT forward now routes the NVFP4 projections through the shared Marlin W4A16 dispatcher (fp4 kept packed; no new quant code), fp4-vs-bf16 wiring gate GREEN (62/62·30039). **W-FP4a GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** on sm_121a the Marlin W4A16 path RAN for all 11 projections (`dense_gemms==11` default — VT_MARLIN_DENSE is default-ON → vLLM's own DENSE Marlin GEMM, not the grouped route; `marlin_gemms==11` under VT_MARLIN_DENSE=0; `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT (max\|diff\|=0), and the fp4 arm is a MEMORY win not a diffusion-forward speed win (per-forward bf16/fp4 3.47× @seq64 → 0.79–0.83× @seq4224–7040; ~16 vs ~66 GB device). Real-checkpoint fp4-resident t2va e2e RUNS (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render bug (device VAE/denoise). vLLM-Omni has no quantized H3 arm (BF16-only) so any comparison is HW/loader-forced-indirect — spec §8 | `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit` | | ✅ | `LagunaForCausalLM` | Poolside Laguna-S-2.1 (118B/8B MoE) | **LONG-CTX DECODE LEVERS LANDED + MEASURED (2026-08-03, `CLAIM-LAGUNA-LONGCTX-LEVERS`): window-bounded SWA reads (`VT_LAGUNA_SWA_WINDOW`, default-ON, BYTE-EXACT) bound the four `DecodeAttnGqa*` kernels' read to the ~512 sliding window (vLLM `laguna.py:412`) — GB10 A/B token-IDENTICAL `=1` vs `=0` at 520-token context (truncation active), MEASURED −0.30 ms/step at ~2k (~0 at ≤512, grows linearly). bf16 paged KV (`VT_LAGUNA_KV_BF16`, default-OFF opt-in) a distributional near-tie left UNRATIFIED. See BENCHMARKS `CLAIM-LAGUNA-LONGCTX-LEVERS`.** — **NVFP4 W4A4 ARM RAN on GB10 (N4, 2026-08-01, `CLAIM-LAGUNA-NVFP4-N4`): the additive safetensors NVFP4 arm (N1a/N1b/N2/N3 — `Nvfp4Weight` expert fields + `LoadLagunaForCausalLMWeights` + `LqGemmNvfp4Fp4` per-expert TRUE-W4A4 + `LagunaFfnBlock` `fp4` branch + `laguna_gen` dir-autodetect; CPU-gated `test_laguna_nvfp4_loader` 3/3·61, GGUF path byte-identical) generates COHERENTLY on the real 67 GiB `poolside/Laguna-S-2.1-NVFP4`. vs the vLLM MARLIN golden (vLLM's exact prompt ids injected): FIRST 2 TOKENS MATCH exactly, then near-tie divergence (our TRUE-W4A4 fp4-activations vs the MARLIN golden's W4A16 bf16-activations — different precision, EXPECTED; shares golden vocab). SPEED (N5, trace-driven, 2026-08-01): 0.16 → ~4.5 tok/s (~28× THIS SESSION), now ~4× from vLLM 18.8. **Lever #2** (nsys found the bf16 tower running host `MatmulNK` on the CUDA queue): route it to the GPU (`LqGemm` bf16 → `CastBf16` + `MatmulBT`, weight stays bf16) → 6.34 → 0.39 s/tok (16×). **Lever #1** (nsys found the emulation expert GEMM at 92%, GPU 87% busy): the engine's native sm120a fp4 tensor-core MMA (`MatmulNvfp4Fp4Native`) reads the SAME linear scales — it was gated OFF behind `VT_NVFP4_FP4_NATIVE`; default it ON in the driver → 0.39 → ~0.20-0.24 s/tok (~2×). Both coherent + near-tie (byte-identical ids to emulation; first token matches golden). Two GB10 memory fixes landed to run (shard-release + context-before-load). OPEN #234 (remaining ~4×): grouped W4A4 MoE (top_k×3 launches → 3), `ResidentNvfp4`, decode CUDA-graph + on-GPU sampling (the host-orchestration tail). Spec `.agents/specs/laguna-nvfp4-arm-2026-07-31.md` §N4/§N5. The GGUF-Q4_K track (below) is the separate keep-quant vehicle.** Prior **FASTER DECODE (W9, 2026-07-31, `CLAIM-LAGUNA-W9-GROUPED`): the 30 un-grouped per-expert keep-quant GEMV launches/step (top_k × {gate,up,down} `LqGemmRowSlice`) fold onto the SHARED `vt::MatmulBTQuantGrouped` op — per token, Pk experts' gate/up/down each collapse to ONE grouped launch over the already-stacked `[E*N,H]` tower (no loader change). Same-binary A/B on real UD-Q4_K_XL (GB10, `--gpu`, drop_caches cold, 24 tok): grouped (`VT_LAGUNA_GROUPED_MOE=1`, default) == per-expert (`=0`) BYTE-IDENTICAL (md5 `754728c6`, both == W6 golden) + decode 0.18 → 0.13 s/tok (1.38×). Routes through the shared vt op (fold policy). Cumulative with W8: decode 0.66 → 0.13 s/tok (5.1×; 1.5 → 7.7 tok/s; 18× → 3.6× vs llama.cpp 27.8). Next lever: device-resident decode (#1). See spec §W9.** Prior **FASTER DECODE (W8, 2026-07-31, `CLAIM-LAGUNA-W8-EMBED`): `LagunaEmbed` no longer converts the whole 1.23 GB embed table to f32 every token (it gathered T rows out of the whole [Vsz,H] table via `ReadF32` — ~311M host element-converts/token, the DOMINANT decode cost the W7 profile under-filed as "#5"); now gathers only the T needed rows directly (BIT-IDENTICAL — same per-element conversion, same rows). GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10, `--gpu`, W6 cached, drop_caches cold, 24 tok): TOKEN-IDENTICAL to the W5/W6 golden (`22345 83 350 785 …`, coherent " Paris.") + decode 0.66 → 0.17 s/tok = 3.9× (1.5 → 5.9 tok/s; 18× → 4.7× vs llama.cpp 27.8). See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md` §W8. Next: grouped-expert GEMM (=A3) then device-resident decode.** Prior **DECODE-SPEED ATTRIBUTED (W7 profile-only, 2026-07-31, `CLAIM-LAGUNA-W7-SPEED`): `nsys` of the W6 decode (real UD-Q4_K_XL GGUF, GB10) attributes the 0.66 s/tok (~1.5 tok/s vs llama.cpp 27.8 on identical bytes, ~15-18x) to HOST-ORCHESTRATION, not kernel compute — GPU active only 32.7% of the step, 67.3% host/idle; 22,115 `cudaStreamSynchronize` (~2,764/step, zero GPU overlap) from the ~1,795 per-GEMM `DrainQueue` in `LagunaForwardGgufCached` + scalar host glue; 39.4% of GPU time is `QuantizeQ8K` activation-quant (per-GEMM), weight GEMVs un-grouped at ~22% of the 240 GB/s peak (llama.cpp ~76%); no H2D/D2H (unified memory). Ranked levers (all in-tree from ds4): device-resident decode 1.5->~5-7 tok/s, grouped-expert GEMM (`MatmulBTQuantGrouped`) +1.5-2x + dedupes the activation-quant, decode CUDA-graph, tuned MMVQ; + free host cleanups (`LagunaEmbed` copies the whole 1.23 GB embed table/token, per-token RoPE-cache rebuild). Honest reachable ~13-20 tok/s, 27.8 a stretch. NO code changed. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`. Prior RUNNABLE + FAST DECODE (W6, 2026-07-31): a per-layer K/V cache + single-token incremental decode replaces W5's O(n²) STATELESS recompute — TOKEN-IDENTICAL (byte-equal ids, md5 `754728c6…` match, == the W5 golden) and 5.05× faster per token: decode 3.33 → 0.66 s/tok on the real UD-Q4_K_XL GGUF (GB10, `--gpu`, keep-quant), same " Paris.…" text. `LagunaKvCache` (mirrors `DeepseekV4KvCache`, MLA-latent → GQA multi-head K/V; caches post-QK-RMSNorm/post-RoPE K + raw V at f32 — bit-exact since RoPE/QK-norm are position-only and attention is causal), MIXED attention per-layer: 12 GLOBAL layers grow unbounded + 36 SLIDING-WINDOW-512 layers EVICT rows beyond the 512 window (gemma2/3 `is_sliding`); `LagunaForwardGgufCached` + shared `LagunaAttention`/`LagunaFfnBlock` helpers used by BOTH forwards (identical float ops; recompute ids unchanged after refactor), `examples/laguna_gen --stateless` A/B flag. No cache bug (bit-exact first run). Next speed = grouped-expert GEMM + device-resident decode (both in-tree from ds4). See `.agents/specs/laguna-s21-w6-2026-07-31.md`. Prior RUNNABLE (W5, 2026-07-31): our engine greedy-generates COHERENT text on the REAL 3-shard UD-Q4_K_XL GGUF (GB10 keep-quant) — "The capital of France is" → " Paris. …", first token "Paris." matches the llama.cpp-Poolside reference. Multi-shard GGUF reader + keep-quant tower (`LoadLagunaFromGgufShards`) + `LagunaForwardGguf` (ds4 keep-quant Gemm/GemmRowSlice) + `examples/laguna_gen`; load 20.6s, peak 71 GiB, 3.27 s/tok stateless recompute (speed=W6).** Prior W3: **W3 REAL forward + 3 new ops landed** (`laguna_ops.cpp`: per-head softplus attn out-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders; `LagunaModel::Forward` now a REAL runnable host-reference composition — variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head — replacing the W1/W2 `VT_CHECK(false)` stub; CPU `-Werror` full-library build clean; `test_laguna_scaffold` **8/8·166** incl. softplus math, router selection+tie-break RED-first, dual-RoPE cos/sin bit-match, variable-Q-head shapes, forward composition on synthetic weights; `test_model_registry` 24/24). W1 oracle DECISION: vLLM native `laguna.py` in pin ⇒ config constructs; dual-oracle = vLLM-NVFP4/-FP8 (fits GB10, BF16 235 GiB does NOT) + llama.cpp-Q4_K token-exact. DEFERRED to W4 (needs 73 GB checkpoint): GGUF keep-quant tower materialization + device/paged production forward + strict dual-oracle greedy gate. ~85-90% reuse (ds4-MoE + gemma-sliding + olmo3-dual-rope + landed Q4_K keep-quant); NEW = the 3 landed host ops + name-map + variable-Q-head device runner. **W4 (2026-07-31, `CLAIM-LAGUNA-W4`, in progress):** the UD-Q4_K_XL GGUF (73.4 GiB, 3 shards) FETCHED to dgx + its metadata/tensor-map READ AUTHORITATIVELY (814 tensors, arch `laguna`, `expert_gating_func=2` sigmoid, `leading_dense_block_count=1`, `expert_weights_scale=2.5`). Three CPU-verified FIDELITY corrections the W1-W3 scaffold got wrong, each grounded in the real GGUF + llama.cpp: (1) **per-head QK-RMSNorm** (`attn_q_norm`/`attn_k_norm` F32[128]) added to params+forward — the scope MISSED it (surfaces only in the tensor map); (2) **dual-RoPE mscale** now uses llama.cpp's `yarn_attn_factor·(1+0.1·ln(factor))` off the GGUF-authoritative `factor=32`/`yarn_attn_factor=1.0` (256K-ctx build, NOT HF's factor-128/1.4852 1M-ctx scalar) — resolves the numerics-delicate residual; (3) **separate** `ffn_gate_exps`/`ffn_up_exps` (Q4_K) + `ffn_down_exps` (Q5_K) + Q8_0 shared/attn (the scaffold assumed merged gate_up). GGUF keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + keep-quant `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + the real-model greedy run vs the llama.cpp-laguna same-quant oracle remain the W5 close (73 GB single-GB10, host-orchestrated) | `MODEL-TEXT-laguna-laguna-for-causal-lm` | diff --git a/.agents/specs/kimi-linear.md b/.agents/specs/kimi-linear.md index acdbb5c04..741d5a1fa 100644 --- a/.agents/specs/kimi-linear.md +++ b/.agents/specs/kimi-linear.md @@ -913,6 +913,65 @@ aot.sh`), or a native `chunk_kda` port. Row STAYS `ACTIVE`. --- +## 16. DEVICE NoPE-MLA attention lever MEASURED-NEGATIVE; STRICT still owed the ACTUAL FA2/chunk_kda kernels (2026-08-07, `row/KIMI-STRICT-CLOSE`, #107) + +The §15 residual (d) — "the 7 NoPE-MLA layers still use a host f64 softmax island … closing p7 +needs paged `mla::ForwardMlaAttentionBlock`" — was attempted in its device-COMPUTE form (the §15 +device-KDA pattern applied to the MLA half) and MEASURED-NEGATIVE on GB10. The one-brick STRICT-close +did NOT land; the honest verdict re-confirms §14's razor. + +**Implementation (`kimi_linear_device.cpp`, additive, default OFF).** New knob `VT_KIMI_DEVICE_MLA` + +helper `MlaAttnCoreDevice`: the NoPE causal softmax over per-head `[k_nope|k_pe(shared)]`/`v` runs +through the shared device op `vt::Attention` (f32 online max-subtracted softmax — vLLM's FA2 +accumulation regime) instead of the f64 host `MlaSoftmaxIsland`. `vt::Attention` carries a single +head-dim for q/k/v while MLA is asymmetric (`qk = qk_nope+qk_rope = 192`, `v = 128`), so the value is +PADDED to `qk` with zeros — the weighted sum over the zero tail is 0, so `out[:, :, :v]` is byte-exact +to the unpadded math (softmax weights depend only on `q·k`). q views `dq` directly as `[T,nah,192]`; +key is built per `(t,h)` as `[k_nope | k_pe(broadcast)]`. Wired into both the f32 and bf16 +`MlaSoftmaxIsland` paths. MLA dims VERIFIED from the real 48.9B `config.json` (not the K3 numbers): +`nah=32, qk_nope=128, qk_rope=64, v_head_dim=128, kv_lora=512, q_lora=None`; 7 full-attn/MLA layers +(`full_attn_layers=[4,8,12,16,20,24,27]`), 20 KDA. + +**Unit gate (RED-first, CPU) GREEN.** `test_kimi_linear_forward` **14/14·825** (was 13/13·656) — +NEW case (g2) `KimiMlaAttnCoreDevice` (pad-V + `vt::Attention`) == a from-first-principles f64 +causal-softmax reference at the Kimi MLA geometry (rtol 3e-3). RED-first verified: a perturbed scale +fails 108 assertions. Env-gated whole-forward runs green (`VT_KIMI_DEVICE_MLA=1` alone and with +`VT_KIMI_DEVICE_KDA=1`, 14/14·825). Same on the GB10 CUDA binary (210 GDN + 23 KDA syms linked). + +**Full 48.9B GB10 gate — MEASURED NEGATIVE (single-load per config, `flock $HOME/gpu.lock`, min-avail +21 GiB, no reboot; the golden is the §12 STRICT `greedy_ids.npy`).** + +| Config | env (all `VT_KIMI_DEVICE_COMPUTE=1`) | /128 | tok/s | verdict | +|---|---|---|---|---| +| control (device-KDA) | `DEVICE_KDA=1` | 122 | 4.24 | reproduces §15 EXACTLY (p0-p6 16/16, p7 10/16) | +| **+ device-MLA** | `DEVICE_KDA=1 DEVICE_MLA=1` | **109** | **3.89** | **REGRESSION both axes** | + +**Why negative (the §14 razor, re-proven).** device-KDA WORKS (106→122) because the recurrence is the +SAME algorithm as vLLM's decode kernel, just f32-on-bf16 — it matches. But vLLM's MLA prefill uses +**FA2** (a specific flash tiling/reduction order); `vt::Attention`'s plain f32 online-softmax is the +right MATH but a DIFFERENT reduction ORDER, so — exactly like §14's host-precision-matching plateau — +it COIN-FLIPS near-ties: it BREAKS p3 16/16→3/16 (into the same `163586×` degenerate repeat the §14 +bf16 knobs caused) while p7 stays diverged at 10/16. And it is SLOWER (4.24→3.89): the per-`(t,h)` +key/value build copies + the 192-dim pad-V waste add overhead to the O(n²) recompute path. An +approximation of vLLM's kernel is not enough — only the ACTUAL kernel matches. + +**Verdict + default.** `VT_KIMI_DEVICE_MLA` STAYS **OFF**, kept as a documented-MEASURED-NEGATIVE A/B +knob (parity-lever precedent: §14's `ISLAND_F32ACC`/output-bf16). device-KDA (122/128, 4.24 tok/s) +remains the best config, itself default OFF (122 ≠ STRICT). Row STAYS `ACTIVE`. + +**STRICT residual, sharpened by this measurement.** p7 (and now the coin-flip class generally) needs +vLLM's ACTUAL kernels, NOT a device approximation: (c) the **chunk_kda** prefill kernel family +(`chunk_kda_scaled_dot_kkt` + `recompute_w_u` + `chunk_gla_fwd_o_gk` + `fused_kda_gate_chunk_cumsum`, +FLA `ops/kda.py`) — mirror-first via a Triton-AOT regen for sm_121a (`scripts/regen-triton-aot.sh` + +new `triton_kernels/*.py`), the spec's named prime suspect; (d) the paged +`mla::ForwardMlaAttentionBlock` (FA2) for the 7 NoPE-MLA layers — NOT the `vt::Attention` approximation +tried here; (e) **paged-incremental decode** — coupled with (d) because it needs a decode/paged +attention op (`query_len ≠ key_len`), which `vt::Attention` cannot express; it kills the O(n²) +full-recompute (the current 4.24 tok/s is the recompute rate). Each is a substantial multi-kernel +brick, not a one-shot; recorded as the named follow-on. + +--- + ## Structured contract (machine-readable — mirrors deepseek-v4-flash.md) ## Scope diff --git a/.agents/state.md b/.agents/state.md index d77540c24..5dd37e5da 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -41023,3 +41023,36 @@ is required") is now IMPLEMENTED as the additive device op `vt::KdaGatedDeltaRul (`chunk_kda`; regen a Triton-AOT cubin for sm_121a via `scripts/regen-triton-aot.sh`, or native port) + paged `mla::ForwardMlaAttentionBlock` (7 NoPE-MLA layers) + paged-incremental decode. Row `ACTIVE`. Box left clean (artifacts removed, memory restored, worker parked, no reboot). + +## 2026-08-07T12:35 — Kimi-Linear: device NoPE-MLA attention lever `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE (122→109/128, slower); STRICT owes the ACTUAL FA2/chunk_kda kernels (row/KIMI-STRICT-CLOSE, #107) +- The §15 residual (d) — the 7 NoPE-MLA layers still run a host f64 softmax island; closing p7 needs + paged `mla::ForwardMlaAttentionBlock` — was attempted in its DEVICE-COMPUTE form (the §15 device-KDA + pattern applied to the MLA half) and MEASURED NEGATIVE on GB10. Additive, default-OFF knob + `VT_KIMI_DEVICE_MLA` + `MlaAttnCoreDevice` (`kimi_linear_device.cpp`): the NoPE causal softmax runs + through the shared device op `vt::Attention` (f32 online max-subtracted softmax = vLLM's FA2 regime) + instead of the f64 host `MlaSoftmaxIsland`. `vt::Attention` carries one head-dim; MLA is asymmetric + (qk=qk_nope+qk_rope=192, v=128), so value is zero-PADDED to qk (weighted-sum over the 0 tail = 0, so + `out[:,:,:v]` is byte-exact). MLA dims VERIFIED from the real 48.9B config (nah=32, qk_nope=128, + qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA) — not the K3 numbers. +- Unit gate (RED-first, CPU) GREEN: `test_kimi_linear_forward` 14/14·825 (was 13/13·656) — NEW case + (g2) `KimiMlaAttnCoreDevice` (pad-V + `vt::Attention`) == a from-first-principles f64 causal-softmax + reference (rtol 3e-3); a perturbed scale fails 108 assertions. Env-gated whole-forward green + (`VT_KIMI_DEVICE_MLA=1` alone and with `DEVICE_KDA=1`). Same on the GB10 CUDA binary (210 GDN + 23 + KDA syms linked, CUTLASS-NVFP4 + FA2 + Triton-AOT). +- Full 48.9B GB10 gate vs the §12 STRICT golden (single-load per config, flock $HOME/gpu.lock, + drop_caches, min-avail 21 GiB, no reboot): control device-KDA reproduces 122/128 + 4.24 tok/s EXACTLY; + **+device-MLA REGRESSES to 109/128 AND 3.89 tok/s**. WHY (the §14 razor, re-proven): device-KDA works + because its recurrence is the SAME algorithm as vLLM's decode kernel (f32-on-bf16); but vLLM's MLA + prefill uses FA2 (a specific reduction ORDER), and `vt::Attention`'s plain online-softmax is the right + MATH in a DIFFERENT order → it COIN-FLIPS near-ties (breaks p3 16→3 into the §14 `163586×` repeat) + while p7 stays diverged, and the per-(t,h) build + 192-dim pad-V waste slow the O(n²) recompute. An + approximation of vLLM's kernel is NOT enough — only the actual kernel matches. +- VERDICT: `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (§14 + `ISLAND_F32ACC` precedent). device-KDA (122/128, 4.24) remains the best, itself default OFF. The + one-brick STRICT-close did NOT land. STRICT residual SHARPENED — needs vLLM's ACTUAL kernels: + (c) chunk_kda prefill family (FLA `ops/kda.py`; Triton-AOT regen for sm_121a via + `scripts/regen-triton-aot.sh` + new `triton_kernels/*.py`), the prime suspect; (d) paged FA2 + `mla::ForwardMlaAttentionBlock` (NOT the vt::Attention approximation); (e) paged-incremental decode + (needs a decode/paged-attn op, query_len≠key_len, which `vt::Attention` cannot express; kills O(n²)). + Each a substantial multi-kernel brick. Row STAYS ACTIVE. Box left clean (build tree/markers removed, + memory restored to 112 GiB avail, worker restarted --restart=always, golden preserved, no reboot). diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 06bcbe3f2..358708837 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -309,7 +309,7 @@ built on it rather than keeping the flattering one. | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids | | `/v1/videos` OpenAI (Sora) shape | **No number owed**: a CPU serving-surface change (request aliases, the MP4 content route, and reference conditioning wiring), unit-gated only, no kernel or generation path touched | Video generation speed stays the MiniMax-H3 FP4 row below | | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | -| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | e2e RUNS (bf16-resident §13); per-channel-decay KDA device kernel `vt::KdaGatedDeltaRule` LANDED+GB10: **106→122/128 + 4.24 tok/s (3.1×)**, beats §14's 120 on both axes, NOT STRICT; default OFF | device-KDA runs vLLM's actual GPU recurrence; bf16 knobs now hurt (122→90). p7 near-tie; residual = KDA chunked-prefill + paged FA2 MLA + incremental. §15 | +| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | e2e RUNS (bf16-resident §13); KDA device op `vt::KdaGatedDeltaRule` GB10 **106→122/128 + 4.24 tok/s (3.1×)**, NOT STRICT, default OFF; device NoPE-MLA `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE 122→109 (§16) | device-KDA = vLLM's actual GPU recurrence; bf16 knobs hurt (122→90); MLA vt::Attention f32-softmax ≠ FA2 order, coin-flips (breaks p3). p7 near-tie; residual = chunk_kda + paged FA2 MLA + incremental §15/§16 | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B UNBLOCKED (the 13-shard bf16 DiT is now indexable) but NOT MEASURED: no bf16 render exists. Detail: benchmark-record + spec §8 | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 5a8f6902c..165b772bc 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -119,7 +119,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on | -| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | e2e runs bf16-resident; per-channel-decay KDA device kernel `vt::KdaGatedDeltaRule` landed+GB10-measured: 106→122/128 (7/8 exact) + 1.35→4.24 tok/s (3.1×), not STRICT | default off; beats §14's bf16-knob 120; residual = KDA chunked-prefill + paged FA2 MLA + incremental (§15) | +| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | e2e runs bf16-resident; KDA device op `vt::KdaGatedDeltaRule` GB10: 106→122/128 + 1.35→4.24 tok/s, not STRICT; `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE 122→109 (§16) | default off; STRICT residual = chunk_kda prefill + paged FA2 MLA + incremental (§16) | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | | `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run | @@ -250,7 +250,7 @@ abstraction, and `world_size == 1` stays byte-identical. | Gap | State | Detail | |---|---|---| -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | e2e RUNS (bf16-resident §13); per-channel-decay KDA device kernel `vt::KdaGatedDeltaRule` LANDED+GB10: 106→122/128 + 4.24 tok/s (3.1×), beats §14's 120, NOT STRICT; default OFF | p7 near-tie left; residual = KDA chunked-prefill + paged FA2 MLA + incremental decode (§15) | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | e2e RUNS (bf16-resident §13); KDA device op `vt::KdaGatedDeltaRule` GB10: 106→122/128 + 4.24 tok/s (3.1×), NOT STRICT, default OFF; `VT_KIMI_DEVICE_MLA` MEASURED-NEGATIVE 122→109 (§16) | p7 near-tie; STRICT residual = chunk_kda prefill + paged FA2 MLA + incremental decode (§16) | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Architecturally blocked | Vision tower lives outside the registered engine forward | diff --git a/docs/STATUS.md b/docs/STATUS.md index 02fdb9b1e..af439931e 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -74,7 +74,7 @@ token-for-token correctness against the pinned oracle. | DeepSeek-V2 MLA | Correctness-complete, speed-pending | Token-exact 8/8 (DeepSeek-V2-Lite); 0.86-0.95x output rate, TTFT faster at c4/c8. A2+A5 MLA norm-rope fold default-ON (`VT_MLA_FUSED_NORM_ROPE`, bit-exact rollback, SACRED 8/8 unchanged; forensics in benchmark-record) — kimi_k3/kimi-linear inherit it | | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | e2e RUNS (bf16-resident §13); per-channel-decay KDA device kernel `vt::KdaGatedDeltaRule` LANDED+GB10: **106→122/128 + 4.24 tok/s (3.1×)**, beats §14 both axes, NOT STRICT; default OFF | CPU+CUDA 13/13·656 + `test_ops_kda_recurrence` 4/4 (§15). p7 near-tie left; residual = KDA chunked-prefill + paged FA2 MLA + incremental (§15) | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | e2e RUNS (bf16-resident §13); KDA device op `vt::KdaGatedDeltaRule` GB10 **122/128, 4.24 tok/s** best, NOT STRICT, default OFF. `VT_KIMI_DEVICE_MLA` (device pad-V attn) MEASURED-NEGATIVE 122→109, 4.24→3.89 (§16) | CPU+CUDA 14/14·825; p7 near-tie; STRICT residual = KDA chunked-prefill + paged FA2 MLA + incremental (§16) | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** | diff --git a/include/vllm/model_executor/models/kimi_linear.h b/include/vllm/model_executor/models/kimi_linear.h index 31e86324b..a5237883d 100644 --- a/include/vllm/model_executor/models/kimi_linear.h +++ b/include/vllm/model_executor/models/kimi_linear.h @@ -434,6 +434,13 @@ std::vector KimiMoeBlockForwardDevice(const MoeHostWeights& w, const std::vector& hidden_normed, const KimiLinearParams& p, int64_t num_tokens, vt::Queue& queue); +// Device MLA attention CORE only (VT_KIMI_DEVICE_MLA path: pad-V + vt::Attention), +// host-in / host-out — the RED-first CPU gate for the NoPE-MLA device attention wiring. +std::vector KimiMlaAttnCoreDevice(const std::vector& q_host, + const std::vector& kv_host, + const std::vector& kpe_host, + const KimiLinearParams& p, int64_t num_tokens, + vt::Queue& queue); std::vector KimiDenseMlpForwardDevice(const MlpHostWeights& w, const std::vector& hidden_normed, const KimiLinearParams& p, diff --git a/src/vllm/model_executor/models/kimi_linear_device.cpp b/src/vllm/model_executor/models/kimi_linear_device.cpp index 4c2cf93bc..5991fce16 100644 --- a/src/vllm/model_executor/models/kimi_linear_device.cpp +++ b/src/vllm/model_executor/models/kimi_linear_device.cpp @@ -174,6 +174,39 @@ bool DeviceKda() { }(); return on; } + +// (5) VT_KIMI_DEVICE_MLA — run the NoPE-MLA attention CORE (causal scores/softmax/ +// weighted-V) through the shared device op vt::Attention instead of the f64 host +// softmax island. This is the MLA twin of VT_KIMI_DEVICE_KDA (spec §14/§15 residual +// (d)): the host island runs an f64 softmax that is MORE precise than vLLM's FA2 and +// coin-flips near-ties, whereas vt::Attention runs the f32 max-subtracted online- +// softmax accumulation vLLM's kernels use (cpu_ops AttentionKernel / cuda_ops). The +// MLA attention has asymmetric head dims (qk = qk_nope+qk_rope = 192, v = 128), which +// vt::Attention (single head-dim D for q/k/v) does not express directly, so the value +// is PADDED to the qk head-dim with zeros (weighted-sum over the zero tail = 0) and the +// out[:, :, :v] slice is the true attention core — bit-exact to the unpadded math since +// softmax weights depend only on q·k. Requires VT_KIMI_DEVICE_COMPUTE=1. Independent of +// the bf16-precision knobs (BF16_ISLANDS still rounds the q/kv/kpe inputs when both are +// set — applied on device before the attention). +// +// ★ MEASURED NEGATIVE (2026-08-07, GB10 full 48.9B gate, spec §16) — kept as a +// documented-negative A/B knob, DEFAULT OFF. On the device-KDA best config +// (VT_KIMI_DEVICE_KDA=1), adding VT_KIMI_DEVICE_MLA REGRESSES 122→109/128 AND slows +// 4.24→3.89 tok/s: (1) vt::Attention's f32 online max-subtracted softmax is NOT vLLM's +// FA2 reduction ORDER — it is a DIFFERENT approximation, so it coin-flips near-ties (it +// BREAKS p3 16/16→3/16 into the `163586×` repeat loop while p7 stays diverged), the same +// §14 plateau class; (2) the per-(t,h) key/value build copies + the 192-dim pad-V waste +// ADD overhead to the O(n²) recompute path. The principled MLA-half STRICT lever is +// vLLM's ACTUAL FA2 via paged mla::ForwardMlaAttentionBlock (residual d, coupled with +// paged-incremental decode e), NOT this softmax approximation — this negative is the +// measurement that proves the approximation is not enough. +bool DeviceMla() { + static const bool on = [] { + const char* e = std::getenv("VT_KIMI_DEVICE_MLA"); + return e != nullptr && e[0] == '1'; + }(); + return on; +} // Round a running f64 accumulator to f32 precision when the knob is on (identity else). inline double AccR(double x) { static const bool f32 = IslandF32Acc(); @@ -393,12 +426,86 @@ DBuf KdaRecurrenceIsland(const Dev& d, DBuf& qn, DBuf& kn, DBuf& vc, DBuf& g1, return DBuf(d, DType::kF32, {T, proj}, core.data()); // upload back to device } +// ── DEVICE MLA attention CORE (VT_KIMI_DEVICE_MLA): the NoPE causal softmax over the +// per-head k_nope|k_pe / v, run through the shared device op vt::Attention (f32 online +// softmax — vLLM's FA2 regime) instead of the f64 host recompute. MLA has asymmetric +// head dims (qk = qk_nope+qk_rope, v = v_head_dim), which vt::Attention (one head-dim D +// for q/k/v) does not express, so the value is PADDED to qk with zeros: the weighted sum +// over the zero tail is 0, so out[:, :, :vh] is the exact attention core (softmax weights +// depend only on q·k, which is unaffected by the padded v). q is already laid out per head +// as [q_nope(qn) | q_pe(qr)] so dq views directly as [T,nah,qk]; key is built per (t,h) as +// [k_nope(qn) | k_pe(qr, SHARED across heads)] and value as [v(vh) | 0]. Scale = qk^-0.5, +// matching the host island / kimi_linear_forward.cpp:223. Returns [T, nah*v_head_dim]. +DBuf MlaAttnCoreDevice(const Dev& d, DBuf& dq, DBuf& dkv, DBuf& dkpe, + const KimiLinearParams& p, int64_t T) { + const int64_t nah = p.num_attention_heads; + const int64_t qn = p.qk_nope_head_dim; + const int64_t qr = p.qk_rope_head_dim; + const int64_t qk = qn + qr; + const int64_t vh = p.v_head_dim; + const int64_t kvw = nah * (qn + vh); + + DBuf key(d, DType::kF32, {T, nah, qk}); + DBuf val(d, DType::kF32, {T, nah, qk}); + val.Zero(d); // pad-V: the [vh, qk) tail stays 0 + { + const size_t qkb = static_cast(qk) * sizeof(float); + const size_t qnb = static_cast(qn) * sizeof(float); + const size_t qrb = static_cast(qr) * sizeof(float); + const size_t vhb = static_cast(vh) * sizeof(float); + const char* kv = static_cast(dkv.ptr()); + const char* kpe = static_cast(dkpe.ptr()); + char* kp = static_cast(key.ptr()); + char* vp = static_cast(val.ptr()); + for (int64_t t = 0; t < T; ++t) { + const char* kpe_t = kpe + static_cast(t) * qrb; + for (int64_t h = 0; h < nah; ++h) { + const char* src = + kv + (static_cast(t) * kvw + static_cast(h) * (qn + vh)) * + sizeof(float); + char* kdst = kp + (static_cast(t) * nah + h) * qkb; + char* vdst = vp + (static_cast(t) * nah + h) * qkb; + d.b.Copy(d.q, kdst, src, qnb); // k_nope[qn] + d.b.Copy(d.q, kdst + qnb, kpe_t, qrb); // k_pe[qr] (shared across heads) + d.b.Copy(d.q, vdst, src + qnb, vhb); // v[vh]; the [vh,qk) tail stays 0 + } + } + } + // VT_KIMI_BF16_ISLANDS: round the attention inputs to bf16 precision on device + // (matching the host island's RoundHostBf16), before the f32 softmax. + if (Bf16Islands()) { + RoundDevBf16(d, dq); + RoundDevBf16(d, key); + RoundDevBf16(d, val); + } + Tensor query = MakeTensor(dq.ptr(), DType::kF32, d.q.device, {T, nah, qk}); + DBuf attn(d, DType::kF32, {T, nah, qk}); + const float scale = static_cast(std::pow(static_cast(qk), -0.5)); + vt::Attention(d.q, attn.t(), query, key.t(), val.t(), vt::AttentionArgs{scale, true}); + + // slice out[:, :, :vh] -> [T, nah*vh] (the pad-V tail is 0 by construction). + DBuf out(d, DType::kF32, {T, nah * vh}); + { + const size_t qkb = static_cast(qk) * sizeof(float); + const size_t vhb = static_cast(vh) * sizeof(float); + const char* ap = static_cast(attn.ptr()); + char* op = static_cast(out.ptr()); + for (int64_t t = 0; t < T; ++t) + for (int64_t h = 0; h < nah; ++h) + d.b.Copy(d.q, op + (static_cast(t) * nah + h) * vhb, + ap + (static_cast(t) * nah + h) * qkb, vhb); + } + return out; +} + // ── HOST-FALLBACK ISLAND: the materialized-MHA attention CORE (causal softmax over // the per-head k_nope|k_pe / v, NoPE so no RoPE). Identical math to kimi_linear_ // forward.cpp:223-258; shared by the f32 and bf16 MLA paths (only the projections -// feeding dq/dkv/dkpe differ). Returns [T, nah*v_head_dim]. +// feeding dq/dkv/dkpe differ). Returns [T, nah*v_head_dim]. When VT_KIMI_DEVICE_MLA is +// set, the attention core runs on device via vt::Attention (MlaAttnCoreDevice) instead. DBuf MlaSoftmaxIsland(const Dev& d, DBuf& dq, DBuf& dkv, DBuf& dkpe, const KimiLinearParams& p, int64_t T) { + if (DeviceMla()) return MlaAttnCoreDevice(d, dq, dkv, dkpe, p, T); const int64_t nah = p.num_attention_heads; const int64_t qn = p.qk_nope_head_dim; const int64_t qr = p.qk_rope_head_dim; @@ -1046,6 +1153,29 @@ std::vector KimiNoPEMlaLayerForwardDevice(const MlaLayerHostWeights& w, return h; } +// Device MLA attention CORE only (pad-V + vt::Attention), host-in / host-out — the +// dedicated RED-first CPU gate for the VT_KIMI_DEVICE_MLA wiring, independent of the +// env flag. q [T,nah*qk], kv [T,nah*(qn+vh)], kpe [T,qr] -> [T,nah*vh]. +std::vector KimiMlaAttnCoreDevice(const std::vector& q_host, + const std::vector& kv_host, + const std::vector& kpe_host, + const KimiLinearParams& p, int64_t num_tokens, + vt::Queue& queue) { + Dev d{vt::GetBackend(queue.device.type), queue}; + const int64_t nah = p.num_attention_heads; + const int64_t qk = p.qk_nope_head_dim + p.qk_rope_head_dim; + const int64_t vh = p.v_head_dim; + const int64_t kvw = nah * (p.qk_nope_head_dim + vh); + const int64_t qr = p.qk_rope_head_dim; + DBuf dq(d, DType::kF32, {num_tokens, nah * qk}, q_host.data()); + DBuf dkv(d, DType::kF32, {num_tokens, kvw}, kv_host.data()); + DBuf dkpe(d, DType::kF32, {num_tokens, qr}, kpe_host.data()); + DBuf out = MlaAttnCoreDevice(d, dq, dkv, dkpe, p, num_tokens); + std::vector h(static_cast(num_tokens) * nah * vh); + out.Download(d, h.data()); + return h; +} + std::vector KimiMoeBlockForwardDevice(const MoeHostWeights& w, const std::vector& hidden_normed, const KimiLinearParams& p, diff --git a/tests/vllm/models/test_kimi_linear_forward.cpp b/tests/vllm/models/test_kimi_linear_forward.cpp index 504174326..83039c418 100644 --- a/tests/vllm/models/test_kimi_linear_forward.cpp +++ b/tests/vllm/models/test_kimi_linear_forward.cpp @@ -683,6 +683,68 @@ TEST_CASE("kimi-linear W7 device: NoPE-MLA layer matches the W2 host reference") for (size_t i = 0; i < dev.size(); ++i) CHECK(Close(dev[i], ref[i], 1e-4, 1e-5)); } +// ─── (g2) device MLA attention core (VT_KIMI_DEVICE_MLA): pad-V + vt::Attention ─ +// RED-first gate for the device NoPE-MLA attention wiring (spec §15 residual (d)): +// the pad-V vt::Attention core (value padded qk_head_dim=qk with zeros; out[:, :, :vh] +// is exact) matches a from-first-principles causal-softmax reference at the Kimi MLA +// geometry (asymmetric qk=qn+qr vs v=vh, k_pe shared across heads). vt::Attention runs +// f32 softmax (vLLM's FA2 regime), the reference f64 — a tight rtol, not bit-exact. +TEST_CASE("kimi-linear W7 device: NoPE-MLA device attention core (pad-V) matches a softmax reference") { + KimiLinearParams p; + p.num_attention_heads = 3; + p.qk_nope_head_dim = 8; + p.qk_rope_head_dim = 4; // qk = 12 + p.v_head_dim = 8; + const int64_t nah = p.num_attention_heads; + const int64_t qn = p.qk_nope_head_dim, qr = p.qk_rope_head_dim, qk = qn + qr; + const int64_t vh = p.v_head_dim, kvw = nah * (qn + vh), T = 7; + + const std::vector q = Rand(T * nah * qk, 401); + const std::vector kv = Rand(T * kvw, 402); + const std::vector kpe = Rand(T * qr, 403); + + // Reference: per-head causal softmax over [k_nope | k_pe(shared)], weighted v (f64). + const double scale = std::pow(static_cast(qk), -0.5); + std::vector ref(static_cast(T) * nah * vh, 0.0f); + std::vector sc(static_cast(T)); + for (int64_t h = 0; h < nah; ++h) + for (int64_t t = 0; t < T; ++t) { + const float* q_nope = &q[t * nah * qk + h * qk]; + const float* q_pe = q_nope + qn; + double mx = -INFINITY; + for (int64_t s = 0; s <= t; ++s) { + const float* k_nope = &kv[s * kvw + h * (qn + vh)]; + const float* kp = &kpe[s * qr]; + double dot = 0.0; + for (int64_t dd = 0; dd < qn; ++dd) dot += static_cast(q_nope[dd]) * k_nope[dd]; + for (int64_t dd = 0; dd < qr; ++dd) dot += static_cast(q_pe[dd]) * kp[dd]; + dot *= scale; + sc[static_cast(s)] = dot; + mx = std::max(mx, dot); + } + double sum = 0.0; + for (int64_t s = 0; s <= t; ++s) { + const double e = std::exp(sc[static_cast(s)] - mx); + sc[static_cast(s)] = e; + sum += e; + } + float* ot = &ref[t * nah * vh + h * vh]; + for (int64_t dd = 0; dd < vh; ++dd) { + double acc = 0.0; + for (int64_t s = 0; s <= t; ++s) { + const float* vs = &kv[s * kvw + h * (qn + vh) + qn]; + acc += (sc[static_cast(s)] / sum) * static_cast(vs[dd]); + } + ot[dd] = static_cast(acc); + } + } + + vt::Queue qcpu = CpuQueue(); + const std::vector dev = vllm::KimiMlaAttnCoreDevice(q, kv, kpe, p, T, qcpu); + REQUIRE(dev.size() == ref.size()); + for (size_t i = 0; i < dev.size(); ++i) CHECK(Close(dev[i], ref[i], 3e-3, 3e-4)); +} + // ─── (h) MoE block + dense MLP: device compute == the W2 reference ──────────── TEST_CASE("kimi-linear W7 device: MoE block + dense MLP match the W2 host reference") { KimiLinearParams p;