From 1798b275db758ef3fedfceae19cdea09a44738b6 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 23:24:54 +0000 Subject: [PATCH 1/2] =?UTF-8?q?claim(kimi-linear):=20ROW=207=20ONE-SURFACE?= =?UTF-8?q?=20runner=20fold=20=E2=80=94=20row/KIMI-RUNNER-FOLD=20claim?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Opens the helper claim for ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3: fold Kimi-Linear onto the shared paged runner (B1 KV enablement, B2 KDA-paged block, B3 NoPE-MLA-paged block, B4 engine bind + thin-client example). Task #281. FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode] From 5b39361ba04981dba34cd64bf4e0369d233cbdd4 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Fri, 7 Aug 2026 23:24:54 +0000 Subject: [PATCH 2/2] =?UTF-8?q?model(kimi=5Flinear):=20ROW=207=20=E2=80=94?= =?UTF-8?q?=20fold=20Kimi-Linear=20onto=20the=20shared=20paged=20runner;?= =?UTF-8?q?=20engine=3D=3DCLI=20128/128,=20golden=20122/128=20profile;=20A?= =?UTF-8?q?BI=20v13=20vllm=5Fcomplete=5Ftokens;=20thin-client=20example=20?= =?UTF-8?q?(merged=20ratchet=208)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Branch: row/KIMI-RUNNER-FOLD (draft PR #122 — the helper claim). ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3→§21, task #281. The runner previously ABORTED on Kimi's KV (MambaSpec check vs config-derived zeros); the fast paged-incremental decode was CLI-only. Now Kimi decode runs THROUGH ModelRegistry::Forward on the runner's own paged state and the engine/server serve it. B1 — KV enablement: LoadHfConfig synthesizes layer_types + the GDN-group geometry from Kimi's nested linear_attn_config (configs/kimi_linear.py:34-148, 1-indexed kda_layers). ADDITIVE: explicit-field configs (qwen3_5) never enter the branch — the priority conjunct is PINNED by a both-present test (explicit fields WIN; mutation-verified RED when the conjunct is dropped); runner.cpp untouched. B2 — KDA-paged block (KdaLayerPagedBf16): vt::KdaChunkPrefill for fresh prefills (VT_KIMI_PAGED_KDA_CHUNK=0 A/B) / vt::KdaGatedDeltaRule (T==1) for decode + continuing prefills over the paged gdn_state group keyed by non_spec_state_indices (GdnStateGather/Scatter); conv taps via CausalConv1dFwd (varlen) / CausalConv1dUpdate (decode) in vLLM's conv_state.chunk(3) [q|k|v] layout. NOT per-head GdnBlockPaged — KDA's per-K-channel decay needs the KDA ops; shared GDN kernels untouched. B3 — NoPE-MLA-paged block: latent rows via vt::ConcatAndCacheMla at attn_meta.slot_mapping (bf16 pages; conv cache dtype follows ResolveKvCacheDType, mirroring kda_state_dtype). PRODUCTION arm = mla::ForwardMlaAttentionBlock (vLLM's absorbed-MQA decode / FA2 prefill, identity-RoPE, scale qk^-0.5, load-time AbsorbKvBProjBf16 into MlaResidentWeights::w_uk_t/w_uv) — VT_KIMI_PAGED_MLA_FA2 DEFAULT ON, GB10-ruled: it reproduces the golden's 122/128 near-tie profile EXACTLY. Diagnostic arm ('0') = the exact f64 island (CPU fold-identity vehicle; GB10 111/128 = the §19 GPU M-tiling near-tie class, not a paging bug). B4 — ONE SURFACE: registry loads the bf16-RESIDENT tower through the engine (ModelFactory::stage_on_load — queue before load, per-tensor stage-and-release, the §13 recipe); ForwardPaged honors device_token_ids (the DEFAULT-ON async device mirror leaves host ids stale — a measured GB10 9/128 divergence, RED-first CPU-pinned); vllm_complete_tokens (ABI v13 after the #123 video-v12 rebase; changelog slotted; floor pin >= 13; dlopen 28/28 + c_header_compile reference); examples/kimi_linear_gen REWRITTEN as a thin vllm.h + vllm::shared client; example-abi-allowlist kimi row REMOVED — merged ratchet MAX_INTERNAL_REACHING = 8 (with #123's two minimax removals), equality pin 8, coupled spec/state claims moved together; the CLI-incremental reference leg preserved as the env-gated test_kimi_linear_fold_gate. GB10 gates (dgx, /dev/shm CUDA build: CUTLASS 4.5.0 + FA2 + Triton AOT sm_121a; golden md5 bfa5bdbf; flock, drop_caches, worker parked, no reboot during the gates): - SACRED post-fold: 35B test_qwen36_paged_engine 2/2·315 PASS; 27B test_qwen27_paged_engine 1/1·235 PASS. - Gate A fold identity: CLI reference leg reproduces §19 EXACTLY (122/128 @ 18.93 tok/s, p7 got byte-equal); ENGINE leg (vllm_engine_load + vllm_complete_tokens, FA2 default) == CLI 128/128 BYTE-IDENTICAL; vs golden 122/128 — the SAME near-tie profile (>=122 bound MET). - Server smoke /v1/completions: PASS — streamed 48 tokens / 2.52 s = 19.0 tok/s wall, coherent; non-streamed coherent; tokenizer via TikTokenConverter (round-trip verified; Kimi ships tiktoken-only). - SPEED: server 19.0 tok/s wall (~0.90x the #111 vLLM ~21 floor) — >= vLLM NOT met; residual: per-step KDA host islands, grouped MoE, decode graph (spec §21). Example two-length diffs (16.9 N=64) carry cold-start pollution — measurement caveat recorded. - The in-session vLLM re-measure ABORTED BY BOX REBOOT (util 0.82 at graph capture — the §19-documented risk; not retried per the safety mandate; the #111 ~21 floor stays the denominator; box recovered clean, worker auto-restored). - CPU: test_kimi_linear_paged 8/8·206 (ForwardPaged logits BYTE-EQUAL the CLI at every step, mutation-RED on 4 seams; slot isolation; batched-prefill slot writes; device_token_ids pin; FA2-arm greedy == exact-arm + f32-page rejection; pid-unique temp fixtures); test_hf_config 18/18·187; test_capi 400; test_dlopen 28/28; full ctest 351/351 (pre-rebase base). Records: spec §21, STATUS/BENCHMARKS/FEATURES keyed rows, model-matrix row, state.md append + NOW refresh, benchmark-record append, ENVIRONMENT.md knobs (VT_KIMI_PAGED_MLA_FA2 documented default-ON, GB10-ruled). FOLLOWING_AGENTS_PROTOCOL Assisted-by: Claude Code:claude-fable-5 [ClaudeCode] --- .agents/NOW.md | 2 +- .agents/benchmark-record.md | 46 + .agents/model-matrix.md | 2 +- .agents/specs/kimi-linear.md | 97 ++ .agents/specs/surface-coverage-2026-08-07.md | 16 +- .agents/state.md | 35 + docs/BENCHMARKS.md | 2 +- docs/ENVIRONMENT.md | 2 + docs/FEATURES.md | 5 +- docs/STATUS.md | 2 +- examples/CMakeLists.txt | 9 +- examples/kimi_linear_gen/main.cpp | 272 ++--- include/vllm.h | 32 +- .../vllm/model_executor/models/kimi_linear.h | 25 + .../model_executor/models/model_registry.h | 5 + scripts/check-surface-coverage.py | 6 +- scripts/example-abi-allowlist.txt | 1 - src/capi/vllm_c.cpp | 81 ++ src/vllm/entrypoints/model_loader.cpp | 11 +- .../models/kimi_linear_device.cpp | 735 ++++++++++++++ .../models/kimi_linear_registry.cpp | 29 +- .../models/kimi_linear_weights.cpp | 46 +- src/vllm/transformers_utils/hf_config.cpp | 54 + tests/CMakeLists.txt | 2 + tests/capi/c_header_compile.c | 7 + tests/capi/test_capi.cpp | 68 +- tests/capi/test_dlopen.cpp | 5 + tests/scripts/test_check_surface_coverage.py | 4 +- .../models/test_kimi_linear_fold_gate.cpp | 240 +++++ tests/vllm/models/test_kimi_linear_paged.cpp | 945 ++++++++++++++++++ tests/vllm/test_hf_config.cpp | 111 ++ 31 files changed, 2695 insertions(+), 202 deletions(-) create mode 100644 tests/vllm/models/test_kimi_linear_fold_gate.cpp create mode 100644 tests/vllm/models/test_kimi_linear_paged.cpp diff --git a/.agents/NOW.md b/.agents/NOW.md index c9edbb8a1..2bce1e1e8 100644 --- a/.agents/NOW.md +++ b/.agents/NOW.md @@ -19,7 +19,7 @@ Working head: `row/backend-rocm-w0` (#41). Prior: benchmark checkpoint | f32-out GEMV audit | Only laguna + ds4 bf16 tower affected; gate models unaffected | Re-verify ds4 tower same-tool | | Invocation-parity prevention | CI guard + checklist landing | Merge; build-verify `kGemvHeuristicAlgos` on dgx | | MiniMax-H3 lane | **bf16 shards STREAM both towers (DiT + encoder); Q4_K_M enc cond cos 0.9975, 3.5° med, DIFFUSE** | render A/B on saved embeds | -| Kimi-Linear-48B | **Best: 122/128, 18.9 tok/s (0.90× vLLM); bf16 stream refuted; strict blocked by p7 near-tie** | ACTIVE: server fold + 0.10× | +| Kimi-Linear-48B | **ROW 7 fold LANDS (#122 §21): engine==CLI 128/128; golden 122/128; SACRED green; v13 tokens ABI** | ACTIVE: 19.0 tok/s vs vLLM ~21 (~0.90×) | | 35B fresh grid | **BOUND** @`1ea26427`: 0.93-1.03x, c16 0.93x. INTAKE + Option A both NEGATIVE | Lever left: prefill glue (#61) | | Qwen3.5-4B revalidation | 0.9971x @`59674cf1` (#35); TTFT/PSS pass, TPOT/ITL open | `docs/bench-evidence/` | | MXFP4 parity | c1 1.020, c2-c8 0.962-0.969. **#82 CLOSED: ptxas-lineage REFUTED (A/B ties our+vLLM PTX all ptxas/JIT; +10us=engine context, not codegen)** | TERMINAL: at parity | diff --git a/.agents/benchmark-record.md b/.agents/benchmark-record.md index 7d027d53c..2ab8eee2f 100644 --- a/.agents/benchmark-record.md +++ b/.agents/benchmark-record.md @@ -15861,3 +15861,49 @@ busy) on the same grounds. Decode is now **91.7 tok/s: 50% of the 182 tok/s roof, 1.75x off llama.cpp's 160.9**, from 8.59 at the start of the session. + +## 2026-08-07 — ROW 7 Kimi-Linear runner fold: GB10 campaign (`row/KIMI-RUNNER-FOLD` #122) + +Build: /dev/shm/kimifold, CUDA 121a, CUTLASS 4.5.0, FA2, Triton AOT sm_121a. Golden md5 +`bfa5bdbf` (§12). flock $HOME/gpu.lock + /tmp/gpu, drop_caches per leg, worker parked, +min-avail >= 21G, no reboot. Engine legs: `kimi-linear-gen` thin ABI client +(`vllm_engine_load` + `vllm_complete_tokens`, max_model_len 4096) over +`~/kimi-linear-engine-dir` (snapshot symlinks + TikTokenConverter tokenizer.json). + +| leg | config | /128 | steady tok/s | note | +|---|---|---|---|---| +| SACRED 35B | `test_qwen36_paged_engine` post-fold | 2/2·315 | — | PASS | +| SACRED 27B | `test_qwen27_paged_engine` post-fold | 1/1·235 | — | PASS | +| CLI reference | `test_kimi_linear_fold_gate`, §19 config | **122** | 18.93 (120 steps) | reproduces §19; p7 10/16 got == §19 | +| engine round 1 | FA2 unset (exact arm), pre-mirror-fix | 9 | 13.07 (16-step diff) | STALE HOST IDS (async device mirror) — root-caused, fixed | +| engine exact arm | post-fix, `VT_KIMI_PAGED_MLA_FA2=0` | 111 | 11.46 (16-step diff) | §19 M-tiling near-tie class: p7→16/16, p4 15/16, p2 token-1 cascade 0/16 | +| **engine FA2 arm** | post-fix (now DEFAULT) | **122** | 9.81 (16-step diff) | **p0-p6 16/16 + p7 got byte-equal CLI ⇒ engine==CLI 128/128** | +| engine FA2, N=64 | async sched | 122-profile (p0/p1 16/16) | **16.87** | two-length diff N=64 vs N=1 — the honest steady rate | +| engine FA2, N=64 | `VT_ASYNC_SCHED=0` | 122-profile | 16.11 | async is NOT the gap | +| default-bind | flipped-default binary, no env | **122** | 9.44 (16-step diff) | binds the shipped default | + +Server smoke (`examples/server`, /v1/completions, converted tokenizer): STREAMED 48 tokens +in 2.52 s = 19.0 tok/s WALL (incl. prefill + first-request warmup — a lower bound on steady +decode); non-streamed haiku coherent; /v1/models lists the model. + +Verdicts: Gate A engine==CLI 128/128 BYTE-IDENTICAL; golden >= 122 bound MET (122, same +profile). Speed: the SERVER 19.0 tok/s wall is the production-surface anchor (~0.90× the +#111 vLLM ~21 floor; CLI 18.93 reproduced) — NOT >= vLLM. MEASUREMENT CAVEAT: the example's +two-length diffs (N=64 16.9; N=16 9.8-11.5) run the LONG leg first and cold, so one-time +CUDA warmup pollutes the subtraction — the diffs UNDERSTATE steady decode; anchor on the +server wall (or run the short leg first / a warmup pass). Residual = per-step KDA host +islands (beta/g1 downloads + host decay gate per layer), grouped MoE via the shared seam, +decode graph. Exact-island arm kept as diagnostic (VT_KIMI_PAGED_MLA_FA2=0), its 111/128 +recorded as the §19 M-tiling near-tie regime, not a bug. + +vLLM same-session re-measure: ATTEMPTED and ABORTED BY BOX REBOOT. The leg (oracle venv, +util 0.82 — the #111-precedented config, no tracing, worker parked, 95G+ free at launch) +loaded all 20 shards and reached torch.compile/graph capture, then the box HARD-REBOOTED +at 00:25:32 (journal boot logs; min-avail had sat at the 15-17G floor during load). This +REPRODUCES the §19 finding that vLLM@0.82 + any additional pressure sits below the +life-critical floor on the 119G unified pool — per the safety mandate ("do not retry +higher", and now: do not retry AT 0.82 with compile/capture on), the leg is NOT retried. +The DENOMINATOR for this campaign therefore remains the #111 recorded floor (~21 tok/s, +16-token aggregate, same prompts/workload). Box recovered clean: single reboot, GPU +visible, local-ai-worker auto-restored by --restart=always; /dev/shm build tree gone +(campaign complete; all gate logs under $HOME). diff --git a/.agents/model-matrix.md b/.agents/model-matrix.md index 3a87c2ff7..af7c8b9cc 100644 --- a/.agents/model-matrix.md +++ b/.agents/model-matrix.md @@ -81,7 +81,7 @@ Engaged architectures (the 47 non-`INVENTORIED` rows): | 🚧 | `Glm4ForCausalLM` | GLM-4-9B-0414 (dense, sandwich norms + partial interleaved rope) | SACRED gate 16/16 vs vLLM 0.25.0 (STRICT token-exact 13/16 + near-tie-band 3/16, max gap 0 nats, 0 forward-divergent; vLLM K=5 self-deterministic → STRICT bar); speed pending | `MODEL-TEXT-glm4-glm4-for-causal-lm` | | 📋 | `Glm4MoeForCausalLM` | GLM-4 MoE | scoped in the GLM/DSA spike, not implemented | `MODEL-TEXT-glm4-moe-glm4-moe-for-causal-lm` | | ✅ | `Glm4MoeLiteForCausalLM` | GLM-4.7-Flash (31.2B MLA + GLM MoE) | SACRED gate 8/8 vs vLLM 0.25.0 (STRICT token-exact 1/8 + near-tie-band 7/8, 69/128 tokens strictly exact, max teacher-forced gap 0.0 nats, 0 forward-divergent; vLLM K=5 self-deterministic → STRICT bar); FIRST e2e coverage of the q_lora query branch AND the noaux_tc sigmoid router (closes the MLA campaign's two gaps, C2); speed pending | `MODEL-TEXT-glm4-moe-lite-glm4-moe-lite-for-causal-lm` | -| 🚧 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B | **PAGED-INCREMENTAL DECODE LANDS the 5× speed win (2026-08-07, §19, `row/KIMI-PAGED-INCREMENTAL` #113):** the §18 real lever (e) BUILT — `KimiDecodeCache` + `ForwardPrefillIncremental`/`ForwardDecodeStepIncremental` (`kimi_linear_device.cpp`): prefill-once (KDA recurrent+conv state carried via `vt::KdaGatedDeltaRule` state in/out + `vt::CausalConv1dFwd` tap-carry; NoPE-MLA latent-KV cached) + recurrent decode-step, MIRRORING vLLM `kimi_gdn_linear_attn._forward` (prefill=`chunk_kda_with_fused_gate` output_final_state / decode=`fused_recurrent_kda` initial_state, `vllm-src` `a4e3cb4`; divergences: host state vs paged slot cache, materialized-MHA MLA vs paged-FA2 — named residuals). CPU byte-exact state-carry gate `test_kimi_linear_forward` **15/15·875** (NEW case l: carried decode == fresh full-recompute byte-identical + greedy-identical). Full 48.9B GB10 (single-load/config, flock, drop_caches, min-avail 18-21 GiB, no reboot, §12 golden md5 `bfa5bdbf`): recompute 122/128 @ 4.23 tok/s (reproduces #111); incremental+recurrence 120/128 @ 16.63; **incremental+chunk-prefill 122/128 @ 18.87/19.03 tok/s (2 runs) — token-IDENTICAL to recompute (Gate A PASS, p7 `got` byte-exact) = 4.5× over recompute, 0.90× of vLLM ~21 (the 5× decode gap 0.20×→0.90×)**. Gate B STRICT NOT reached (122/128): chunk-prefill in the RIGHT vehicle reproduces recompute EXACTLY, does NOT close p7 — REFUTES the #111 "p7 in the right vehicle → STRICT" hypothesis; p7 intrinsic (§13/§14 f32-vs-bf16 near-tie at a comma). Decode decomposition (nsys, ours, 99 steps, same-tool): **~90% is the SAME cuBLAS `internal::gemvx::kernel` vLLM calls (batch-1 GEMV-parity)**, KdaScanKernel 2.3%, MoE glue 2.3%, CastBf16 3%; chunk kernels 20 inst = prefill only (prefill=chunk/decode=recurrent IN VIVO). Killing O(n²) ALONE reaches parity-class; no lever load-bearing beyond it — residual = ~15% host-orchestration idle + 3% CastBf16 (a bf16 residual stream = the ONE lever ALSO closing the p7-STRICT near-tie) + paged-FA2 MLA decode. vLLM-live-nsys@0.82 NOT run (box-safety: 95-98 GiB reservation + nsys below the 15 GiB LIFE-CRITICAL floor). `--incremental` opt-in; `VT_KIMI_DEVICE_KDA`/`_CHUNK` STAY OFF (122/128 ≠ STRICT). Row STAYS 🚧. **chunk_kda PREFILL PHASE-2 MEASURED — op CORRECT, chunk-EVERY-STEP REGRESSES 122→102 (2026-08-07, §18, `row/KIMI-CHUNK-KDA-P2` #111):** the `chunk_kda` prefill family regenerated + vendored for ALL 6 arches (reproducible — only new `kda_*`+MANIFEST; GDN cubins byte-identical; drift GREEN) + wired through the new op `vt::KdaChunkPrefill` (the 6-cubin `_chunk_kda_fwd_with_cumulative_g`; `cuda_gdn.cu.o` -Werror clean; RED-first unit `test_ops_kda_chunk_prefill` **2/2·4** on GB10 — chunk-vs-recurrence mean_abs **4.68e-5**, wrong-gate **72×**; GDN untouched 66/66·4242). Full 48.9B GB10 gate (flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+chunk-prefill (`VT_KIMI_DEVICE_KDA_CHUNK`) REGRESSES to 102/128, 4.08 tok/s** (p3 16→3, p6 16→11). Root cause: the island's O(n²) recompute applies chunk EVERY decode step over the growing sequence — NOT vLLM's prefill=chunk/decode=recurrent split — so it coin-flips near-ties the recurrence-every-step (control) doesn't (the recurrence matches vLLM's DECODE; chunk only matches its PREFILL). vLLM speed arm (§12 recipe, util 0.82, triton MoE, eager, single-seq; min-avail 15 GiB, no reboot): **~21 tok/s median** 16-token aggregate (25.3 cold-discarded; TTFT not isolable in 0.25.0) vs ours **4.24** (recurrence) / **4.08** (chunk) STEADY decode → **ours/vLLM ≈ 0.20** (vLLM ~5× faster on decode — the O(n²)-recompute vs paged-incremental distance, = the coupled STRICT+speed lever). `VT_KIMI_DEVICE_KDA_CHUNK` STAYS OFF (a regression isn't a flip); device-KDA (122, OFF) still best. The op + regen are the validated prefill half of the named real lever (e) paged-incremental decode (chunk-prefill ONCE + recurrent-decode over PERSISTENT state — kills the O(n²); the STRICT + speed lever, coupled). Row STAYS 🚧. **DEVICE-KDA GB10 122/128 + 4.24 tok/s (§15, #104); device NoPE-MLA lever MEASURED-NEGATIVE (2026-08-07, §16, `row/KIMI-STRICT-CLOSE` #107):** the per-channel-decay device recurrence `vt::KdaGatedDeltaRule` moves 106→**122/128** (p0-p6 16/16; sole p7 pos-6 comma near-tie) AND **1.35→4.24 tok/s (3.1×)** — vLLM's ACTUAL f32-on-bf16 arithmetic, beats §14's host-precision 120. The §15 residual (d) was attempted in device-COMPUTE form: `VT_KIMI_DEVICE_MLA` routes the 7 NoPE-MLA layers' softmax core through `vt::Attention` (pad-V: value zero-padded qk_nope+qk_rope=192 vs v=128, `out[:,:,:v]` byte-exact). CPU RED-first gate GREEN (`test_kimi_linear_forward` **14/14·825**, pad-V==f64 ref rtol 3e-3; perturbation fails 108). Full 48.9B GB10 gate (single-load, flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+device-MLA REGRESSES to 109/128 AND 3.89 tok/s** — `vt::Attention`'s f32 online-softmax is the right math but a DIFFERENT reduction order than vLLM's FA2, so it coin-flips near-ties (breaks p3 16→3 into §14's `163586×` repeat) and the per-(t,h) build slows the O(n²) recompute. `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (§14 `ISLAND_F32ACC` precedent). MLA dims VERIFIED from the real config (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA). Both device knobs default OFF (122 ≠ STRICT, K=3-deterministic golden). STRICT residual, sharpened: needs vLLM's ACTUAL kernels — (c) chunk_kda prefill family (Triton-AOT regen for sm_121a) + (d) paged FA2 `mla::ForwardMlaAttentionBlock` (NOT the vt::Attention approximation) + (e) paged-incremental decode (needs a decode/paged-attn op, query_len≠key_len; kills the O(n²)) — each a substantial multi-kernel brick (§16). Row STAYS 🚧. **FULL-MODEL GB10 e2e RUNS — NEAR-TIE 106/128 (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the bf16-resident path CLEARS the f32-loader block — the full 48.9B model now runs e2e on one GB10. dgx CUDA build (`-Werror` clean, 14 GDN AOT symbols nm-linked, `test_kimi_linear_forward` 13/13·656 in the CUDA binary); `kimi-linear-gen --gpu` greedy-decodes the §12 8-prompt battery x16 vs `greedy_ids.npy`. MEMORY: load 117.6s, host RSS PEAK **1.7 GiB** (stage-then-ReleaseHost), device peak 98.5 GiB, min-avail **21.6 GiB** (above the 15 GiB floor, matches the ~25 GiB pool-math headroom), NO OOM/reboot. TOKEN gate **NEAR-TIE 106/128 (82.8%)** — prompts 0,1,3,4,5,6 are 16/16 token-exact, p2/p7 diverge at punctuation/word near-ties; 96 consecutive exact tokens across 6 prompts prove the WIRING (a wiring bug can't). Root cause (honest): the f32 residual stream + host-f64 islands are MORE precise than vLLM's bf16 device kernels, so they flip the argmax where vLLM's deterministic bf16 top-1 has a small margin. STRICT path = the named W7-speed residuals (device GDN/MLA islands -> bf16 stream matching vLLM's rounding). 1.59 tok/s (recompute+island rate). `VT_KIMI_DEVICE_COMPUTE` STAYS OFF (parity-enablers: near-tie != token-exact). Row STAYS 🚧. **bf16-RESIDENT loader/forward IMPLEMENTED + CPU-gated (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the §13 design is coded — `LoadKimiLinearResidentBf16Weights`/`StageKimiResidentBf16`/`BuildKimiResidentFromHost` (`kimi_linear_weights.cpp`; `LoadBf16Direct` -> `OwnedTensor`, per-tensor stage-to-`d_dev` + `ReleaseHost`, tiny vectors host f32), `KimiLinearResidentWeights` (`kimi_linear.h`), bf16 device forward `DeviceForwardBodyBf16` + `Gemm Bf16` cast-act at ~20 GEMM sites with the two host-fallback islands EXTRACTED+shared (`kimi_linear_device.cpp`), `ForwardDevice` resident-path dispatch (`kimi_linear.cpp`), and the `kimi-linear-gen` e2e harness. CPU **13/13·656** (12/12·614 f32 path UNTOUCHED + NEW tiny-config bf16-vs-f32 gate). PENDING: dgx CUDA build + full-model GB10 e2e vs the STRICT golden. Row STAYS 🚧. **bf16-RESIDENT brick POOL-MATH+DESIGN (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** pool math CLOSES (91.5 GiB bf16 device-resident + ~2.4 GiB act/norms/ctx ≈ 94 GiB, ~25 GiB headroom); design grounded §13 (Laguna `GemmBf16` cast-act + `OwnedTensor::d_dev`, `LoadBf16Direct`, f32 `MaterializeHost` kept for the unit gate). Impl (loader/forward rewrite + gate + e2e) pending. Row STAYS 🚧. **§8 GOLDEN CAPTURED — STRICT (2026-08-06, `row/MODEL-KIMI-LINEAR-E2E`):** the §8 SACRED oracle golden is captured on GB10 (0.25.0-stage, util 0.82, moe=triton, min 15 GiB avail, NO reboot), **8/8 prompts DETERMINISTIC over K=3 → STRICT gate**, committed at `tests/parity/goldens/kimi_linear_greedy/`. Full our-engine e2e BLOCKED on OUR f32 loader (materializes ~183 GiB > 119 pool), the bf16-residency residual; row STAYS 🚧. **W7 GPU-VERIFY (2026-08-06, branch `row/MODEL-KIMI-LINEAR-GPU`):** the device compute runs **12/12·614 GREEN on GB10 sm_121a CUDA build**, BOTH arms (`VT_KIMI_DEVICE_COMPUTE=1` + host-ref); prod stack (CUTLASS-NVFP4 GEMM + FA2 ENABLED + Triton-AOT GDN, 14 cubins nm-verified); f32 device==W2 ref, no divergence, no DeepSeek-class trap. Oracle gateability re-confirmed (0.25.0-stage registers `KimiLinearForCausalLM`). e2e §8 SACRED golden STILL disk-blocked (91.5 GiB checkpoint absent, dgx root 100% full, 34G free). Row STAYS 🚧. **W7 DBuf-resident device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`): the real device compute (`ForwardDeviceCompute`, `kimi_linear_device.cpp`) composes the whole 27-layer KDA/NoPE-MLA + 256-expert-MoE hybrid over pooled f32 `DBuf`s through the SHARED `vt::` ops (embed/`FusedChain` add+RMSNorm/`MatmulBT` projections/`CausalConv1dFwd` convs/`L2Norm`/`RmsNormGated`/`MoeRouterTopK` sigmoid-`noaux_tc`/`MoeSiluMul`/`MoeCombine`/lm_head), returning DEVICE-RESIDENT logits; 2 documented HOST-FALLBACK islands (the KDA per-k-channel gated-delta recurrence + its exp/softplus decay gate — `vt::GdnDecode` carries only a per-HEAD scalar decay; the NoPE-MLA softmax core — the paged `mla::ForwardMlaAttentionBlock` device path is born-on-runner) are the W7-speed residuals. CPU-gated vs the W2 host reference (the CPU backend runs the SAME `vt::` dispatch): `test_kimi_linear_forward` **12/12·614** (per-op KDA/NoPE-MLA/MoE/dense device==ref within f32-accumulation tolerance; the whole `ForwardDeviceCompute` == ref logits + greedy-token-identical + device-resident). Runner opt-in via `VT_KIMI_DEVICE_COMPUTE=1` (default OFF keeps the CPU-verified W6 host-ref compose). GPU numerics (bf16 activations, GDN Triton-AOT cubins, paged het-KV, grouped-MoE slabs) + the e2e SACRED golden stay a NAMED pending (box down) — row STAYS 🚧. ON TOP OF **W6 DEVICE forward SEAM** (`CLAIM-KIMI-LINEAR-W6`): the born-on-the-runner `ForwardDevice` (the DEFAULT `gather_logits` runner path) no longer refuses — it composes the `[rows,vocab]` logits via the CPU reference and hands them back DEVICE-RESIDENT (a pooled `DBuf`, wrapped like deepseek_v2 `WrapDeviceLogits`; `on_device()==true` on CPU+CUDA) so the on-GPU sampler consumes them with NO host download. Kimi-Linear now ROUTES device-resident (`check-runner-routing-consistency` reclassifies it, refuse-skipped stubs 2→1, NO allowlist; `check-fusion-consistency` green); `test_kimi_linear_forward` **7/7·300** (adds the `ForwardDevice`==host-ref device-resident gate). The DBuf-resident device COMPUTE (KDA via the GDN family, NoPE-MLA via `mla::ForwardMlaAttentionBlock`, DeepSeek-V2 grouped-MoE over the paged het-KV; full plan in `kimi_linear.cpp`) is the GPU-verify-pending W7 residual. ON TOP OF **W2-W6 CPU REFERENCE forward** (`CLAIM-KIMI-LINEAR-W2`): the real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives (KDA layer via `vllm::kimi_kda` refs + the gated-delta recurrence; NoPE-MLA materialized-MHA ref; sigmoid `noaux_tc` MoE + shared expert; dense SwiGLU); loader now materializes host float weights; `test_kimi_linear_forward` 6/6·246 (per-op gates + finite whole forward + greedy decode). ON TOP OF **W1 scaffolding** (registry + `ParseKimiLinearParams` 20 KDA + 7 NoPE-MLA + index-verified name-map + het-KV spec). e2e-gateable (FITS one GB10, 0.77× pool). RESIDUAL = the DEVICE born-on-runner forward (KDA kernel/absorbed-MLA/grouped-MoE slabs) + the W0/W7 e2e SACRED golden. Row → `ACTIVE` (device SEAM wired; the DBuf device compute + e2e SACRED golden pending) | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` | +| 🚧 | `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B | **PAGED-INCREMENTAL DECODE LANDS the 5× speed win (2026-08-07, §19, `row/KIMI-PAGED-INCREMENTAL` #113):** the §18 real lever (e) BUILT — `KimiDecodeCache` + `ForwardPrefillIncremental`/`ForwardDecodeStepIncremental` (`kimi_linear_device.cpp`): prefill-once (KDA recurrent+conv state carried via `vt::KdaGatedDeltaRule` state in/out + `vt::CausalConv1dFwd` tap-carry; NoPE-MLA latent-KV cached) + recurrent decode-step, MIRRORING vLLM `kimi_gdn_linear_attn._forward` (prefill=`chunk_kda_with_fused_gate` output_final_state / decode=`fused_recurrent_kda` initial_state, `vllm-src` `a4e3cb4`; divergences: host state vs paged slot cache, materialized-MHA MLA vs paged-FA2 — named residuals). CPU byte-exact state-carry gate `test_kimi_linear_forward` **15/15·875** (NEW case l: carried decode == fresh full-recompute byte-identical + greedy-identical). Full 48.9B GB10 (single-load/config, flock, drop_caches, min-avail 18-21 GiB, no reboot, §12 golden md5 `bfa5bdbf`): recompute 122/128 @ 4.23 tok/s (reproduces #111); incremental+recurrence 120/128 @ 16.63; **incremental+chunk-prefill 122/128 @ 18.87/19.03 tok/s (2 runs) — token-IDENTICAL to recompute (Gate A PASS, p7 `got` byte-exact) = 4.5× over recompute, 0.90× of vLLM ~21 (the 5× decode gap 0.20×→0.90×)**. Gate B STRICT NOT reached (122/128): chunk-prefill in the RIGHT vehicle reproduces recompute EXACTLY, does NOT close p7 — REFUTES the #111 "p7 in the right vehicle → STRICT" hypothesis; p7 intrinsic (§13/§14 f32-vs-bf16 near-tie at a comma). Decode decomposition (nsys, ours, 99 steps, same-tool): **~90% is the SAME cuBLAS `internal::gemvx::kernel` vLLM calls (batch-1 GEMV-parity)**, KdaScanKernel 2.3%, MoE glue 2.3%, CastBf16 3%; chunk kernels 20 inst = prefill only (prefill=chunk/decode=recurrent IN VIVO). Killing O(n²) ALONE reaches parity-class; no lever load-bearing beyond it — residual = ~15% host-orchestration idle + 3% CastBf16 (a bf16 residual stream = the ONE lever ALSO closing the p7-STRICT near-tie) + paged-FA2 MLA decode. vLLM-live-nsys@0.82 NOT run (box-safety: 95-98 GiB reservation + nsys below the 15 GiB LIFE-CRITICAL floor). `--incremental` opt-in; `VT_KIMI_DEVICE_KDA`/`_CHUNK` STAY OFF (122/128 ≠ STRICT). Row STAYS 🚧. **chunk_kda PREFILL PHASE-2 MEASURED — op CORRECT, chunk-EVERY-STEP REGRESSES 122→102 (2026-08-07, §18, `row/KIMI-CHUNK-KDA-P2` #111):** the `chunk_kda` prefill family regenerated + vendored for ALL 6 arches (reproducible — only new `kda_*`+MANIFEST; GDN cubins byte-identical; drift GREEN) + wired through the new op `vt::KdaChunkPrefill` (the 6-cubin `_chunk_kda_fwd_with_cumulative_g`; `cuda_gdn.cu.o` -Werror clean; RED-first unit `test_ops_kda_chunk_prefill` **2/2·4** on GB10 — chunk-vs-recurrence mean_abs **4.68e-5**, wrong-gate **72×**; GDN untouched 66/66·4242). Full 48.9B GB10 gate (flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+chunk-prefill (`VT_KIMI_DEVICE_KDA_CHUNK`) REGRESSES to 102/128, 4.08 tok/s** (p3 16→3, p6 16→11). Root cause: the island's O(n²) recompute applies chunk EVERY decode step over the growing sequence — NOT vLLM's prefill=chunk/decode=recurrent split — so it coin-flips near-ties the recurrence-every-step (control) doesn't (the recurrence matches vLLM's DECODE; chunk only matches its PREFILL). vLLM speed arm (§12 recipe, util 0.82, triton MoE, eager, single-seq; min-avail 15 GiB, no reboot): **~21 tok/s median** 16-token aggregate (25.3 cold-discarded; TTFT not isolable in 0.25.0) vs ours **4.24** (recurrence) / **4.08** (chunk) STEADY decode → **ours/vLLM ≈ 0.20** (vLLM ~5× faster on decode — the O(n²)-recompute vs paged-incremental distance, = the coupled STRICT+speed lever). `VT_KIMI_DEVICE_KDA_CHUNK` STAYS OFF (a regression isn't a flip); device-KDA (122, OFF) still best. The op + regen are the validated prefill half of the named real lever (e) paged-incremental decode (chunk-prefill ONCE + recurrent-decode over PERSISTENT state — kills the O(n²); the STRICT + speed lever, coupled). Row STAYS 🚧. **DEVICE-KDA GB10 122/128 + 4.24 tok/s (§15, #104); device NoPE-MLA lever MEASURED-NEGATIVE (2026-08-07, §16, `row/KIMI-STRICT-CLOSE` #107):** the per-channel-decay device recurrence `vt::KdaGatedDeltaRule` moves 106→**122/128** (p0-p6 16/16; sole p7 pos-6 comma near-tie) AND **1.35→4.24 tok/s (3.1×)** — vLLM's ACTUAL f32-on-bf16 arithmetic, beats §14's host-precision 120. The §15 residual (d) was attempted in device-COMPUTE form: `VT_KIMI_DEVICE_MLA` routes the 7 NoPE-MLA layers' softmax core through `vt::Attention` (pad-V: value zero-padded qk_nope+qk_rope=192 vs v=128, `out[:,:,:v]` byte-exact). CPU RED-first gate GREEN (`test_kimi_linear_forward` **14/14·825**, pad-V==f64 ref rtol 3e-3; perturbation fails 108). Full 48.9B GB10 gate (single-load, flock, min-avail 21 GiB, no reboot): control device-KDA reproduces **122/128, 4.24 tok/s** EXACTLY; **+device-MLA REGRESSES to 109/128 AND 3.89 tok/s** — `vt::Attention`'s f32 online-softmax is the right math but a DIFFERENT reduction order than vLLM's FA2, so it coin-flips near-ties (breaks p3 16→3 into §14's `163586×` repeat) and the per-(t,h) build slows the O(n²) recompute. `VT_KIMI_DEVICE_MLA` STAYS OFF, kept as a documented-MEASURED-NEGATIVE A/B knob (§14 `ISLAND_F32ACC` precedent). MLA dims VERIFIED from the real config (nah=32, qk_nope=128, qk_rope=64, v=128, kv_lora=512, q_lora=None; 7 full-attn/20 KDA). Both device knobs default OFF (122 ≠ STRICT, K=3-deterministic golden). STRICT residual, sharpened: needs vLLM's ACTUAL kernels — (c) chunk_kda prefill family (Triton-AOT regen for sm_121a) + (d) paged FA2 `mla::ForwardMlaAttentionBlock` (NOT the vt::Attention approximation) + (e) paged-incremental decode (needs a decode/paged-attn op, query_len≠key_len; kills the O(n²)) — each a substantial multi-kernel brick (§16). Row STAYS 🚧. **FULL-MODEL GB10 e2e RUNS — NEAR-TIE 106/128 (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the bf16-resident path CLEARS the f32-loader block — the full 48.9B model now runs e2e on one GB10. dgx CUDA build (`-Werror` clean, 14 GDN AOT symbols nm-linked, `test_kimi_linear_forward` 13/13·656 in the CUDA binary); `kimi-linear-gen --gpu` greedy-decodes the §12 8-prompt battery x16 vs `greedy_ids.npy`. MEMORY: load 117.6s, host RSS PEAK **1.7 GiB** (stage-then-ReleaseHost), device peak 98.5 GiB, min-avail **21.6 GiB** (above the 15 GiB floor, matches the ~25 GiB pool-math headroom), NO OOM/reboot. TOKEN gate **NEAR-TIE 106/128 (82.8%)** — prompts 0,1,3,4,5,6 are 16/16 token-exact, p2/p7 diverge at punctuation/word near-ties; 96 consecutive exact tokens across 6 prompts prove the WIRING (a wiring bug can't). Root cause (honest): the f32 residual stream + host-f64 islands are MORE precise than vLLM's bf16 device kernels, so they flip the argmax where vLLM's deterministic bf16 top-1 has a small margin. STRICT path = the named W7-speed residuals (device GDN/MLA islands -> bf16 stream matching vLLM's rounding). 1.59 tok/s (recompute+island rate). `VT_KIMI_DEVICE_COMPUTE` STAYS OFF (parity-enablers: near-tie != token-exact). Row STAYS 🚧. **bf16-RESIDENT loader/forward IMPLEMENTED + CPU-gated (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** the §13 design is coded — `LoadKimiLinearResidentBf16Weights`/`StageKimiResidentBf16`/`BuildKimiResidentFromHost` (`kimi_linear_weights.cpp`; `LoadBf16Direct` -> `OwnedTensor`, per-tensor stage-to-`d_dev` + `ReleaseHost`, tiny vectors host f32), `KimiLinearResidentWeights` (`kimi_linear.h`), bf16 device forward `DeviceForwardBodyBf16` + `Gemm Bf16` cast-act at ~20 GEMM sites with the two host-fallback islands EXTRACTED+shared (`kimi_linear_device.cpp`), `ForwardDevice` resident-path dispatch (`kimi_linear.cpp`), and the `kimi-linear-gen` e2e harness. CPU **13/13·656** (12/12·614 f32 path UNTOUCHED + NEW tiny-config bf16-vs-f32 gate). PENDING: dgx CUDA build + full-model GB10 e2e vs the STRICT golden. Row STAYS 🚧. **bf16-RESIDENT brick POOL-MATH+DESIGN (2026-08-06, `row/MODEL-KIMI-LINEAR-BF16`):** pool math CLOSES (91.5 GiB bf16 device-resident + ~2.4 GiB act/norms/ctx ≈ 94 GiB, ~25 GiB headroom); design grounded §13 (Laguna `GemmBf16` cast-act + `OwnedTensor::d_dev`, `LoadBf16Direct`, f32 `MaterializeHost` kept for the unit gate). Impl (loader/forward rewrite + gate + e2e) pending. Row STAYS 🚧. **§8 GOLDEN CAPTURED — STRICT (2026-08-06, `row/MODEL-KIMI-LINEAR-E2E`):** the §8 SACRED oracle golden is captured on GB10 (0.25.0-stage, util 0.82, moe=triton, min 15 GiB avail, NO reboot), **8/8 prompts DETERMINISTIC over K=3 → STRICT gate**, committed at `tests/parity/goldens/kimi_linear_greedy/`. Full our-engine e2e BLOCKED on OUR f32 loader (materializes ~183 GiB > 119 pool), the bf16-residency residual; row STAYS 🚧. **W7 GPU-VERIFY (2026-08-06, branch `row/MODEL-KIMI-LINEAR-GPU`):** the device compute runs **12/12·614 GREEN on GB10 sm_121a CUDA build**, BOTH arms (`VT_KIMI_DEVICE_COMPUTE=1` + host-ref); prod stack (CUTLASS-NVFP4 GEMM + FA2 ENABLED + Triton-AOT GDN, 14 cubins nm-verified); f32 device==W2 ref, no divergence, no DeepSeek-class trap. Oracle gateability re-confirmed (0.25.0-stage registers `KimiLinearForCausalLM`). e2e §8 SACRED golden STILL disk-blocked (91.5 GiB checkpoint absent, dgx root 100% full, 34G free). Row STAYS 🚧. **W7 DBuf-resident device COMPUTE landed, CPU-gated** (`CLAIM-KIMI-LINEAR-W7`): the real device compute (`ForwardDeviceCompute`, `kimi_linear_device.cpp`) composes the whole 27-layer KDA/NoPE-MLA + 256-expert-MoE hybrid over pooled f32 `DBuf`s through the SHARED `vt::` ops (embed/`FusedChain` add+RMSNorm/`MatmulBT` projections/`CausalConv1dFwd` convs/`L2Norm`/`RmsNormGated`/`MoeRouterTopK` sigmoid-`noaux_tc`/`MoeSiluMul`/`MoeCombine`/lm_head), returning DEVICE-RESIDENT logits; 2 documented HOST-FALLBACK islands (the KDA per-k-channel gated-delta recurrence + its exp/softplus decay gate — `vt::GdnDecode` carries only a per-HEAD scalar decay; the NoPE-MLA softmax core — the paged `mla::ForwardMlaAttentionBlock` device path is born-on-runner) are the W7-speed residuals. CPU-gated vs the W2 host reference (the CPU backend runs the SAME `vt::` dispatch): `test_kimi_linear_forward` **12/12·614** (per-op KDA/NoPE-MLA/MoE/dense device==ref within f32-accumulation tolerance; the whole `ForwardDeviceCompute` == ref logits + greedy-token-identical + device-resident). Runner opt-in via `VT_KIMI_DEVICE_COMPUTE=1` (default OFF keeps the CPU-verified W6 host-ref compose). GPU numerics (bf16 activations, GDN Triton-AOT cubins, paged het-KV, grouped-MoE slabs) + the e2e SACRED golden stay a NAMED pending (box down) — row STAYS 🚧. ON TOP OF **W6 DEVICE forward SEAM** (`CLAIM-KIMI-LINEAR-W6`): the born-on-the-runner `ForwardDevice` (the DEFAULT `gather_logits` runner path) no longer refuses — it composes the `[rows,vocab]` logits via the CPU reference and hands them back DEVICE-RESIDENT (a pooled `DBuf`, wrapped like deepseek_v2 `WrapDeviceLogits`; `on_device()==true` on CPU+CUDA) so the on-GPU sampler consumes them with NO host download. Kimi-Linear now ROUTES device-resident (`check-runner-routing-consistency` reclassifies it, refuse-skipped stubs 2→1, NO allowlist; `check-fusion-consistency` green); `test_kimi_linear_forward` **7/7·300** (adds the `ForwardDevice`==host-ref device-resident gate). The DBuf-resident device COMPUTE (KDA via the GDN family, NoPE-MLA via `mla::ForwardMlaAttentionBlock`, DeepSeek-V2 grouped-MoE over the paged het-KV; full plan in `kimi_linear.cpp`) is the GPU-verify-pending W7 residual. ON TOP OF **W2-W6 CPU REFERENCE forward** (`CLAIM-KIMI-LINEAR-W2`): the real host `KimiLinearModel::Forward` composes the whole 27-layer hybrid from the landed primitives (KDA layer via `vllm::kimi_kda` refs + the gated-delta recurrence; NoPE-MLA materialized-MHA ref; sigmoid `noaux_tc` MoE + shared expert; dense SwiGLU); loader now materializes host float weights; `test_kimi_linear_forward` 6/6·246 (per-op gates + finite whole forward + greedy decode). ON TOP OF **W1 scaffolding** (registry + `ParseKimiLinearParams` 20 KDA + 7 NoPE-MLA + index-verified name-map + het-KV spec). e2e-gateable (FITS one GB10, 0.77× pool). RESIDUAL = the DEVICE born-on-runner forward (KDA kernel/absorbed-MLA/grouped-MoE slabs) + the W0/W7 e2e SACRED golden. **ROW 7 RUNNER FOLD (2026-08-07, §21, #122): Kimi decode now runs THROUGH `ModelRegistry::Forward` on the runner's paged state (B1 config synthesis, B2 KDA-paged via `KdaChunkPrefill`/`KdaGatedDeltaRule` over `gdn_state`, B3 paged NoPE-MLA with `mla::ForwardMlaAttentionBlock` default-ON, B4 `vllm_complete_tokens` ABI v13 + thin-client example). GB10: engine==CLI 128/128 byte-identical, vs golden 122/128 (the intrinsic near-tie profile), SACRED post-fold green (35B 315/315, 27B 235/235); server stream 19.0 tok/s wall vs vLLM ~21 (~0.90×) = the open speed residual.** Row stays `ACTIVE` on the speed thread | `MODEL-TEXT-kimi-linear-kimi-linear-for-causal-lm` | | 📋 | `KimiK3ForConditionalGeneration` | Kimi K3 (2.8T MoE + MoonViT-V2, DERIVE-AND-SHIP) | **W2/W5 CPU scaffolding landed** (registry stub + nested text/vision/quant config descent + text-backbone structural name-map + REFUSE-by-name forward + MXFP4-refuse loader; clean CPU build, scaffold gate 6/6). text backbone IS `KimiLinearForCausalLM` (KDA+MLA+MoE hybrid, HEAVY reuse); **does NOT fit GB10 (~1.56 TB MXFP4, ~12×)** and NOT in the pinned oracle ⇒ no on-box golden — DERIVED, proxy-gated on Kimi-Linear-48B; forward + MXFP4 + KDA delta + MoonViT-V2 not implemented (NOT-YET-BUILDABLE) | `MODEL-MM-kimi-k3-kimi-k3-for-conditional-generation` | | 🚧 | `MiniMaxH3DiTModel` | MiniMax-H3 (33.1B omni-modal video+audio DiT, DERIVE-AND-SHIP) | **W1/W2 landed**: packed layout (fl2va + ref2va, fp64 position grid BIT-EXACT), latent packing, euler-ancestral eta0 scheduler, and the full DiT forward all parity-gated against the UPSTREAM vLLM-Omni modules executed at reduced dimensions (**max abs diff 1.6e-7**, 10/10 cases / 2539 assertions). NOT autoregressive (no KV cache, no sampler, no logits) and **e2e HW-BLOCKED** (~354 GB checkpoint, ~133 GB/rank on 4x B300 vs 119 GiB unified); bf16 production stream + request planning + the ComfyUI-GGUF arm also landed (535 REAL tensors resolve onto our contract, geometry from shapes alone). **HW verdict CORRECTED: quantized arms FIT (~41 GB in 119 GiB)**, so e2e + speed are reachable; encoder/VAEs/audio VAE DONE (4.2e-9 vs the checkpoint's remote code); NVFP4 layout GATED as identical to ours (speed path is loader wiring); BOTH VAE DECODERS done (audio 4.2e-9, video ViT3D 8.9e-8); video tiling + 3D-CNN encoder (conditioning only) pending; encoder TEXT tower done (1.2e-7); **serving `/v1/videos` DONE and the DEVICE-RESIDENT forward (W2b, f32) LANDED + GPU-VERIFIED on Thor sm_110 at video 1.49e-7 / audio 8.94e-8**; bf16 stream + fusion folds + the FP4 path (needs sm_121a) + a real-checkpoint run pending. **2026-08-05: the AUDIO-VAE ENCODER is ported** (DAC analysis stack + `pre_block` AttnProjection + `mean_proj`, gated stage by stage vs the checkpoint's own remote code at 2.98e-8 / 1.64e-7 / 1.86e-8) with its own checkpoint loader gated on the real 1087-tensor manifest — so **ref2va AUDIO and VIDEO+AUDIO references are now WIRED** (audio rows move by 0.51 / 0.71; a different waveform still moves them by 7.1e-4). Both VAEs are now complete in both directions. **bf16 13-SHARD RELEASE INDEXES 2026-08-07 (`row/H3-BF16-SHARDED-DIT`)**: `MiniMaxH3ShardedCheckpoint` resolves the ORIGINAL 66.3 GB release through its own `model.safetensors.index.json` (a tensor named in the index but missing from its shard throws BY NAME), `EnumerateMiniMaxH3ShardedTensors` feeds the shared shapes-only geometry parser, and `LoadMiniMaxH3DitFromShards` is the host-f32 reference loader. Gated CPU-only at 72/72/54497 (post-rebase): every tensor resolves to the shard the index named AND to the bytes written there, the derived geometry equals the single-file path field for field, and a SPARSE 13-shard release with the REAL 535 tensors at REAL shapes (66.3 GB declared, 144 KB on disk) derives the SHIPPED geometry (50/5376/56/128/14336/24/32/1x2x2/5120). **STREAMS 2026-08-07 (`row/H3-BF16-SHARDED-STREAM`)**: `StreamMiniMaxH3ShardedToDeviceBf16` uploads it one tensor at a time — a BF16 tensor bound for a bf16 device slot goes straight from the mmap with ZERO host buffer, so peak host is bounded by ONE tensor (observed `host_peak=8192`, `direct=37 converted=9`); bit-exact vs the non-streamed `StageMiniMaxH3DitWeights` reference over all 46 views with identical logits, rope.inv_freq host-resident, 73/73/55203. Spec §8.14. **bf16 TEXT ENCODER + THE CONDITIONING NUMBER 2026-08-07 (`row/H3-ENC-BF16-COND-DIFF`)**: the 14-shard 63 GB bf16 Qwen3-VL-32B encoder streams to device too (`StreamMiniMaxH3EncoderShardsToDevice`, q/k/v and gate/up fused ON DEVICE), `--encoder-only` runs the tower alone (peak ~96 -> ~49 GiB by not loading the DiT first), and the widening is gated BIT-IDENTICAL vs an f32-staged tower so the A/B cannot be confounded. MEASURED on Thor over 233 tokens: Q4_K_M vs bf16 conditioning is cos 0.99745 mean / 0.909 min, rel RMS 6.85% excluding the attention sink, median rotation 3.5 deg — same energy as a ONE-WORD prompt edit but DIFFUSE (232/233 tokens rotate vs 172/233). Whether the RENDER changes is NOT established. 75/75/55609. Spec §8.15. This UNBLOCKS the quantization-quality question; no bf16-vs-quant render or speed number is claimed. Spec §8.13. **W-FP4a LANDED (CPU) 2026-08-06 (`row/H3-FP4-SPEED`)**: the device DiT forward now routes the NVFP4 projections through the shared Marlin W4A16 dispatcher (fp4 kept packed; no new quant code), fp4-vs-bf16 wiring gate GREEN (62/62·30039). **W-FP4a GB10 leg LANDED 2026-08-06 (`row/H3-FP4-GPU-E2E`, PR #64):** on sm_121a the Marlin W4A16 path RAN for all 11 projections (`dense_gemms==11` default — VT_MARLIN_DENSE is default-ON → vLLM's own DENSE Marlin GEMM, not the grouped route; `marlin_gemms==11` under VT_MARLIN_DENSE=0; `fallback_gemms==0`), fp4-vs-bf16 BYTE-EXACT (max\|diff\|=0), and the fp4 arm is a MEMORY win not a diffusion-forward speed win (per-forward bf16/fp4 3.47× @seq64 → 0.79–0.83× @seq4224–7040; ~16 vs ~66 GB device). Real-checkpoint fp4-resident t2va e2e RUNS (real 18.75 GB NVFP4 DiT + VAEs + GGUF Qwen3-VL-32B encoder → valid mp4/wav; DiT s/step 5.45/20.0/209 s @512/768/REF-209f) but frames are a non-scene patch-grid at 12/20/50 steps → OPEN render bug (device VAE/denoise). vLLM-Omni has no quantized H3 arm (BF16-only) so any comparison is HW/loader-forced-indirect — spec §8. **ONE-SURFACE ROW 2 LANDED 2026-08-08 (`row/H3-VIDEO-ABI`)**: the whole assembly pipeline is library-owned (`vllm::multimodal::MiniMaxH3VideoEngine`, `minimax_h3_video.cpp`) behind the ABI v12 `vllm_video_*` entry points; `/v1/videos` routes through the SAME seam; `minimax_h3_gen`+`minimax_h3_mux` are thin `vllm.h` clients, frames+WAV byte-identical to the pre-fold binary on the committed fold fixture (`test_minimax_h3_video_fold` 3-arm gate + the v12 `test_capi` section); GB10 real-video re-verify via the v12 ABI = named residual | `MODEL-DIFFUSION-minimax-h3-mini-max-h3-dit` | | ✅ | `LagunaForCausalLM` | Poolside Laguna-S-2.1 (118B/8B MoE) | **LONG-CTX DECODE LEVERS LANDED + MEASURED (2026-08-03, `CLAIM-LAGUNA-LONGCTX-LEVERS`): window-bounded SWA reads (`VT_LAGUNA_SWA_WINDOW`, default-ON, BYTE-EXACT) bound the four `DecodeAttnGqa*` kernels' read to the ~512 sliding window (vLLM `laguna.py:412`) — GB10 A/B token-IDENTICAL `=1` vs `=0` at 520-token context (truncation active), MEASURED −0.30 ms/step at ~2k (~0 at ≤512, grows linearly). bf16 paged KV (`VT_LAGUNA_KV_BF16`, default-OFF opt-in) a distributional near-tie left UNRATIFIED. See BENCHMARKS `CLAIM-LAGUNA-LONGCTX-LEVERS`.** — **NVFP4 W4A4 ARM RAN on GB10 (N4, 2026-08-01, `CLAIM-LAGUNA-NVFP4-N4`): the additive safetensors NVFP4 arm (N1a/N1b/N2/N3 — `Nvfp4Weight` expert fields + `LoadLagunaForCausalLMWeights` + `LqGemmNvfp4Fp4` per-expert TRUE-W4A4 + `LagunaFfnBlock` `fp4` branch + `laguna_gen` dir-autodetect; CPU-gated `test_laguna_nvfp4_loader` 3/3·61, GGUF path byte-identical) generates COHERENTLY on the real 67 GiB `poolside/Laguna-S-2.1-NVFP4`. vs the vLLM MARLIN golden (vLLM's exact prompt ids injected): FIRST 2 TOKENS MATCH exactly, then near-tie divergence (our TRUE-W4A4 fp4-activations vs the MARLIN golden's W4A16 bf16-activations — different precision, EXPECTED; shares golden vocab). SPEED (N5, trace-driven, 2026-08-01): 0.16 → ~4.5 tok/s (~28× THIS SESSION), now ~4× from vLLM 18.8. **Lever #2** (nsys found the bf16 tower running host `MatmulNK` on the CUDA queue): route it to the GPU (`LqGemm` bf16 → `CastBf16` + `MatmulBT`, weight stays bf16) → 6.34 → 0.39 s/tok (16×). **Lever #1** (nsys found the emulation expert GEMM at 92%, GPU 87% busy): the engine's native sm120a fp4 tensor-core MMA (`MatmulNvfp4Fp4Native`) reads the SAME linear scales — it was gated OFF behind `VT_NVFP4_FP4_NATIVE`; default it ON in the driver → 0.39 → ~0.20-0.24 s/tok (~2×). Both coherent + near-tie (byte-identical ids to emulation; first token matches golden). Two GB10 memory fixes landed to run (shard-release + context-before-load). OPEN #234 (remaining ~4×): grouped W4A4 MoE (top_k×3 launches → 3), `ResidentNvfp4`, decode CUDA-graph + on-GPU sampling (the host-orchestration tail). Spec `.agents/specs/laguna-nvfp4-arm-2026-07-31.md` §N4/§N5. The GGUF-Q4_K track (below) is the separate keep-quant vehicle.** Prior **FASTER DECODE (W9, 2026-07-31, `CLAIM-LAGUNA-W9-GROUPED`): the 30 un-grouped per-expert keep-quant GEMV launches/step (top_k × {gate,up,down} `LqGemmRowSlice`) fold onto the SHARED `vt::MatmulBTQuantGrouped` op — per token, Pk experts' gate/up/down each collapse to ONE grouped launch over the already-stacked `[E*N,H]` tower (no loader change). Same-binary A/B on real UD-Q4_K_XL (GB10, `--gpu`, drop_caches cold, 24 tok): grouped (`VT_LAGUNA_GROUPED_MOE=1`, default) == per-expert (`=0`) BYTE-IDENTICAL (md5 `754728c6`, both == W6 golden) + decode 0.18 → 0.13 s/tok (1.38×). Routes through the shared vt op (fold policy). Cumulative with W8: decode 0.66 → 0.13 s/tok (5.1×; 1.5 → 7.7 tok/s; 18× → 3.6× vs llama.cpp 27.8). Next lever: device-resident decode (#1). See spec §W9.** Prior **FASTER DECODE (W8, 2026-07-31, `CLAIM-LAGUNA-W8-EMBED`): `LagunaEmbed` no longer converts the whole 1.23 GB embed table to f32 every token (it gathered T rows out of the whole [Vsz,H] table via `ReadF32` — ~311M host element-converts/token, the DOMINANT decode cost the W7 profile under-filed as "#5"); now gathers only the T needed rows directly (BIT-IDENTICAL — same per-element conversion, same rows). GATED on the real 3-shard UD-Q4_K_XL GGUF (GB10, `--gpu`, W6 cached, drop_caches cold, 24 tok): TOKEN-IDENTICAL to the W5/W6 golden (`22345 83 350 785 …`, coherent " Paris.") + decode 0.66 → 0.17 s/tok = 3.9× (1.5 → 5.9 tok/s; 18× → 4.7× vs llama.cpp 27.8). See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md` §W8. Next: grouped-expert GEMM (=A3) then device-resident decode.** Prior **DECODE-SPEED ATTRIBUTED (W7 profile-only, 2026-07-31, `CLAIM-LAGUNA-W7-SPEED`): `nsys` of the W6 decode (real UD-Q4_K_XL GGUF, GB10) attributes the 0.66 s/tok (~1.5 tok/s vs llama.cpp 27.8 on identical bytes, ~15-18x) to HOST-ORCHESTRATION, not kernel compute — GPU active only 32.7% of the step, 67.3% host/idle; 22,115 `cudaStreamSynchronize` (~2,764/step, zero GPU overlap) from the ~1,795 per-GEMM `DrainQueue` in `LagunaForwardGgufCached` + scalar host glue; 39.4% of GPU time is `QuantizeQ8K` activation-quant (per-GEMM), weight GEMVs un-grouped at ~22% of the 240 GB/s peak (llama.cpp ~76%); no H2D/D2H (unified memory). Ranked levers (all in-tree from ds4): device-resident decode 1.5->~5-7 tok/s, grouped-expert GEMM (`MatmulBTQuantGrouped`) +1.5-2x + dedupes the activation-quant, decode CUDA-graph, tuned MMVQ; + free host cleanups (`LagunaEmbed` copies the whole 1.23 GB embed table/token, per-token RoPE-cache rebuild). Honest reachable ~13-20 tok/s, 27.8 a stretch. NO code changed. See `.agents/specs/laguna-s21-w7-speed-2026-07-31.md`. Prior RUNNABLE + FAST DECODE (W6, 2026-07-31): a per-layer K/V cache + single-token incremental decode replaces W5's O(n²) STATELESS recompute — TOKEN-IDENTICAL (byte-equal ids, md5 `754728c6…` match, == the W5 golden) and 5.05× faster per token: decode 3.33 → 0.66 s/tok on the real UD-Q4_K_XL GGUF (GB10, `--gpu`, keep-quant), same " Paris.…" text. `LagunaKvCache` (mirrors `DeepseekV4KvCache`, MLA-latent → GQA multi-head K/V; caches post-QK-RMSNorm/post-RoPE K + raw V at f32 — bit-exact since RoPE/QK-norm are position-only and attention is causal), MIXED attention per-layer: 12 GLOBAL layers grow unbounded + 36 SLIDING-WINDOW-512 layers EVICT rows beyond the 512 window (gemma2/3 `is_sliding`); `LagunaForwardGgufCached` + shared `LagunaAttention`/`LagunaFfnBlock` helpers used by BOTH forwards (identical float ops; recompute ids unchanged after refactor), `examples/laguna_gen --stateless` A/B flag. No cache bug (bit-exact first run). Next speed = grouped-expert GEMM + device-resident decode (both in-tree from ds4). See `.agents/specs/laguna-s21-w6-2026-07-31.md`. Prior RUNNABLE (W5, 2026-07-31): our engine greedy-generates COHERENT text on the REAL 3-shard UD-Q4_K_XL GGUF (GB10 keep-quant) — "The capital of France is" → " Paris. …", first token "Paris." matches the llama.cpp-Poolside reference. Multi-shard GGUF reader + keep-quant tower (`LoadLagunaFromGgufShards`) + `LagunaForwardGguf` (ds4 keep-quant Gemm/GemmRowSlice) + `examples/laguna_gen`; load 20.6s, peak 71 GiB, 3.27 s/tok stateless recompute (speed=W6).** Prior W3: **W3 REAL forward + 3 new ops landed** (`laguna_ops.cpp`: per-head softplus attn out-gate + ungrouped sigmoid-noaux router + dual per-layer RoPE cos/sin builders; `LagunaModel::Forward` now a REAL runnable host-reference composition — variable-Q-head GQA + dual RoPE + sliding-window mask + softplus gate + dense L0 / ungrouped-MoE L1..47 + untied lm_head — replacing the W1/W2 `VT_CHECK(false)` stub; CPU `-Werror` full-library build clean; `test_laguna_scaffold` **8/8·166** incl. softplus math, router selection+tie-break RED-first, dual-RoPE cos/sin bit-match, variable-Q-head shapes, forward composition on synthetic weights; `test_model_registry` 24/24). W1 oracle DECISION: vLLM native `laguna.py` in pin ⇒ config constructs; dual-oracle = vLLM-NVFP4/-FP8 (fits GB10, BF16 235 GiB does NOT) + llama.cpp-Q4_K token-exact. DEFERRED to W4 (needs 73 GB checkpoint): GGUF keep-quant tower materialization + device/paged production forward + strict dual-oracle greedy gate. ~85-90% reuse (ds4-MoE + gemma-sliding + olmo3-dual-rope + landed Q4_K keep-quant); NEW = the 3 landed host ops + name-map + variable-Q-head device runner. **W4 (2026-07-31, `CLAIM-LAGUNA-W4`, in progress):** the UD-Q4_K_XL GGUF (73.4 GiB, 3 shards) FETCHED to dgx + its metadata/tensor-map READ AUTHORITATIVELY (814 tensors, arch `laguna`, `expert_gating_func=2` sigmoid, `leading_dense_block_count=1`, `expert_weights_scale=2.5`). Three CPU-verified FIDELITY corrections the W1-W3 scaffold got wrong, each grounded in the real GGUF + llama.cpp: (1) **per-head QK-RMSNorm** (`attn_q_norm`/`attn_k_norm` F32[128]) added to params+forward — the scope MISSED it (surfaces only in the tensor map); (2) **dual-RoPE mscale** now uses llama.cpp's `yarn_attn_factor·(1+0.1·ln(factor))` off the GGUF-authoritative `factor=32`/`yarn_attn_factor=1.0` (256K-ctx build, NOT HF's factor-128/1.4852 1M-ctx scalar) — resolves the numerics-delicate residual; (3) **separate** `ffn_gate_exps`/`ffn_up_exps` (Q4_K) + `ffn_down_exps` (Q5_K) + Q8_0 shared/attn (the scaffold assumed merged gate_up). GGUF keep-quant tower materialization (`Mw`/`Sew` mirror of ds4) + keep-quant `ForwardGguf` (vt::MatmulBT/GemmRowSlice) + the real-model greedy run vs the llama.cpp-laguna same-quant oracle remain the W5 close (73 GB single-GB10, host-orchestrated) | `MODEL-TEXT-laguna-laguna-for-causal-lm` | diff --git a/.agents/specs/kimi-linear.md b/.agents/specs/kimi-linear.md index d61a6d725..af91d268e 100644 --- a/.agents/specs/kimi-linear.md +++ b/.agents/specs/kimi-linear.md @@ -1462,6 +1462,103 @@ campaign-completable to production quality; SCOPED here as the named born-on-run --- +## 21. ROW 7 — FOLDED ONTO THE SHARED PAGED RUNNER; engine==CLI 128/128 IDENTITY, golden 122/128 PROFILE PRESERVED (2026-08-07, `row/KIMI-RUNNER-FOLD`, #122) + +The §20.3-scoped production runner fold LANDS (ARCH-ONE-SURFACE ROW 7, task #281): Kimi-Linear's +decode runs THROUGH `ModelRegistry::Forward` on the runner's OWN paged state, the engine/server +serve it at the paged-incremental class of rate, and `examples/kimi_linear_gen` is a thin +public-ABI client (`vllm.h` + `vllm::shared`). + +### The bricks (file:line) +- **B1 — KV enablement** (`src/vllm/transformers_utils/hf_config.cpp`): `LoadHfConfig` synthesizes + `layer_types` + the GDN-group geometry (`linear_num_key/value_heads`, `linear_key/value_head_dim`, + `linear_conv_kernel_dim`) from Kimi's nested `linear_attn_config` (configs/kimi_linear.py:34-148, + 1-indexed `kda_layers`), so the §20.3 runner ABORT (the MambaSpec check against config-derived + {0,0},{0,0,0}) is gone and the per-layer loop allocates 20 KDA state groups + 7 MLA latent pages. + ADDITIVE: explicit-field configs (qwen3_5) never enter the branch; `runner.cpp` UNTOUCHED. +- **B2 — KDA-paged block** (`kimi_linear_device.cpp` `KdaLayerPagedBf16`): `vt::KdaChunkPrefill` for + fresh prefills (vLLM's prompt path; `VT_KIMI_PAGED_KDA_CHUNK=0` A/B) / `vt::KdaGatedDeltaRule` + (T==1) for decode + continuing prefills over the paged `gdn_state` group keyed by + `non_spec_state_indices` (GdnStateGather/Scatter); conv taps via `CausalConv1dFwd` (varlen) / + `CausalConv1dUpdate` (decode) in vLLM's `conv_state.chunk(3)` [q|k|v] row layout. NOT per-head + `GdnBlockPaged` — KDA's per-K-channel decay needs the KDA ops; the shared GDN kernels untouched. +- **B3 — NoPE-MLA-paged block**: latent rows written through `vt::ConcatAndCacheMla` at + `attn_meta.slot_mapping` (bf16 pages — vLLM's cache dtype; the KDA conv cache dtype now also + follows `ResolveKvCacheDType`, mirroring `kda_state_dtype`'s cache-dtype override). TWO arms: + **PRODUCTION = `mla::ForwardMlaAttentionBlock`** — vLLM's ACTUAL absorbed-MQA decode / FA2 + prefill, identity-RoPE (cos=1/sin=0), scale qk^-0.5, load-time `AbsorbKvBProjBf16` into new + `MlaResidentWeights::w_uk_t/w_uv` (`VT_KIMI_PAGED_MLA_FA2`, default ON — GB10-ruled below); the + DIAGNOSTIC arm (`=0`) is the exact f64 softmax island over kv_b-up-projected paged rows, the + CPU fold-identity vehicle. +- **B4 — ONE SURFACE**: the registry loader loads the bf16-RESIDENT tower through the engine + (`ModelFactory::stage_on_load`: queue selected BEFORE the load — CUDA context first + per-tensor + stage-and-release, the §13 recipe; `model_loader.cpp` queue branch, additive); ENG-ASYNC-SCHED W4 + honored (`ForwardPaged` embeds `device_token_ids` — the async device mirror leaves host ids + deliberately stale; missing this was a measured GB10 9/128 divergence, RED-first CPU-pinned); + `vllm_complete_tokens` (ABI v13) — pre-tokenized completion returning generated ids; + `examples/kimi_linear_gen` REWRITTEN as a thin `vllm.h` client; example-abi-allowlist kimi row + REMOVED (merged ratchet 8, with #123's two minimax removals); the CLI-incremental REFERENCE leg preserved as the env-gated + `tests/vllm/models/test_kimi_linear_fold_gate.cpp` (VT_KIMI_MODEL_DIR/VT_KIMI_GOLDEN_DIR). + +### Gates (GB10 dgx.casa, /dev/shm CUDA build — CUTLASS 4.5.0 + FA2 + Triton AOT sm_121a; golden md5 `bfa5bdbf`; flock both locks, drop_caches, worker parked, min-avail ≥ 21G, NO reboot) +- **CPU**: `test_kimi_linear_paged` 8/8·206 — (a) runner allocates Kimi's het-KV groups from a REAL + config.json; (b) paged-runner tokens == CLI tokens (f32 AND production bf16 caches); (b2) + `ForwardPaged` logits BYTE-EQUAL the CLI logits at every step (real GDN builder; mutation-RED on + dropped ssm scatter / zeroed decode state / dropped conv scatter / dropped MLA cache write); (b3) + `device_token_ids` over stale host ids; shared-MLA-arm greedy == exact-arm greedy + f32-page + rejection; batched-prefill distinct KDA slots; (c) 2-request slot isolation. `test_hf_config` + 17/17·180, `test_capi` 35/35·290 (ABI v13 case mutation-verified), full ctest **351/351**. +- **SACRED (re-run AFTER the fold, same build)**: 35B `test_qwen36_paged_engine` **2/2·315 PASS**; + 27B `test_qwen27_paged_engine` **1/1·235 PASS** — the shared GDN/runner path is untouched. +- **Gate A — fold identity (the binding §20.3 gate)**: reference leg (CLI-incremental, + §19-winning config, via `test_kimi_linear_fold_gate`) reproduces §19 EXACTLY — **122/128 @ + 18.93 tok/s** (p7 10/16, got-string byte-equal to §19). Engine leg (thin ABI client → + `vllm_engine_load` + `vllm_complete_tokens`, FA2 arm): **ENGINE == CLI 128/128 BYTE-IDENTICAL** + — p0-p6 16/16 vs golden AND p7's full 16-token got-string equal to the CLI's + (`276,6315,7275,382,2512,2470,387,658,18705,58084,824,2234,397,73874,2366,16626`). vs the + golden: **122/128 — the SAME near-tie profile** (≥122 bound MET, no drop). +- **The FA2-default ruling (measured, 2 arms)**: FA2 arm 122/128 == the golden profile → DEFAULT ON + (vLLM's actual kernels + parity-enablers-ship-as-defaults). The diagnostic exact-island arm + measured **111/128** — the §19-documented GPU M-dimension-tiling near-tie class (re-up-projecting + the whole prefix at M=S vs the CLI's M=T append-time GEMM): p7 flips TOWARD golden (16/16!), p4 + one flip that recovers, p2's token-1 flip cascades 0/16. NOT a paging bug (CPU byte-exact; FA2 + shares every projection + cache write). Kept as the documented diagnostic arm. +- **The async-mirror catch (round 1)**: the first engine run DIVERGED 9/128 — `ForwardPaged` + embedded the host `token_ids` the DEFAULT-ON async device mirror deliberately leaves STALE for + decode rows. Fixed by honoring `device_token_ids` (the qwen3_5 DeviceTokenIdsScope contract); + CPU-pinned RED-first. Models outside qwen3_5/kimi still ignore this field — flagged as a + repo-wide audit residual. +- **SPEED**: the SERVER stream is the cleanest production-surface anchor — **48 tokens / 2.52 s = + 19.0 tok/s wall** (including prefill + first-request warmup ⇒ a LOWER bound on steady decode), + i.e. the fold PRESERVES the §19 paged-incremental class (CLI reference 18.93 reproduced; + **~0.90× the #111 vLLM ~21 floor**). The example's two-length diffs read lower (N=64: 16.9 + async / 16.1 sync; N=16: 9.8-11.5) because their long leg runs FIRST and cold — one-time CUDA + warmup pollutes the subtraction; recorded as a measurement caveat, not a regression. ≥ vLLM ~21 + is still NOT met; the residual levers: device KDA decay gate + beta (kill the per-step host + islands ForwardPaged kept from the CLI), grouped MoE via the shared seam, decode CUDA graph. +- **Tokenizer enablement (server surface)**: Kimi ships tiktoken-only; converted to + `tokenizer.json` via `transformers` `TikTokenConverter` (encode round-trip verified vs the slow + remote-code tokenizer) — staged as `~/kimi-linear-engine-dir` (snapshot symlinks + the converted + tokenizer). A shippable-converter residual is noted. +- **Server smoke (`/v1/completions` through `examples/server`, the ONE-SURFACE deliverable)**: + PASS — model listed (`/v1/models`), STREAMED completion coherent ("The capital of France is" → + " Paris. The …", 48 tokens / 2.52 s = **19.0 tok/s streamed wall** — consistent with the CLI + 18.93 class), non-streamed haiku coherent, greedy `finish_reason: length`, usage populated. +- **vLLM same-session re-measure: ABORTED — BOX REBOOT.** The #111-precedented config (oracle + venv, util 0.82, no tracing) loaded 20/20 shards then hard-rebooted the box at torch.compile/ + graph capture (min-avail had sat at the 15-17G floor) — reproducing §19's measured box-safety + finding. NOT retried per the safety mandate; the denominator remains the #111 recorded ~21 + floor (same prompts/workload). Box recovered clean; worker auto-restored. + +### Status +Row `ACTIVE`: ROW 7 fold LANDED — engine/server surface serves Kimi via the shared paged runner at +122/128-profile fidelity; STRICT remains CLOSED (§20, intrinsic near-tie); the speed residual is +now the last open thread (server surface ~19.0 tok/s wall vs vLLM ~21, ~0.90×; levers: per-step +host-island removal (device decay gate + beta), MoE grouped GEMM through the shared seam, decode +CUDA graph). + +--- + ## Structured contract (machine-readable — mirrors deepseek-v4-flash.md) ## Scope diff --git a/.agents/specs/surface-coverage-2026-08-07.md b/.agents/specs/surface-coverage-2026-08-07.md index aac30e84d..eccf47aee 100644 --- a/.agents/specs/surface-coverage-2026-08-07.md +++ b/.agents/specs/surface-coverage-2026-08-07.md @@ -79,11 +79,15 @@ client. **ROW 1 UPDATE (2026-08-07): `parakeet_transcribe` is the SECOND clean A client** — the Parakeet fold rewrote it against `vllm.h` + `vllm::shared` only, and the ratchet fell 12 -> 11. **ROW 2 UPDATE (2026-08-08): `minimax_h3_gen` and `minimax_h3_mux` are the THIRD and FOURTH clean ABI clients** (the video fold, ABI -v12 `vllm_video_*`), and the ratchet fell 11 -> 9. The remaining 9 reach -`include/vllm/**` / `vt/**` and are transition-tracked in -`scripts/example-abi-allowlist.txt`: - -- Capability drivers: `deepseek_v4_gen`, `laguna_gen`, `kimi_linear_gen`, `server`. +v12 `vllm_video_*`), and the ratchet fell 11 -> 9. **ROW 7 UPDATE (2026-08-07): +`kimi_linear_gen` is the FIFTH clean ABI client** — the Kimi-Linear paged-runner fold +made the fast paged-incremental decode the ENGINE's production path, grew +`vllm_complete_tokens` (ABI v13, pre-tokenized completion returning generated token +ids) and rewrote the example against `vllm.h` + `vllm::shared` only; the ratchet fell +9 -> 8. The remaining 8 reach `include/vllm/**` / `vt/**` and are transition-tracked +in `scripts/example-abi-allowlist.txt`: + +- Capability drivers: `deepseek_v4_gen`, `laguna_gen`, `server`. - Dev/diagnostic (internal-by-nature, folded for consistency): `bench` (via `bench_core.h`), `tokenize`, `dump_container`, `dequant_nvfp4`, `quant_gemm_bench`. - Out of the gated `examples/` tree: `benchmarks/vulkan_gemm_ab.cpp` (Vulkan A/B harness). @@ -91,7 +95,7 @@ v12 `vllm_video_*`), and the ratchet fell 11 -> 9. The remaining 9 reach **Policy (developer-directed 2026-08-07): no permanent exemptions.** Every allowlist entry — drivers AND dev/diagnostic tools — is a transition-tracker pointing at a fold row; the guard fails on any internal include not tracked, and a shrink-only ratchet -(`MAX_INTERNAL_REACHING`, 9 since ROW 2; 11 since ROW 1) means the count can only fall as folds land, never grow to +(`MAX_INTERNAL_REACHING`, 8 since ROW 7; 9 since ROW 2; 11 since ROW 1) means the count can only fall as folds land, never grow to admit a new violation. The public header set is DERIVED from the CMake install rules (exactly `include/vllm.h` today), not hardcoded. The guard catches BOTH breach vectors: a `#include "vllm/..."|"vt/..."|"src/..."` AND a CMake `-I` grant into the internal tree diff --git a/.agents/state.md b/.agents/state.md index b84e96069..200b027af 100644 --- a/.agents/state.md +++ b/.agents/state.md @@ -42333,3 +42333,38 @@ the GB10 speed recipe must move to the seam. (3) Server-arm numeric deltas disclosed above. (4) The CPU host-f32 GGUF arm is off the ABI (keep-quant is the gated arm). (5) /v1/videos job/status/content stay VideoJobStore-served (unchanged); no async-job C-ABI shape yet. + +## 2026-08-08 — ROW 7: Kimi-Linear FOLDED onto the shared paged runner (engine==CLI 128/128; golden 122/128 profile; speed residual named) + +`row/KIMI-RUNNER-FOLD` (#122), ARCH-ONE-SURFACE ROW 7 / kimi-linear.md §20.3→§21, task #281. +The full digits and file:line map live in spec §21; the binding facts: + +- **B1**: `LoadHfConfig` synthesizes `layer_types` + GDN geometry from `linear_attn_config` + (additive; `runner.cpp` untouched) — the §20.3 runner ABORT is gone. +- **B2/B3**: `KimiLinearModel::ForwardPaged` — KDA over the paged `gdn_state` group + (`KdaChunkPrefill` prefill / `KdaGatedDeltaRule` decode, conv in vLLM's chunk(3) layout); + NoPE-MLA latent through `ConcatAndCacheMla` with **`mla::ForwardMlaAttentionBlock` DEFAULT-ON** + (GB10-ruled: that arm reproduces the golden's 122/128 near-tie profile EXACTLY; the diagnostic + f64-island arm measured 111/128 = the §19 GPU M-tiling near-tie class, not a paging bug). +- **B4**: engine loads the bf16-resident tower (`stage_on_load`, §13 recipe); + `vllm_complete_tokens` (ABI v13 after the #123 video-v12 rebase); `kimi_linear_gen` = thin `vllm.h` client; allowlist kimi row removed (merged ratchet 8 with #123's two minimax removals); + CLI reference leg preserved as env-gated `test_kimi_linear_fold_gate`. +- **Gate A**: engine==CLI **128/128 BYTE-IDENTICAL** (p7 got-strings equal); vs golden **122/128** + (>=122 bound MET). SACRED re-run post-fold: 35B 315/315, 27B 235/235. CPU: paged suite 8/8·206 + (logits byte-equal, mutation-verified ×4), ctest 351/351. +- **The async-mirror catch**: round-1 engine leg diverged 9/128 — ForwardPaged embedded the host + ids the DEFAULT-ON async device mirror leaves stale; fixed by honoring `device_token_ids` + (RED-first CPU pin). ★ AUDIT RESIDUAL: models outside qwen3_5/kimi still ignore this field — + any of them served on GB10 through the async engine would hit the same divergence. +- **SPEED (honest)**: server stream **19.0 tok/s wall** (48 tok incl. prefill+warmup — the + production-surface anchor) vs CLI 18.93 (reproduced) vs vLLM ~21 → ~0.90×, >= vLLM NOT met. + The example's two-length diffs (16.9 N=64; 9.8-11.5 N=16) ran the long leg first and cold, so + warmup pollutes the subtraction — measurement caveat recorded. Residual levers: device KDA decay + gate + beta (per-step host islands), grouped MoE via the shared seam, decode graph. +- **Tokenizer**: Kimi ships tiktoken-only; converted to tokenizer.json via transformers + TikTokenConverter (round-trip verified) → `~/kimi-linear-engine-dir` for the engine/server legs; + a shippable-converter residual is noted. +- **vLLM same-session re-measure: ABORTED BY BOX REBOOT** (util 0.82 + torch.compile/graph + capture pushed the unified pool below the floor at 00:25 — reproduces the §19 measured + box-safety violation; NOT retried per the safety mandate). Denominator stays the #111 + recorded ~21 floor. Box recovered clean; worker auto-restored (--restart=always). diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md index 24018a230..acd083f44 100644 --- a/docs/BENCHMARKS.md +++ b/docs/BENCHMARKS.md @@ -319,7 +319,7 @@ built on it rather than keeping the flattering one. | Multimodal image, audio, video | Correctness gated, speed unmeasured | Per-modality speed grids | | `/v1/videos` OpenAI (Sora) shape + ONE-SURFACE fold ROW 2 | **No number owed**: CPU serving-surface changes only; the video fold (ABI v12, seam-routed `/v1/videos`, thin clients) is byte-identical plumbing, no speed claim; server defaults now the ratified recipe (disclosed) | Speed stays the MiniMax-H3 FP4 row below; GB10 re-verify via the v12 ABI = named residual | | Qwen3-dense decode CUDA-graph | Token-exact pass, ~4.3% e2e directional | Steady-state per-step tok/s | -| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | **Paged-incremental (§19) GB10: 18.9 tok/s @ 122/128 = coherent best.** bf16 stream REFUTED (§20/#118: 122→4/128 KDA repeat-loop, no speed win); STRICT unreachable (§14-§20 levers closed), p7 near-tie | vLLM ~21 (16-tok aggregate floor); ours **0.90× vLLM**. Last 0.10× + STRICT need vLLM's real kernels via the SERVER fold (scoped) | +| Kimi-Linear-48B-A3B (KDA+MLA+MoE) | **RUNNER FOLD LANDS (ROW 7, §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (near-tie profile); FA2 MLA default-ON; SACRED green.** Server 19.0 tok/s wall; CLI 18.93 reproduced | vLLM ~21 (#111 floor; in-session re-measure ABORTED by GB10 reboot at util 0.82, §21): **~0.90×**, >= vLLM NOT met; residual = KDA host islands + grouped MoE + decode graph | | vLLM 0.26 re-benchmark | Pending | Re-run the binding grids on the advanced pin | | MiniMax-H3 FP4 speed (W-FP4a) | **Measured GB10 (`row/H3-FP4-GPU-E2E`).** Marlin W4A16 byte-exact vs bf16; fp4 a memory win, 0.8x bf16/forward. Real-ckpt fp4-resident e2e RUNS (mp4/wav) | fp4 speed CLOSED. bf16-vs-quant A/B: ENCODER half MEASURED (§8.15), DiT half NOT (no bf16 render exists). Detail: benchmark-record + spec §8 | | MiniMax-H3 render coherence (`row/H3-RENDER-CLOSE` #77) | **CLOSED: a COHERENT scene on GB10.** #70/#74 white was wrong-PARTITION usage (t2va on the ref2va ckpt); t2va on the FL2VA GGUF renders a prompt-matched orange cat (adj-cos 0.95 vs 0.06, no patch-grid) | Verified first: t2va inputs byte-exact vs upstream; CUDA device==host at seq 1920. Follow-up `H3-TASK-PARTITION-GUARD`: the task/partition mismatch now RAISES 1:1 with `_resolve_task` (spec §8.6-8.7) | diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index 716554eeb..1558ba67f 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -71,6 +71,8 @@ portable/reference path. In normal operation leave them unset. | `VT_KIMI_DEVICE_KDA` | off (opt-in) | `=1` runs the Kimi-Linear KDA per-k-channel gated-delta RECURRENCE through the net-new device op `vt::KdaGatedDeltaRule` (`cuda_gdn.cu` `KdaScanKernel`, the per-channel-decay `g[T,H,D]` variant of GDN's per-head-scalar decay; FLA `fused_recurrent_gated_delta_rule_fwd_kernel` IS_KDA=True) instead of the f64 host recompute. The principled path to STRICT AND the speed lever (spec §14/§15): the recurrence runs vLLM's actual f32-on-bf16 arithmetic on device rather than a host f64 recompute that is more precise than vLLM and coin-flips near-ties. Requires `VT_KIMI_DEVICE_COMPUTE=1`; the elementwise decay gate + beta stay host. Unit-gated (`test_ops_kda_recurrence`: broadcast-g == `vt::GdnPrefill` bit-identical, distinct-per-channel vs f64 ref, CPU↔CUDA parity). Default OFF (parity-enabler: flip only with the token gate green) → byte-identical | | `VT_KIMI_DEVICE_KDA_CHUNK` | off (opt-in) | `=1` processes the PROMPT-length KDA with the CHUNKED prefill family (`vt::KdaChunkPrefill`: the vendored FLA Triton-AOT cubins `kda_gate_cumsum` -> `kkt` -> `solve_tril` -> `recompute_w_u` -> `chunk_delta_h` -> `chunk_gla_o`) instead of the RECURRENT form, as vLLM does (`kimi_gdn_linear_attn.py:141`); decode stays recurrent. Spec §15 STRICT residual (c): vLLM processes the prompt chunked and we still recur, and the different reduction ORDER coin-flips the p7 near-tie. Requires `VT_KIMI_DEVICE_KDA=1`. Default OFF (parity-enabler; flip ON only with the token gate green) | | `VT_KDA_CHUNK_TRITON` | **on** (default) | `=0` restores the recurrent form inside the KDA chunk-prefill op, bypassing the Triton-AOT cubin path when the geometry is otherwise eligible (`cuda_gdn.cu`). An A/B escape hatch for isolating the AOT kernels from the recurrence; ON is the shipped behaviour | +| `VT_KIMI_PAGED_KDA_CHUNK` | on | `=0` disables the CHUNKED KDA prefill (`vt::KdaChunkPrefill`, vLLM's prompt path `kimi_gdn_linear_attn.py:141`) inside the ROW 7 shared-paged-runner forward (`KimiLinearModel::ForwardPaged`) and falls back to the recurrence for fresh prefill requests — the fold-identity A/B. Decode and continuing (`has_initial_state`) prefills always use the recurrence, exactly as vLLM. Default ON is the §19 Gate-A-winning regime | +| `VT_KIMI_PAGED_MLA_FA2` | **on** | Default ON (GB10-ruled 2026-08-07, kimi-linear.md §21): the ROW 7 paged-runner forward routes the 7 NoPE-MLA layers through `mla::ForwardMlaAttentionBlock` (vLLM's ACTUAL absorbed-MQA decode / FA2 prefill over the paged latent cache, identity-RoPE, scale qk^-0.5, load-time `AbsorbKvBProjBf16`) — measured on the real 48.9B it reproduces the golden's 122/128 near-tie profile EXACTLY (p0-p6 16/16, p7 10/16, got-string byte-equal to the CLI reference). `=0` selects the DIAGNOSTIC exact f64-softmax-island arm (the CPU fold-identity vehicle; GB10 111/128 = the §19 GPU M-tiling near-tie regime, recorded, not a paging bug) | | `VT_KIMI_DEVICE_MLA` | off (opt-in) | `=1` routes the Kimi-Linear 7 NoPE-MLA layers' causal-softmax core through the shared device op `vt::Attention` (f32 online max-subtracted softmax, value zero-padded qk_nope+qk_rope=192 vs v=128) instead of the f64 host `MlaSoftmaxIsland`. Requires `VT_KIMI_DEVICE_COMPUTE=1`. W7-speed STRICT-lever A/B (spec §16, #107), **MEASURED NEGATIVE** — `vt::Attention`'s f32 online-softmax is the right math but a DIFFERENT reduction order than vLLM's FA2, so it coin-flips near-ties (full 48.9B gate 122→109/128 AND 4.24→3.89 tok/s). Kept as a documented-MEASURED-NEGATIVE A/B knob; the STRICT path is the ACTUAL paged FA2 `mla::ForwardMlaAttentionBlock`, not this approximation. Default OFF → byte-identical | | `VT_KIMI_DEVICE_KDA_CHUNK` | off (opt-in) | `=1` processes the PROMPT-length Kimi-Linear KDA through the CHUNKED prefill kernel family (`vt::KdaChunkPrefill`: the vendored FLA Triton-AOT cubins `kda_gate_cumsum` → `kkt` → `solve_tril` → `recompute_w_u` → `chunk_delta_h` → `chunk_gla_o`) instead of the recurrence, exactly mirroring vLLM's prefill=chunk / decode=recurrent split (`kimi_gdn_linear_attn.py:141`). Requires `VT_KIMI_DEVICE_KDA=1`; decode (query_len==1) stays the recurrence. In the O(n²) recompute vehicle applying chunk to every step REGRESSED 122→102/128 (spec §18, #111) — the chunk is the vLLM-parity PREFILL half of the paged-incremental decode (spec §19: chunk-prefill ONCE then recurrent-decode the persistent state), where it carries vLLM's exact prompt-order arithmetic. Default OFF (parity-enabler) → the recurrence path is byte-identical | | `VT_WHISPER_ENC_EAGER` | off (flash-tiled attention on) | Set to `1` to force the naive per-key block-reduction attention in the Voxtral/Whisper audio encoder instead of the default flash-tiled kernel. Rollback / A-B knob; token-identical to the default path | diff --git a/docs/FEATURES.md b/docs/FEATURES.md index fe29b6124..21694513b 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -119,7 +119,7 @@ speed-pending, which [BENCHMARKS.md](BENCHMARKS.md) tracks. | `Glm4ForCausalLM` | GLM-4-9B-0414 | near-tie 16/16 vs vLLM 0.25.0 | pending | | `Glm4MoeLiteForCausalLM` | zai-org/GLM-4.7-Flash (31.2B, MLA MoE) | near-tie 8/8 vs vLLM 0.25.0 | pending | | `LagunaForCausalLM` | poolside/Laguna-S-2.1-NVFP4, GGUF-Q4_K, Laguna-XS | byte-exact near-tie (distributional vs vLLM) | vLLM parity+ 1.03x, default on, via the `laguna-gen` CLI; the registered engine forward VT_CHECKs non-bf16 (`ARCH-ONE-SURFACE` fold) | -| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Paged-incremental decode (§19) GB10: 18.9 tok/s (0.90× vLLM) @ 122/128 = coherent best**; STRICT unreachable (bf16 stream REFUTED §20/#118 122→4/128; p7 intrinsic near-tie) | CLI opt-in `--incremental`; SERVER fold scoped (ARCH-ONE-SURFACE req 4; runner aborts on Kimi KV) | +| `KimiLinearForCausalLM` | Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE) | **Folded onto the shared paged runner (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; vs golden 122/128 (the intrinsic near-tie profile); FA2 paged MLA default-ON; SACRED post-fold green** | Served via `vllm_engine_load` + `vllm_complete_tokens` (ABI v13); server 19.0 tok/s wall vs vLLM ~21 (~0.90×), speed residual open | | `KimiK3ForConditionalGeneration` | Kimi-K3 (2.8T MoE) | scaffold: registry+config+enumeration gated, forward refuses | HW-infeasible (~1.56 TB); no run | | `ParakeetForCTC`, `ParakeetForRNNT`, `ParakeetForTDT` | nvidia/parakeet-ctc-0.6b/-1.1b, -rnnt-0.6b, -tdt-0.6b-v3 (transcribed, ids exact vs HF `generate()`, P4/P6 2026-08-07; not retained) + committed synthetic fold fixture | ASR transcription-only (`SupportsTranscription` mirror; text paths refuse by task); fold gate byte-identical to the pre-refactor pipeline | n/a (CPU correctness-grade ASR via `vllm_transcribe` + `/v1/audio/transcriptions`) | | `CohereForCausalLM` | Command-R / Cohere (and Cohere2) | scaffold: W0 tiny-random oracle run-verified; real-checkpoint gate blocked | no run | @@ -237,6 +237,7 @@ Build with `-DVLLM_CPP_VULKAN=ON`; off by default. | Capability | C-ABI surface | Embedder-reachable | |---|---|---| | Text completion (blocking + streaming) | `vllm_complete`, `vllm_complete_stream` | reachable | +| Pre-tokenized completion (token-id prompts, ABI v13) | `vllm_complete_tokens` | reachable | | OpenAI chat (tools, streaming) | `vllm_chat`, `vllm_chat_stream` | reachable | | Async request submission | `vllm_request_submit` | reachable | | Structured output / grammars | `structured_json`, `structured_grammar` | reachable | @@ -270,7 +271,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Gap | State | Detail | |---|---|---| -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Paged-incremental (§19) GB10: 18.9 tok/s (0.90× vLLM) @ 122/128 = coherent best**; STRICT unreachable: bf16 stream REFUTED (§20/#118: 122→4/128), §14-§20 all levers closed, p7 near-tie | speed lever LANDS (0.90×); STRICT closed as near-tie; SERVER fold scoped (ARCH-ONE-SURFACE req 4) | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Runner fold LANDS (ROW 7 §21, #122): the ENGINE/SERVER surface serves Kimi at the 122/128 golden profile (engine==CLI 128/128); STRICT stays closed (intrinsic p7 near-tie)** | server 19.0 tok/s wall / CLI 18.9 vs vLLM ~21 (~0.90×), speed residual named (§21) | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Architecturally blocked | Vision tower lives outside the registered engine forward | diff --git a/docs/STATUS.md b/docs/STATUS.md index ffef39836..d0b72bde2 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -75,7 +75,7 @@ token-for-token correctness against the pinned oracle. | DeepSeek-V2 MLA | Correctness-complete, speed-pending | Token-exact 8/8 (DeepSeek-V2-Lite); 0.86-0.95x output rate, TTFT faster at c4/c8. A2+A5 MLA norm-rope fold default-ON (`VT_MLA_FUSED_NORM_ROPE`, bit-exact rollback, SACRED 8/8 unchanged; forensics in benchmark-record) — kimi_k3/kimi-linear inherit it | | GLM-4 dense (sandwich norms, partial rope) | Correctness-complete, speed-pending | Token-exact 16/16 (GLM-4-9B-0414); first GLM-family model; partial interleaved RoPE + Gemma2 sandwich norms + biased qkv | | GLM-4.7-Flash (MLA + GLM MoE) | Correctness-complete, speed-pending | Token-exact 8/8 (GLM-4.7-Flash, 31.2B); reuses the DeepSeek-V2 MLA stack; first e2e coverage of the q_lora query branch + noaux_tc sigmoid router with routed-scaling | -| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **Paged-incremental (§19) GB10: 18.9 tok/s (0.90× vLLM) @ 122/128 = coherent best.** STRICT unreachable — bf16 stream REFUTED (§20/#118: 122→4/128), §14-§20 levers closed, p7 near-tie | `test_kimi_linear_forward` **15/15·875** (knob OFF); GB10 control 122 (3×) vs +bf16 4. SERVER fold scoped (ARCH-ONE-SURFACE req 4; `runner.cpp:489`) | +| Kimi-Linear-48B-A3B (KDA + NoPE-MLA + MoE hybrid) | **RUNNER FOLD LANDS (ROW 7 §21, #122): engine==CLI 128/128 byte-identical; golden 122/128 (near-tie profile); FA2 MLA default-ON; `vllm_complete_tokens` (ABI v13).** STRICT stays CLOSED. Server 19.0 tok/s wall (~0.90× vLLM floor) = speed open | paged suite 8/8·206; SACRED post-fold 35B 315/315 + 27B 235/235; thin ABI client (ratchet 8) | | Gemma-3 dense (GeGLU, dual rope, sandwich norms) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-3-1b-it); first Gemma-family model; GeGLU (gelu_pytorch_tanh) + dual per-layer RoPE theta + Gemma-RMSNorm sandwich norms + sqrt(hidden) embed-scale + query_pre_attn_scalar scaling | | Gemma-2 dense (attn + final logit soft-cap) | Correctness-complete, speed-pending | Near-tie-band 48/48 (gemma-2-2b-it): 44/48 strict on vLLM's greedy + 4/48 at 0.0-nat ties in vLLM's own logits; proves the attention + final logit soft-cap primitives (attn_logit_softcapping 50 + final 30); the inverse of Gemma-3 (both soft-caps, no QK-norm) | | Gemma-1 dense (the original Gemma) | Correctness-complete, speed-pending | STRICT token-exact 48/48 greedy (gemma-2b); two fused norms/layer, head_dim scale, GeGLU + sqrt(hidden) embed-scale, tied lm_head; no soft-cap/QK-norm/sliding. **D1 (2026-07-31): the whole Gemma family (1/2/3/4) folded to the default-ON bf16 merged-QKV descriptor (`MergedQkvEnabled`); re-gated Gemma-2 SACRED 48/48 (global+sliding) + Gemma-4 STRICT 32/32 — its existing gate held** | diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index 5de59e4ac..f0c6f9ed8 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -22,11 +22,12 @@ add_executable(laguna-gen laguna_gen/main.cpp) target_link_libraries(laguna-gen PRIVATE vllm::vllm) vllm_cpp_set_warnings(laguna-gen) -# kimi-linear-gen: the single-GB10 Kimi-Linear-48B-A3B full-model e2e harness (§13 -# bf16-resident loader/forward — greedy decode the 8-prompt golden battery through -# the bf16 ForwardDeviceCompute and token-compare to the STRICT oracle golden). +# kimi-linear-gen: the Kimi-Linear-48B-A3B golden token battery, a THIN CLIENT +# of the public C ABI (ONE SURFACE / ROW 7 §20.3 B4): vllm_engine_load + the +# v13 vllm_complete_tokens over the pre-tokenized golden prompts. Links the +# packaged shared library like vllm-cli — no internal headers. add_executable(kimi-linear-gen kimi_linear_gen/main.cpp) -target_link_libraries(kimi-linear-gen PRIVATE vllm::vllm) +target_link_libraries(kimi-linear-gen PRIVATE vllm::shared) vllm_cpp_set_warnings(kimi-linear-gen) # vllm-bench (M2.1): the throughput/latency benchmark harness — the gate-#1 diff --git a/examples/kimi_linear_gen/main.cpp b/examples/kimi_linear_gen/main.cpp index 9a6b3248c..4f41537d0 100644 --- a/examples/kimi_linear_gen/main.cpp +++ b/examples/kimi_linear_gen/main.cpp @@ -1,76 +1,37 @@ -// kimi-linear-gen — the §13 full-model e2e harness for Kimi-Linear-48B-A3B on one -// GB10. It loads the bf16-RESIDENT weights (LoadKimiLinearResidentBf16Weights — NEVER -// the 183 GiB f32 MaterializeHost; the pool-math path stages each large matmul weight -// to cudaMalloc'd d_dev and ReleaseHost's its host mirror, so the LOAD peak holds only -// one tensor's host bytes on top of the growing ~91.5 GiB device residency) and greedy- -// decodes the §12 8-prompt battery x N tokens through the bf16 ForwardDeviceCompute -// (VT_KIMI_DEVICE_COMPUTE arm), comparing token-exact to the STRICT oracle golden -// (tests/parity/goldens/kimi_linear_greedy/greedy_ids.npy). +// kimi-linear-gen — THIN PUBLIC-ABI CLIENT (ONE SURFACE / ROW 7, §20.3 B4). // -// GB10 load recipe (context-first + shard-release, mirror examples/laguna_gen/ -// main.cpp:185-237): create the CUDA context BEFORE loading weights (the driver's -// reservation would OOM against the page-cache pressure of a post-load reservation), -// then drop the mmap'd shards after the memcpy loader. +// The Kimi-Linear greedy token battery against the STRICT oracle golden, driven +// ENTIRELY through the flat C ABI (include/vllm.h): vllm_engine_load builds the +// full engine (the bf16-resident §13 loader + the shared paged runner the fold +// landed — KDA state in the MambaSpec group, NoPE-MLA latent in the paged MLA +// group), and vllm_complete_tokens (ABI v13) generates from the golden's +// pre-tokenized prompts. The former private harness (bf16-resident loader + +// KimiDecodeCache incremental decode driven through internal headers) is gone: +// the fast paged-incremental decode IS the engine's production path now, so +// this example is exactly what an embedder with only libvllm + vllm.h can do. // -// kimi-linear-gen --model --golden -// [--gpu] [--steps N] [--prompts M] [--load-only] +// kimi-linear-gen --model [--golden ] [--steps N] +// [--prompts M] [--load-only] +// +// The golden dir layout matches the §12 capture: p{i}_prompt.i32 (raw LE int32 +// token ids) + greedy_ids.npy ([P,T] +#include #include #include #include #include -#include #include #include #include #include -#include "vllm/model_executor/model_loader/safetensors_reader.h" -#include "vllm/model_executor/models/kimi_linear.h" -#include "vllm/transformers_utils/hf_config.h" -#include "vt/backend.h" -#include "vt/device.h" +#include "vllm.h" namespace fs = std::filesystem; namespace { -double CurResidentGiB() { - std::ifstream f("/proc/self/status"); - std::string line; - while (std::getline(f, line)) - if (line.rfind("VmRSS:", 0) == 0) { - long kb = 0; - std::sscanf(line.c_str() + 6, "%ld", &kb); - return static_cast(kb) / (1024.0 * 1024.0); - } - return 0.0; -} -double PeakResidentGiB() { - std::ifstream f("/proc/self/status"); - std::string line; - while (std::getline(f, line)) - if (line.rfind("VmHWM:", 0) == 0) { - long kb = 0; - std::sscanf(line.c_str() + 6, "%ld", &kb); - return static_cast(kb) / (1024.0 * 1024.0); - } - return 0.0; -} - -std::vector OpenSafetensorsDir(const std::string& dir) { - std::vector paths; - for (const auto& e : fs::directory_iterator(dir)) - if (e.is_regular_file() && e.path().extension() == ".safetensors") - paths.push_back(e.path().string()); - if (paths.empty()) throw std::runtime_error("no *.safetensors shards in " + dir); - std::sort(paths.begin(), paths.end()); - std::vector shards; - shards.reserve(paths.size()); - for (const std::string& p : paths) shards.push_back(vllm::SafetensorsFile::Open(p)); - return shards; -} - // Read a raw little-endian int32 file (the golden p{i}_prompt.i32 prompts). std::vector ReadI32File(const std::string& path) { std::ifstream f(path, std::ios::binary); @@ -102,7 +63,6 @@ NpyInts ReadNpyInts(const std::string& path) { const bool i8 = header.find(" [--golden ] [--gpu] " - "[--steps N] [--prompts M] [--load-only]\n"); + "usage: --model [--golden ] [--steps N] " + "[--prompts M] [--load-only]\n"); return 2; } - // Create the CUDA context BEFORE loading weights (GB10 load recipe). - vt::Backend* gpu_backend = nullptr; - vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; - if (use_gpu) { - gpu_backend = &vt::GetBackend(vt::DeviceType::kCUDA); - q = gpu_backend->CreateQueue(); - std::fprintf(stderr, "[kimi] GPU context reserved on CUDA device %d (before load)\n", - q.device.index); - } else { - std::fprintf(stderr, "[kimi] CPU queue (won't fit the full model — smoke only)\n"); - } - - const std::string config_path = (fs::path(model) / "config.json").string(); - std::fprintf(stderr, "[kimi] config %s\n", config_path.c_str()); - const vllm::HfConfig config = vllm::LoadHfConfig(config_path); - std::vector shards = OpenSafetensorsDir(model); - std::fprintf(stderr, - "[kimi] %zu safetensors shard(s); loading bf16-resident tower " - "(RSS before %.1f GiB)...\n", - shards.size(), CurResidentGiB()); + std::fprintf(stderr, "[kimi] libvllm %s (ABI %d, header %d)\n", vllm_version(), + vllm_abi_version(), VLLM_ABI_VERSION); + vllm_model_params mp = vllm_model_params_default(); + mp.model_path = model.c_str(); + mp.max_model_len = max_model_len; + vllm_engine* eng = nullptr; const auto t0 = std::chrono::steady_clock::now(); - vllm::KimiLinearWeights w = vllm::LoadKimiLinearResidentBf16Weights( - shards, config, use_gpu ? &q : nullptr); + const vllm_status lst = vllm_engine_load(&mp, &eng); const auto t1 = std::chrono::steady_clock::now(); - std::fprintf(stderr, - "[kimi] LOADED bf16-resident: layers=%lld experts=%lld vocab=%lld | " - "load %.1fs | RSS %.1f GiB PEAK %.1f GiB\n", - (long long)w.params.num_hidden_layers, (long long)w.params.num_experts, - (long long)w.params.vocab_size, - std::chrono::duration(t1 - t0).count(), CurResidentGiB(), - PeakResidentGiB()); - - // Drop the mmap'd shards (the loader copied every tensor into owned buffers). - const size_t n_shards = shards.size(); - shards.clear(); - shards.shrink_to_fit(); - std::fprintf(stderr, "[kimi] released %zu mmap'd shard(s); RSS %.1f GiB\n", n_shards, - CurResidentGiB()); - if (load_only) return 0; - - // Greedy decode + compare to the STRICT golden. - const int64_t V = w.params.vocab_size; - vllm::v1::CommonAttentionMetadata attn_meta{}; - std::vector attn_kv; - vt::Backend& be = vt::GetBackend(q.device.type); + if (lst != VLLM_OK) { + std::fprintf(stderr, "[kimi] engine load FAILED: %s\n", vllm_last_error()); + return 1; + } + std::fprintf(stderr, "[kimi] engine loaded in %.1fs\n", + std::chrono::duration(t1 - t0).count()); + if (load_only) { + vllm_engine_free(eng); + return 0; + } NpyInts gold; - bool have_golden = !golden.empty(); + const bool have_golden = !golden.empty(); if (have_golden) { gold = ReadNpyInts((fs::path(golden) / "greedy_ids.npy").string()); std::fprintf(stderr, "[kimi] golden greedy_ids shape [%lld,%lld]\n", @@ -214,78 +152,59 @@ int main(int argc, char** argv) { (long long)(gold.shape.size() > 1 ? gold.shape[1] : 0)); } + vllm_sampling_params sp = vllm_sampling_params_default(); + sp.temperature = 0.0f; // greedy + sp.max_tokens = steps; + sp.ignore_eos = 1; + int total = 0, matched = 0; - double first_tok_s = 0.0; - int steady_steps = 0; - double steady_s = 0.0; + double first_prompt_full_s = 0.0, first_prompt_one_s = 0.0; for (int pi = 0; pi < prompts; ++pi) { std::vector prompt; if (have_golden) { const std::string pp = (fs::path(golden) / ("p" + std::to_string(pi) + "_prompt.i32")).string(); - prompt = ReadI32File(pp); + try { prompt = ReadI32File(pp); } catch (const std::exception& e) { + std::fprintf(stderr, "[kimi] prompt %d: %s — skip\n", pi, e.what()); + continue; + } } if (prompt.empty()) { std::fprintf(stderr, "[kimi] prompt %d empty, skip\n", pi); continue; } - std::vector seq = prompt; - std::vector gen; - auto argmax_row = [&](const vllm::ForwardLogits& fl) { - std::vector row(static_cast(V)); - be.Copy(q, row.data(), fl.device_tensor.data, row.size() * sizeof(float)); - be.Synchronize(q); - int best = 0; - float bv = row[0]; - for (int64_t o = 1; o < V; ++o) - if (row[static_cast(o)] > bv) { bv = row[static_cast(o)]; best = static_cast(o); } - return best; - }; - if (incremental) { - // §18 paged-incremental: PREFILL the prompt ONCE (fills the cache), then decode - // one token per step from the CARRIED state — no O(n²) recompute. - vllm::KimiDecodeCache cache; - std::vector positions(prompt.size()); - for (size_t t = 0; t < prompt.size(); ++t) positions[t] = static_cast(t); - const std::vector li = {static_cast(prompt.size() - 1)}; - const auto tp0 = std::chrono::steady_clock::now(); - vllm::ForwardLogits fl = vllm::KimiLinearModel::ForwardPrefillIncremental( - prompt, positions, w, q, cache, li); - int best = argmax_row(fl); - const auto tp1 = std::chrono::steady_clock::now(); - if (pi == 0) first_tok_s = std::chrono::duration(tp1 - tp0).count(); - gen.push_back(best); - for (int s = 1; s < steps; ++s) { - const auto ts = std::chrono::steady_clock::now(); - vllm::ForwardLogits d = vllm::KimiLinearModel::ForwardDecodeStepIncremental( - best, cache.seq_len, w, q, cache); - best = argmax_row(d); - const auto te = std::chrono::steady_clock::now(); - steady_s += std::chrono::duration(te - ts).count(); - ++steady_steps; - gen.push_back(best); - } - } else { - for (int s = 0; s < steps; ++s) { - std::vector positions(seq.size()); - for (size_t t = 0; t < seq.size(); ++t) positions[t] = static_cast(t); - const std::vector li = {static_cast(seq.size() - 1)}; - const auto ts = std::chrono::steady_clock::now(); - vllm::ForwardLogits fl = vllm::KimiLinearModel::ForwardDeviceCompute( - seq, positions, attn_meta, attn_kv, w, q, li); - int best = argmax_row(fl); - const auto te = std::chrono::steady_clock::now(); - const double dt = std::chrono::duration(te - ts).count(); - if (pi == 0 && s == 0) first_tok_s = dt; - else { steady_s += dt; ++steady_steps; } - gen.push_back(best); - seq.push_back(best); + std::vector gen(static_cast(steps), 0); + int32_t n_gen = 0; + const auto ts = std::chrono::steady_clock::now(); + const vllm_status st = vllm_complete_tokens( + eng, prompt.data(), static_cast(prompt.size()), &sp, gen.data(), + static_cast(gen.size()), &n_gen, nullptr); + const auto te = std::chrono::steady_clock::now(); + if (st != VLLM_OK) { + std::fprintf(stderr, "[kimi] prompt %d FAILED: %s\n", pi, vllm_last_error()); + vllm_engine_free(eng); + return 1; + } + if (pi == 0) { + first_prompt_full_s = std::chrono::duration(te - ts).count(); + // Two-length diff for the steady decode rate: re-run the same prompt at + // max_tokens=1 and subtract (prefix caching is off by default on the + // hybrid archs, so both runs pay the same prefill). + vllm_sampling_params sp1 = sp; + sp1.max_tokens = 1; + int32_t one_tok = 0; + int32_t n_one = 0; + const auto u0 = std::chrono::steady_clock::now(); + if (vllm_complete_tokens(eng, prompt.data(), + static_cast(prompt.size()), &sp1, &one_tok, + 1, &n_one, nullptr) == VLLM_OK) { + const auto u1 = std::chrono::steady_clock::now(); + first_prompt_one_s = std::chrono::duration(u1 - u0).count(); } } - // compare to golden row pi. if (have_golden && pi < (gold.shape.empty() ? 0 : static_cast(gold.shape[0]))) { const int64_t T = gold.shape.size() > 1 ? gold.shape[1] : 0; int row_match = 0; - const int n = static_cast(std::min(T, steps)); + const int n = static_cast(std::min(T, n_gen)); std::string got, exp; for (int t = 0; t < n; ++t) { const int64_t g = gold.data[static_cast(pi) * T + t]; @@ -295,13 +214,15 @@ int main(int argc, char** argv) { ++total; if (static_cast(o) == g) { ++matched; ++row_match; } } - std::fprintf(stderr, "[kimi] prompt %d: %d/%d tokens match golden\n", pi, row_match, n); - if (row_match != n) { - std::fprintf(stderr, " got: %s\n exp: %s\n", got.c_str(), exp.c_str()); - } + std::fprintf(stderr, "[kimi] prompt %d: %d/%d tokens match golden\n", pi, + row_match, n); + if (row_match != n) + std::fprintf(stderr, " got: %s\n exp: %s\n", got.c_str(), + exp.c_str()); } else { std::string got; - for (int t = 0; t < steps; ++t) got += std::to_string(gen[static_cast(t)]) + ","; + for (int t = 0; t < n_gen; ++t) + got += std::to_string(gen[static_cast(t)]) + ","; std::fprintf(stderr, "[kimi] prompt %d gen: %s\n", pi, got.c_str()); } } @@ -309,10 +230,15 @@ int main(int argc, char** argv) { if (have_golden) std::fprintf(stderr, "\n[kimi] TOKEN MATCH: %d/%d (%s)\n", matched, total, matched == total ? "STRICT PASS" : "DIVERGENCE"); - if (steady_steps > 0) - std::fprintf(stderr, "[kimi] first-step %.3fs | steady %.3f s/step (%.2f tok/s) over %d steps\n", - first_tok_s, steady_s / steady_steps, steady_steps / steady_s, steady_steps); - std::fprintf(stderr, "[kimi] RSS %.1f GiB PEAK %.1f GiB\n", CurResidentGiB(), - PeakResidentGiB()); + if (first_prompt_full_s > 0.0 && first_prompt_one_s > 0.0 && steps > 1) { + const double steady = (first_prompt_full_s - first_prompt_one_s) / + static_cast(steps - 1); + std::fprintf(stderr, + "[kimi] p0 wall %.3fs (N=%d) vs %.3fs (N=1) => steady %.3f " + "s/tok (%.2f tok/s, two-length diff)\n", + first_prompt_full_s, steps, first_prompt_one_s, steady, + steady > 0 ? 1.0 / steady : 0.0); + } + vllm_engine_free(eng); return (have_golden && matched != total) ? 1 : 0; } diff --git a/include/vllm.h b/include/vllm.h index 7d3c415c3..1880348e3 100644 --- a/include/vllm.h +++ b/include/vllm.h @@ -102,8 +102,15 @@ extern "C" { * points on such a handle report VLLM_ERR_INVALID_ARGUMENT with an actionable * message instead of serving, and vllm_transcribe on a TEXT handle does the * same. A pre-v11 caller that never loads a Parakeet directory is - * byte-identical. */ -#define VLLM_ABI_VERSION 12 + * byte-identical. + * v13: vllm_complete_tokens — blocking completion from a PRE-TOKENIZED prompt + * (vLLM's TokensPrompt), returning the generated token ids (and optionally the + * detokenized vllm_completion). The entry point for embedders that manage + * their own tokenization and for token-exact gates/benchmarks that compare + * whole token streams against a reference (the Kimi-Linear paged-runner fold + * battery, ARCH-ONE-SURFACE ROW 7, is the first consumer). Purely additive — + * no struct changed. */ +#define VLLM_ABI_VERSION 13 /* ── Export macro ───────────────────────────────────────────────────────────── * Marks the symbols that make up the stable ABI. Default visibility now; Task 3 @@ -370,6 +377,27 @@ VLLM_API vllm_status vllm_complete(vllm_engine* engine, const char* prompt, const vllm_sampling_params* params, vllm_completion* out); +/* ── Pre-tokenized completion (ABI v13) ─────────────────────────────────────── + * Run a single blocking completion for a PRE-TOKENIZED prompt (vLLM's + * TokensPrompt): tokenization is skipped and generation starts from + * `prompt_tokens` directly. + * - prompt_tokens / n_prompt_tokens: the prompt token ids, BORROWED for the + * duration of the call. n_prompt_tokens must be > 0. + * - out_tokens / max_out_tokens: caller-owned buffer that receives the + * GENERATED token ids; *n_out_tokens is set to the number written + * (<= max_out_tokens). Generation length is bounded by params->max_tokens + * as usual — a smaller buffer only truncates what is REPORTED, never the + * generation. out_tokens may be NULL iff max_out_tokens == 0. + * - out: OPTIONAL (may be NULL). When non-NULL it is filled exactly like + * vllm_complete (detokenized text owned by the caller, finish_reason, + * token counts). + * Returns VLLM_OK on success; a VLLM_ERR_* code with vllm_last_error() set on + * failure (*n_out_tokens zeroed, out zeroed when supplied). */ +VLLM_API vllm_status vllm_complete_tokens( + vllm_engine* engine, const int32_t* prompt_tokens, int32_t n_prompt_tokens, + const vllm_sampling_params* params, int32_t* out_tokens, + int32_t max_out_tokens, int32_t* n_out_tokens, vllm_completion* out); + /* ── Streaming completion (M3.5 Task 2) ─────────────────────────────────────── * vllm_token_callback: invoked once per engine-step delta for the streaming * request, then once more with finished == true to carry the finish. diff --git a/include/vllm/model_executor/models/kimi_linear.h b/include/vllm/model_executor/models/kimi_linear.h index 8861d5818..fee0f8f38 100644 --- a/include/vllm/model_executor/models/kimi_linear.h +++ b/include/vllm/model_executor/models/kimi_linear.h @@ -252,6 +252,13 @@ struct KdaResidentWeights { struct MlaResidentWeights { OwnedTensor q_proj, kv_a_proj_with_mqa, kv_b_proj, o_proj; // bf16 std::vector kv_a_layernorm; // f32 [kv_lora] + // ROW 7 (§20.3c) — the ABSORBED decode forms for the paged-FA2 NoPE-MLA arm + // (mla::ForwardMlaAttentionBlock): W_UK_T [nah, qk_nope, kv_lora] and W_UV + // [nah, kv_lora, v_head] bf16, computed from kv_b_proj at LOAD time + // (mla::AbsorbKvBProjBf16 — the host bf16 bytes are released after staging, so + // absorption cannot be deferred). Populated by BOTH resident builders; empty + // only on a pre-fold checkpoint of the struct (the paged-FA2 arm refuses). + OwnedTensor w_uk_t, w_uv; }; struct MlpResidentWeights { OwnedTensor gate_proj, up_proj, down_proj; // bf16 @@ -560,6 +567,24 @@ class KimiLinearModel { static ForwardLogits ForwardDecodeStepIncremental( int32_t token, int64_t position, const KimiLinearWeights& weights, vt::Queue& queue, KimiDecodeCache& cache); + + // ─── ROW 7 — THE SHARED-PAGED-RUNNER FOLD (§20.3, ARCH-ONE-SURFACE req 4) ──── + // The born-on-the-runner PRODUCTION forward: the whole 27-layer hybrid over the + // runner's OWN paged state — the KDA conv+recurrent state lives in the runner's + // MambaSpec `gdn_state` group keyed by `gdn_meta.non_spec_state_indices_tensor` + // (mirror vLLM kimi_gdn_linear_attn._forward's (conv_state, recurrent_state) + // slot handling), and the NoPE-MLA latent-KV lives in the paged `attn_kv` MLA + // group written through vt::ConcatAndCacheMla at `attn_meta.slot_mapping`. + // Prefill = vt::KdaChunkPrefill (vLLM's prompt path; recurrence when the + // request continues an existing state); decode = vt::KdaGatedDeltaRule (T==1). + // Batched decode-first (nd decodes then np prefills, the GDN builder's + // segmentation); byte-for-byte the DeviceForwardBodyBf16Incremental per-token + // compute with the per-sequence host KimiDecodeCache replaced by the paged + // groups — which is what makes the CLI-incremental battery the fold-identity + // reference. Returns DEVICE-RESIDENT [rows,vocab] logits for the on-GPU + // sampler (the third MUST-route seam). + static ForwardLogits ForwardPaged(const ModelForwardInput& input, + const KimiLinearWeights& weights); }; // KV-cache spec builder. The HETEROGENEOUS per-layer topology (spike §3): ONE MLA diff --git a/include/vllm/model_executor/models/model_registry.h b/include/vllm/model_executor/models/model_registry.h index 2eb305de6..f3afa8f7f 100644 --- a/include/vllm/model_executor/models/model_registry.h +++ b/include/vllm/model_executor/models/model_registry.h @@ -264,6 +264,11 @@ struct ModelFactory { // Preserves the already-gated per-arch scheduler default. This is execution // policy, not an upstream _ModelInfo capability. bool is_dense_model = false; + // ROW 7 (kimi-linear.md §20.3): the loader wants a `ModelSource::load_queue` + // selected BEFORE the weights load — the GB10 recipe (CUDA context first, then + // per-tensor stage-and-release; Kimi-Linear's 91.5 GiB bf16-resident loader). + // Default false: every existing arch's engine load path is byte-identical. + bool stage_on_load = false; }; struct ModelRegistration { diff --git a/scripts/check-surface-coverage.py b/scripts/check-surface-coverage.py index 9c3cf5428..199a3337f 100644 --- a/scripts/check-surface-coverage.py +++ b/scripts/check-surface-coverage.py @@ -103,7 +103,9 @@ # as folds land (developer-directed 2026-08-07: no permanent exemptions, the allowlist # shrinks). A new internal-reacher cannot be added without consciously RAISING this # ceiling — a reviewable red flag, which is the point. Lower it as each example is folded -# onto the public ABI. 9 since the ROW 2 MiniMax-H3 video fold (2026-08-08: +# onto the public ABI. 8 since the ROW 7 Kimi-Linear runner fold (kimi_linear_gen +# became a clean vllm.h client of the v13 vllm_complete_tokens after the +# paged-runner fold); 9 since the ROW 2 MiniMax-H3 video fold (2026-08-08: # minimax_h3_gen AND minimax_h3_mux became clean vllm.h clients of the v12 # vllm_video_* slice); 11 after the ROW 1 Parakeet fold (2026-08-07: # parakeet_transcribe); before it, 12 = server + the 4 spec-named drivers + @@ -112,7 +114,7 @@ # COUPLED: the ratchet claims in .agents/specs/surface-coverage-2026-08-07.md and the # state log, and the equality pin in tests/scripts/test_check_surface_coverage.py, must # move in the SAME change as this constant. -MAX_INTERNAL_REACHING = 9 +MAX_INTERNAL_REACHING = 8 # An include INTO the internal C++ tree: `#include "vllm/..."` (the engine/model/layer # headers under include/vllm/, NOT the flat public "vllm.h"), `#include "vt/..."` (the diff --git a/scripts/example-abi-allowlist.txt b/scripts/example-abi-allowlist.txt index f56a38397..faf5ce9bd 100644 --- a/scripts/example-abi-allowlist.txt +++ b/scripts/example-abi-allowlist.txt @@ -24,7 +24,6 @@ # --- Capability drivers whose FAST PATH is reachable ONLY here, not through the ABI --- examples/deepseek_v4_gen | fold=ARCH-ONE-SURFACE | DeepSeek-V4-Flash keep-quant GGUF greedy decode (DeepseekV4ForwardGguf(Cached) + DeepseekV4KvCache) is CLI-only; the registered DeepseekV4ForCausalLM forward is a W3-W8 stub (deepseek_v4_registry.cpp:22). Grow ABI keep-quant GGUF load+decode, rewrite as ABI client, delete the bespoke forward examples/laguna_gen | fold=ARCH-ONE-SURFACE | Laguna-S-2.1 keep-quant GGUF (multi-shard) + NVFP4 W4A4 device-resident decode (LagunaForwardGguf(Cached), Marlin residents, fp4-shared) is CLI-only; registered LagunaForCausalLM forward VT_CHECK(false)s on non-bf16 (laguna.cpp:156). Grow ABI, rewrite as client, delete bespoke forward -examples/kimi_linear_gen | fold=ARCH-ONE-SURFACE | Kimi-Linear-48B bf16-resident streaming loader + §18/§19 paged-incremental decode (ForwardPrefillIncremental/ForwardDecodeStepIncremental/KimiDecodeCache) is CLI-only (the recompute forward IS shared). Grow ABI incremental-decode path, rewrite as client, delete examples/server | fold=ARCH-ONE-SURFACE | The reference OpenAI server constructs the engine, metrics, video_runner and mm seam directly from internal C++ headers (LoadedEngine/AsyncLLM, minimax_h3.h, chat_mm). It should stand on the public surface (C ABI, or a curated public C++ API the ABI wraps). Grow the surface, rewrite the server against it # --- Dev / diagnostic tools: NO permanent exemption (developer-directed 2026-08-07) --- diff --git a/src/capi/vllm_c.cpp b/src/capi/vllm_c.cpp index 995cb62ac..ae5e90613 100644 --- a/src/capi/vllm_c.cpp +++ b/src/capi/vllm_c.cpp @@ -722,6 +722,87 @@ VLLM_API vllm_status vllm_complete(vllm_engine* engine, const char* prompt, } } +VLLM_API vllm_status vllm_complete_tokens( + vllm_engine* engine, const int32_t* prompt_tokens, int32_t n_prompt_tokens, + const vllm_sampling_params* params, int32_t* out_tokens, + int32_t max_out_tokens, int32_t* n_out_tokens, vllm_completion* out) { + if (n_out_tokens != nullptr) *n_out_tokens = 0; + if (out != nullptr) { + out->text = nullptr; + out->finish_reason = nullptr; + out->prompt_tokens = 0; + out->completion_tokens = 0; + } + if (engine == nullptr || prompt_tokens == nullptr || params == nullptr || + n_out_tokens == nullptr) { + SetError( + "vllm_complete_tokens: engine, prompt_tokens, params or n_out_tokens " + "is null"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (n_prompt_tokens <= 0) { + SetError("vllm_complete_tokens: n_prompt_tokens must be > 0"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (max_out_tokens < 0) { + SetError("vllm_complete_tokens: max_out_tokens must be >= 0"); + return VLLM_ERR_INVALID_ARGUMENT; + } + if (out_tokens == nullptr && max_out_tokens > 0) { + SetError("vllm_complete_tokens: out_tokens is null with max_out_tokens > 0"); + return VLLM_ERR_INVALID_ARGUMENT; + } + try { + const vllm::SamplingParams sp = + ToSamplingParams(*params, vllm::RequestOutputKind::kCumulative); + vllm::v1::AsyncLLM& e = engine->loaded->async_engine(); + const std::string request_id = + std::to_string(engine->next_request_id.fetch_add(1)); + RequestGuard guard{e, request_id}; + // The PRE-TOKENIZED add_request overload (vLLM TokensPrompt): builds the + // request from prompt_token_ids directly, skipping tokenization. + std::vector ids(prompt_tokens, prompt_tokens + n_prompt_tokens); + vllm::v1::AsyncRequest request = e.add_request(request_id, std::move(ids), sp); + vllm::RequestOutput result; + for (;;) { + result = e.get_output(request); + if (result.finished) break; + } + guard.disarm(); + + if (result.outputs.empty()) { + SetError("vllm_complete_tokens: engine produced no output sequence"); + return VLLM_ERR_RUNTIME; + } + const vllm::CompletionOutput& o = result.outputs[0]; + const int32_t n = + std::min(static_cast(o.token_ids.size()), max_out_tokens); + for (int32_t i = 0; i < n; ++i) out_tokens[i] = o.token_ids[static_cast(i)]; + *n_out_tokens = n; + if (out != nullptr) { + char* text = DupString(vllm::entrypoints::openai::SanitizeUtf8(o.text)); + if (text == nullptr) { + SetError("vllm_complete_tokens: out-of-memory copying completion text"); + return VLLM_ERR_RUNTIME; + } + out->text = text; + out->finish_reason = o.finish_reason.has_value() + ? CanonicalFinishReason(*o.finish_reason) + : nullptr; + out->prompt_tokens = static_cast(result.prompt_token_ids.size()); + out->completion_tokens = static_cast(o.token_ids.size()); + } + ClearError(); + return VLLM_OK; + } catch (const std::exception& e) { + SetError(std::string("vllm_complete_tokens: ") + e.what()); + return VLLM_ERR_RUNTIME; + } catch (...) { + SetError("vllm_complete_tokens: unknown error"); + return VLLM_ERR_UNKNOWN; + } +} + VLLM_API vllm_status vllm_complete_stream(vllm_engine* engine, const char* prompt, const vllm_sampling_params* params, diff --git a/src/vllm/entrypoints/model_loader.cpp b/src/vllm/entrypoints/model_loader.cpp index 64b08070b..00c778eee 100644 --- a/src/vllm/entrypoints/model_loader.cpp +++ b/src/vllm/entrypoints/model_loader.cpp @@ -975,7 +975,16 @@ std::unique_ptr LoadedEngine::FromModelDir( // Live architecture dispatch: consume config.architectures in order and let // the matched registration own the weight-name map/loader. Unknown dense // configs now reject instead of falling through num_experts == 0. - if (!registration.factory->is_dense_model || !DirectDeviceLoadRequested()) { + // + // ROW 7 (kimi-linear.md §20.3): a factory with `stage_on_load` (Kimi-Linear's + // 91.5 GiB bf16-resident loader) takes the queue-selected branch below so the + // CUDA context exists BEFORE the weights load and each tensor stages then + // releases its host mirror (the §13 GB10 recipe). Every other arch resolves + // this condition exactly as before — byte-identical. + const bool queue_load = + (registration.factory->is_dense_model && DirectDeviceLoadRequested()) || + registration.factory->stage_on_load; + if (!queue_load) { std::unique_ptr model = ModelRegistry::Load( config, ModelSource::FromSafetensorsOwned(shards)); maybe_attach_mtp(*model); diff --git a/src/vllm/model_executor/models/kimi_linear_device.cpp b/src/vllm/model_executor/models/kimi_linear_device.cpp index ede348fce..0ec6047ad 100644 --- a/src/vllm/model_executor/models/kimi_linear_device.cpp +++ b/src/vllm/model_executor/models/kimi_linear_device.cpp @@ -45,16 +45,20 @@ #include #include #include +#include #include #include #include #include #include +#include "vllm/model_executor/models/deepseek_v2.h" // MlaBatchSplit (ROW 7 fold) #include "vllm/model_executor/models/dense_device_glue.h" // dense_attn::{Dev,DBuf,MakeTensor} #include "vllm/model_executor/models/device_pool.h" // Pool() #include "vllm/model_executor/models/kimi_kda.h" +#include "vllm/model_executor/models/mla_attention.h" // mla::ForwardMlaAttentionBlock #include "vllm/platforms/interface.h" // platforms::GetPlatform (is_cpu) +#include "vllm/v1/attention/backends/gdn_attn.h" // GDNAttentionMetadata #include "vt/backend.h" #include "vt/dtype.h" #include "vt/ops.h" @@ -1629,6 +1633,728 @@ DBuf DeviceForwardBodyBf16Incremental(const Dev& d, const KimiLinearWeights& wei return logits; } +// ═══ ROW 7 — THE SHARED-PAGED-RUNNER FOLD (kimi-linear.md §20.3) ════════════════ +// The born-on-the-runner PRODUCTION forward: byte-for-byte the +// DeviceForwardBodyBf16Incremental per-token compute with the single-sequence +// host KimiDecodeCache replaced by the runner's OWN paged state groups — +// * KDA conv+recurrent state in the MambaSpec `gdn_state` group, keyed by +// `gdn_meta.non_spec_state_indices_tensor` (mirror vLLM +// kimi_gdn_linear_attn.py:296-440 `_forward`: `constant_caches` = +// (conv_state, recurrent_state) indexed by non_spec_state_indices_tensor; +// prefill = chunk_kda_with_fused_gate, decode = fused_recurrent_kda); +// * NoPE-MLA latent-KV in the paged `attn_kv` MLA group, written through +// vt::ConcatAndCacheMla at `attn_meta.slot_mapping` (the MLAAttentionSpec +// page: ONE 576-wide latent row per token, mla_attention.py:553-620 order). +// Batches are decode-first (the GDN builder's segmentation): nd single-token +// decodes then np prefills. NOT GdnBlockPaged — KDA's per-K-channel decay +// g[T,Hv,Dk] needs the KDA ops (vt::KdaChunkPrefill / vt::KdaGatedDeltaRule); +// the shared per-head GDN kernels stay untouched (qwen3_5 byte-identical). + +// VT_KIMI_PAGED_KDA_CHUNK (default ON) — process fresh prefill requests with the +// CHUNKED KDA kernel family (vt::KdaChunkPrefill — vLLM's prompt path and the +// §19 Gate-A-winning config); '0' falls back to the recurrence for A/B. Decode +// and continuing (has_initial_state) prefills always use the recurrence, exactly +// as vLLM (decode: fused_recurrent_kda; our chunk op takes a fresh zero state). +bool PagedKdaChunkEnabled() { + static const bool on = [] { + const char* e = std::getenv("VT_KIMI_PAGED_KDA_CHUNK"); + return e == nullptr || e[0] != '0'; + }(); + return on; +} + +// VT_KIMI_PAGED_MLA_FA2 (default ON — GB10-RULED 2026-08-07): route the 7 +// NoPE-MLA layers through mla::ForwardMlaAttentionBlock — vLLM's ACTUAL +// absorbed-MQA decode / FA2 prefill over the paged latent cache (§20.3c), +// identity RoPE, scale qk^-0.5. MEASURED on the real 48.9B (§21): this arm +// reproduces the golden's near-tie profile EXACTLY — 122/128 with p0-p6 16/16 +// and p7 10/16, the same 122/128 the CLI reference and the §12/§19 batteries +// carry — so per the parity-enablers-ship-as-defaults policy it IS the +// production arm. '0' selects the diagnostic EXACT arm (the f64 softmax island +// over kv_b-up-projected paged rows — byte-comparable to the CLI on CPU, the +// fold-identity vehicle): on GB10 it measured 111/128, the §19-documented +// GPU M-dimension-tiling near-tie perturbation (re-up-projecting the whole +// prefix at M=S vs the CLI's M=T append-time GEMM flips near-tie tokens: +// p7 flips TOWARD golden 16/16, p4 one flip that recovers, p2's token-1 flip +// cascades to 0/16) — a numeric-regime difference, not a paging bug (the CPU +// gate is byte-exact and FA2 shares every projection/cache write). +// NOT memoized (a cheap getenv per layer): the CPU gates pin the arm per test +// case, and a per-process latch would weld the whole binary to one arm. +bool PagedMlaFa2() { + const char* e = std::getenv("VT_KIMI_PAGED_MLA_FA2"); + return e == nullptr || e[0] != '0'; +} + +// Contiguous row-range view over a rank-N device tensor: rows [start, start+len). +inline Tensor RowsView(const Tensor& t, int64_t start, int64_t len, + const std::vector& shape) { + VT_CHECK(!shape.empty() && shape[0] == len, + "kimi paged: RowsView shape[0] must equal len"); + int64_t row = 1; + for (int i = 1; i < t.rank; ++i) row *= t.shape[i]; + return MakeTensor(static_cast(t.data) + + static_cast(start * row) * vt::SizeOf(t.dtype), + t.dtype, t.device, shape); +} + +// The decode-first non-spec segmentation the GDN builder emits, validated for +// the Kimi paged forward (no spec rows — this checkpoint has no MTP head). +struct KimiPagedSeg { + int nd = 0, np = 0; + int64_t nd_tok = 0, np_tok = 0; + const std::vector* sidx = nullptr; // per-request GDN state slots + const std::vector* qsl = nullptr; // [nreq+1] cumulative offsets + const std::vector* his = nullptr; // per-request has_initial (np>0) +}; +KimiPagedSeg KimiSegment(const v1::GDNAttentionMetadata& gm, int64_t T) { + KimiPagedSeg s; + VT_CHECK(gm.num_spec_decodes == 0 && gm.num_spec_decode_tokens == 0, + "kimi paged: spec-decode rows are not expressible (the 48B-Instruct " + "checkpoint has no MTP head; num_nextn_predict_layers=0)"); + s.nd = gm.num_decodes; + s.np = gm.num_prefills; + s.nd_tok = gm.num_decode_tokens; + s.np_tok = gm.num_prefill_tokens; + VT_CHECK(s.nd_tok + s.np_tok == T, "kimi paged: decode+prefill tokens != T"); + VT_CHECK(s.nd_tok == s.nd, "kimi paged: decode segment must be 1 token/request"); + VT_CHECK(gm.non_spec_state_indices_tensor.has_value() && + gm.non_spec_query_start_loc.has_value(), + "kimi paged: GDN metadata is missing state indices / query offsets"); + s.sidx = &*gm.non_spec_state_indices_tensor; + s.qsl = &*gm.non_spec_query_start_loc; + VT_CHECK(static_cast(s.sidx->size()) >= s.nd + s.np, + "kimi paged: state index vector shorter than the batch"); + if (s.np > 0) { + VT_CHECK(gm.has_initial_state.has_value() && + static_cast(gm.has_initial_state->size()) >= s.nd + s.np, + "kimi paged: prefill batch is missing has_initial_state"); + s.his = &*gm.has_initial_state; + } + return s; +} + +// KDA layer over the PAGED conv+recurrent state (the paged form of +// KdaLayerDeviceBf16Inc). The projection/conv/L2/gate/gated-norm op sequence is +// byte-identical; only the state residency changes: conv taps + recurrent state +// are gathered from / scattered to the runner's `gdn_state` group rows named by +// the per-request state slots. Layout of one conv row: [q taps | k taps | v taps] +// each [proj, K-1] — vLLM's `conv_state.chunk(3)` (kimi_gdn_linear_attn.py:331). +DBuf KdaLayerPagedBf16(const Dev& d, const KdaResidentWeights& w, const Tensor& dh, + const KimiLinearParams& p, int64_t T, const KimiPagedSeg& seg, + const GdnStateCache& state) { + const int64_t H = p.hidden_size; + const int64_t nh = p.kda_num_heads; + const int64_t hd = p.kda_head_dim; + const int64_t proj = nh * hd; + const int64_t K = p.kda_short_conv_kernel_size; + const int64_t conv_dim = 3 * proj; + const int64_t nreq = seg.nd + seg.np; + const float scale = static_cast(std::pow(static_cast(hd), -0.5)); + + VT_CHECK(state.conv_state.shape[1] == conv_dim && + state.conv_state.shape[2] == K - 1 && + state.ssm_state.shape[1] == nh && state.ssm_state.shape[2] == hd && + state.ssm_state.shape[3] == hd, + "kimi paged: runner GDN state geometry disagrees with linear_attn_config"); + + DBuf rq(d, DType::kF32, {T, proj}); + GemmBf16(d, rq.t(), dh, w.q_proj, proj, H); + DBuf rk(d, DType::kF32, {T, proj}); + GemmBf16(d, rk.t(), dh, w.k_proj, proj, H); + DBuf rv(d, DType::kF32, {T, proj}); + GemmBf16(d, rv.t(), dh, w.v_proj, proj, H); + + // ── conv (3 separate q/k/v short convs over the paged conv row) ── + DBuf didx(d, DType::kI32, {nreq}, seg.sidx->data()); + DBuf dcs(d, DType::kF32, {nreq, conv_dim, K - 1}); + vt::GdnStateGather(d.q, dcs.t(), state.conv_state, didx.t()); + const size_t sec_bytes = static_cast(proj) * (K - 1) * sizeof(float); + const size_t row_bytes = static_cast(conv_dim) * (K - 1) * sizeof(float); + auto section_out = [&](int64_t sec) { + DBuf s(d, DType::kF32, {nreq, proj, K - 1}); + for (int64_t r = 0; r < nreq; ++r) + d.b.Copy(d.q, static_cast(s.ptr()) + static_cast(r) * sec_bytes, + static_cast(dcs.ptr()) + static_cast(r) * row_bytes + + static_cast(sec) * sec_bytes, + sec_bytes); + return s; + }; + auto section_back = [&](DBuf& s, int64_t sec) { + for (int64_t r = 0; r < nreq; ++r) + d.b.Copy(d.q, + static_cast(dcs.ptr()) + static_cast(r) * row_bytes + + static_cast(sec) * sec_bytes, + static_cast(s.ptr()) + static_cast(r) * sec_bytes, + sec_bytes); + }; + DBuf cs_q = section_out(0); + DBuf cs_k = section_out(1); + DBuf cs_v = section_out(2); + + DBuf qc(d, DType::kF32, {T, proj}); + DBuf kc(d, DType::kF32, {T, proj}); + DBuf vc(d, DType::kF32, {T, proj}); + if (seg.np > 0) { + // Any prefill: varlen conv over the whole non-spec stream (decodes lead, + // each a 1-token slice with has_initial=1) — qwen3_5.cpp's np>0 conv branch, + // with Kimi's three separate convs in place of the merged one. + std::vector his32(seg.his->begin(), seg.his->begin() + nreq); + DBuf dqsl(d, DType::kI32, {nreq + 1}, seg.qsl->data()); + DBuf dhis(d, DType::kI32, {nreq}, his32.data()); + vt::CausalConv1dFwd(d.q, qc.t(), rq.t(), WF32(d, w.q_conv, {proj, K}), nullptr, + cs_q.t(), dqsl.t(), dhis.t(), vt::CausalConv1dArgs{true}); + vt::CausalConv1dFwd(d.q, kc.t(), rk.t(), WF32(d, w.k_conv, {proj, K}), nullptr, + cs_k.t(), dqsl.t(), dhis.t(), vt::CausalConv1dArgs{true}); + vt::CausalConv1dFwd(d.q, vc.t(), rv.t(), WF32(d, w.v_conv, {proj, K}), nullptr, + cs_v.t(), dqsl.t(), dhis.t(), vt::CausalConv1dArgs{true}); + } else { + // Pure decode: single-token conv step per sequence on the compact gathered + // rows (mamba causal_conv1d_update; numerically the same window sum as + // CausalConv1dFwd(T=1, has_initial=1) — the CLI ConvSiluInc form). + vt::CausalConv1dUpdate(d.q, qc.t(), rq.t(), WF32(d, w.q_conv, {proj, K}), nullptr, + cs_q.t(), vt::CausalConv1dArgs{true}); + vt::CausalConv1dUpdate(d.q, kc.t(), rk.t(), WF32(d, w.k_conv, {proj, K}), nullptr, + cs_k.t(), vt::CausalConv1dArgs{true}); + vt::CausalConv1dUpdate(d.q, vc.t(), rv.t(), WF32(d, w.v_conv, {proj, K}), nullptr, + cs_v.t(), vt::CausalConv1dArgs{true}); + } + section_back(cs_q, 0); + section_back(cs_k, 1); + section_back(cs_v, 2); + { + Tensor conv_cache = state.conv_state; + vt::GdnStateScatter(d.q, conv_cache, dcs.t(), didx.t()); + } + + // ── post-conv: q/k L2 norm + the low-rank decay/gate projections (== Inc) ── + DBuf qn(d, DType::kF32, {T, proj}); + DBuf kn(d, DType::kF32, {T, proj}); + { + Tensor qc2 = MakeTensor(qc.ptr(), DType::kF32, d.q.device, {T * nh, hd}); + Tensor qn2 = MakeTensor(qn.ptr(), DType::kF32, d.q.device, {T * nh, hd}); + Tensor kc2 = MakeTensor(kc.ptr(), DType::kF32, d.q.device, {T * nh, hd}); + Tensor kn2 = MakeTensor(kn.ptr(), DType::kF32, d.q.device, {T * nh, hd}); + vt::L2Norm(d.q, qn2, qc2, vt::L2NormArgs{1e-6f}); + vt::L2Norm(d.q, kn2, kc2, vt::L2NormArgs{1e-6f}); + } + DBuf braw(d, DType::kF32, {T, nh}); + GemmBf16(d, braw.t(), dh, w.b_proj, nh, H); + DBuf fa(d, DType::kF32, {T, hd}); + GemmBf16(d, fa.t(), dh, w.f_a_proj, hd, H); + DBuf g1(d, DType::kF32, {T, proj}); + GemmBf16(d, g1.t(), fa.t(), w.f_b_proj, proj, hd); + DBuf ga(d, DType::kF32, {T, hd}); + GemmBf16(d, ga.t(), dh, w.g_a_proj, hd, H); + DBuf g2(d, DType::kF32, {T, proj}); + GemmBf16(d, g2.t(), ga.t(), w.g_b_proj, proj, hd); + + // beta = sigmoid(b_proj) — host elementwise, exactly the Inc island's form. + std::vector hbraw(static_cast(T) * nh); + braw.Download(d, hbraw.data()); + RoundHostBf16(hbraw); + std::vector hbeta(hbraw.size()); + for (size_t i = 0; i < hbeta.size(); ++i) + hbeta[i] = static_cast(Sigmoid(hbraw[i])); + DBuf dbeta(d, DType::kF32, {T, nh}, hbeta.data()); + + // Raw per-channel gate projection, downloaded once; the recurrent segments + // compute the decay gate on host (kimi_kda::KdaDecayGate — the Inc island), + // the chunk segment hands the RAW g1 to the fused on-device gate. + std::vector hg1(static_cast(T) * proj); + g1.Download(d, hg1.data()); + RoundHostBf16(hg1); + + DBuf dcore(d, DType::kF32, {T, proj}); + + // ── decode segment [0, nd): batched T==1 recurrence over the paged state ── + if (seg.nd > 0) { + const int64_t ndt = seg.nd_tok; + DBuf dss(d, DType::kF32, {seg.nd, nh, hd, hd}); + Tensor didx_dec = RowsView(didx.t(), 0, seg.nd, {seg.nd}); + vt::GdnStateGather(d.q, dss.t(), state.ssm_state, didx_dec); + const std::vector hg_dec(hg1.begin(), + hg1.begin() + static_cast(ndt) * proj); + const std::vector gch = + kimi_kda::KdaDecayGate(hg_dec, w.a_log, w.dt_bias, ndt, nh, hd); + DBuf dg(d, DType::kF32, {ndt, nh, hd}, gch.data()); + std::vector qsl_dec(static_cast(seg.nd) + 1); + for (int64_t i = 0; i <= seg.nd; ++i) qsl_dec[static_cast(i)] = + static_cast(i); + DBuf dqsl(d, DType::kI32, {seg.nd + 1}, qsl_dec.data()); + Tensor q3v = RowsView(qn.t(), 0, ndt, {ndt, nh, hd}); + Tensor k3v = RowsView(kn.t(), 0, ndt, {ndt, nh, hd}); + Tensor v3v = RowsView(vc.t(), 0, ndt, {ndt, nh, hd}); + Tensor b2v = RowsView(dbeta.t(), 0, ndt, {ndt, nh}); + Tensor o3v = RowsView(dcore.t(), 0, ndt, {ndt, nh, hd}); + vt::KdaGatedDeltaRule(d.q, o3v, q3v, k3v, v3v, dg.t(), b2v, dss.t(), dqsl.t(), + vt::GdnArgs{scale}); + Tensor ssm_cache = state.ssm_state; + vt::GdnStateScatter(d.q, ssm_cache, dss.t(), didx_dec); + } + + // ── prefill segment: per request — chunk (fresh) or recurrence (continuing) ── + for (int r = 0; r < seg.np; ++r) { + const int req = seg.nd + r; + const int64_t tok0 = (*seg.qsl)[static_cast(req)]; + const int64_t tok1 = (*seg.qsl)[static_cast(req) + 1]; + const int64_t Tr = tok1 - tok0; + if (Tr <= 0) continue; + const bool has_init = seg.his != nullptr && (*seg.his)[static_cast(req)] != 0; + DBuf dss1(d, DType::kF32, {1, nh, hd, hd}); + Tensor didx_r = RowsView(didx.t(), req, 1, {1}); + const int32_t hi32[1] = {has_init ? 1 : 0}; + DBuf dhi(d, DType::kI32, {1}, hi32); + Tensor dhi_t = dhi.t(); + vt::GdnStateGather(d.q, dss1.t(), state.ssm_state, didx_r, &dhi_t); + const int32_t qsl1[2] = {0, static_cast(Tr)}; + DBuf dqsl1(d, DType::kI32, {2}, qsl1); + Tensor q3v = RowsView(qn.t(), tok0, Tr, {Tr, nh, hd}); + Tensor k3v = RowsView(kn.t(), tok0, Tr, {Tr, nh, hd}); + Tensor v3v = RowsView(vc.t(), tok0, Tr, {Tr, nh, hd}); + Tensor b2v = RowsView(dbeta.t(), tok0, Tr, {Tr, nh}); + Tensor o3v = RowsView(dcore.t(), tok0, Tr, {Tr, nh, hd}); + if (PagedKdaChunkEnabled() && !has_init && Tr > 1) { + // vLLM's PROMPT path: the chunk kernels fuse the gate from the RAW g1. + DBuf da_log(d, DType::kF32, {nh}, w.a_log.data()); + DBuf ddt(d, DType::kF32, {static_cast(w.dt_bias.size())}, + w.dt_bias.empty() ? nullptr : w.dt_bias.data()); + Tensor gr3 = RowsView(g1.t(), tok0, Tr, {Tr, nh, hd}); + vt::KdaChunkPrefill(d.q, o3v, q3v, k3v, v3v, gr3, b2v, da_log.t(), ddt.t(), + dss1.t(), dqsl1.t(), vt::GdnArgs{scale}); + } else { + const std::vector hg_r( + hg1.begin() + static_cast(tok0) * proj, + hg1.begin() + static_cast(tok1) * proj); + const std::vector gch = + kimi_kda::KdaDecayGate(hg_r, w.a_log, w.dt_bias, Tr, nh, hd); + DBuf dg(d, DType::kF32, {Tr, nh, hd}, gch.data()); + vt::KdaGatedDeltaRule(d.q, o3v, q3v, k3v, v3v, dg.t(), b2v, dss1.t(), + dqsl1.t(), vt::GdnArgs{scale}); + } + Tensor ssm_cache = state.ssm_state; + vt::GdnStateScatter(d.q, ssm_cache, dss1.t(), didx_r); + } + + // ── sigmoid-gated RMSNorm output + o_proj (== Inc) ── + DBuf dcn(d, DType::kF32, {T, proj}); + { + Tensor x3 = MakeTensor(dcore.ptr(), DType::kF32, d.q.device, {T, nh, hd}); + Tensor g3 = MakeTensor(g2.ptr(), DType::kF32, d.q.device, {T, nh, hd}); + Tensor o3 = MakeTensor(dcn.ptr(), DType::kF32, d.q.device, {T, nh, hd}); + vt::RmsNormGated(d.q, o3, x3, g3, WF32(d, w.o_norm, {hd}), + vt::RmsNormGatedArgs{p.rms_norm_eps, /*sigmoid_gate=*/true}); + } + DBuf out(d, DType::kF32, {T, H}); + GemmBf16(d, out.t(), dcn.t(), w.o_proj, H, proj); + return out; +} + +// ── the per-forward shared MLA step state (positions/slot_mapping/FA2 meta) ──── +struct KimiMlaStep { + std::vector owned; + Tensor positions; // [T] i32 device + Tensor slot_mapping; // [T] i64 device + // FA2 arm only: + mla::MlaBlockMetadata meta; + MlaBatchSplit split; + DBuf* rope_cache = nullptr; // identity [rows, qr] bf16 (owned) + v1::TritonMLAImpl impl; +}; + +template +Tensor KimiUploadInto(const Dev& d, std::vector& owned, DType dt, + const std::vector& shape, const T* host) { + owned.emplace_back(d, dt, shape, host); + return owned.back().t(); +} + +// Build the per-step MLA metadata (the Kimi-local, EAGER-only port of +// deepseek_v2.cpp BuildMlaStep — `MLACommonMetadataBuilder.build` +// mla_attention.py:1652-1830, non-DCP; no CUDA-graph constraints because the +// Kimi paged forward is eager). +void BuildKimiMlaStep(const Dev& d, const std::vector& positions, + const v1::CommonAttentionMetadata& am, + const KimiLinearParams& p, int64_t block_size, bool fa2, + KimiMlaStep& s) { + const int64_t T = static_cast(positions.size()); + s.positions = KimiUploadInto(d, s.owned, DType::kI32, {T}, positions.data()); + s.slot_mapping = + KimiUploadInto(d, s.owned, DType::kI64, {T}, am.slot_mapping.data()); + if (!fa2) return; + + s.split = BuildMlaBatchSplit(am); + const MlaBatchSplit& sp = s.split; + const int64_t cols = am.block_table_num_cols; + s.meta.num_decode_tokens = sp.num_decode_tokens; + if (sp.num_decodes > 0) { + s.meta.decode.block_table = KimiUploadInto( + d, s.owned, DType::kI32, {sp.num_decodes, cols}, am.block_table_tensor.data()); + s.meta.decode.seq_lens = KimiUploadInto(d, s.owned, DType::kI32, + {sp.num_decodes}, am.seq_lens.data()); + s.meta.decode.max_seq_len = sp.decode_max_seq_len; + } + if (sp.num_prefills > 0) { + s.meta.prefill_cu_seqlens_q = + KimiUploadInto(d, s.owned, DType::kI32, {sp.num_prefills + 1}, + sp.prefill_cu_seqlens_q.data()); + s.meta.prefill_block_table = KimiUploadInto( + d, s.owned, DType::kI32, {sp.num_prefills, cols}, + am.block_table_tensor.data() + + static_cast(sp.num_decodes) * static_cast(cols)); + s.meta.max_query_len = sp.prefill_max_query_len; + if (sp.num_prefills_with_context > 0) { + const int64_t workspace = mla::DetermineChunkedPrefillWorkspaceSize( + p.max_position_embeddings, am.num_reqs, block_size); + const mla::MlaChunkedContextMetadata cm = mla::BuildMlaChunkedContext( + sp.prefill_context_lens, sp.prefill_cu_seqlens_q, workspace, block_size); + s.meta.prefill_tokens_with_context = cm.prefill_tokens_with_context; + s.meta.chunk_workspace_tokens = workspace; + const int64_t np = cm.num_prefills; + const int32_t row = std::max(cm.max_token_num_over_chunk, 1); + for (int32_t i = 0; i < cm.num_chunks; ++i) { + mla::MlaChunkDeviceMetadata cd; + cd.cu_seq_lens = KimiUploadInto( + d, s.owned, DType::kI32, {np + 1}, + cm.cu_seq_lens.data() + static_cast(i) * (np + 1)); + cd.starts = KimiUploadInto(d, s.owned, DType::kI32, {np}, + cm.starts.data() + static_cast(i) * np); + cd.token_to_seq = + KimiUploadInto(d, s.owned, DType::kI32, {row}, + cm.token_to_seq.data() + static_cast(i) * row); + cd.total_tokens = cm.chunk_total_token[static_cast(i)]; + cd.max_seq_len = cm.max_seq_lens[static_cast(i)]; + s.meta.chunks.push_back(cd); + } + } + } +} + +// NoPE-MLA layer over the PAGED latent cache — the EXACT-island arm (default; +// the fold-identity vehicle). Projections + kv_a_layernorm + the cache write are +// on-device; the attention core re-derives each request's per-head K/V from the +// paged 576-wide latent rows (kv_b up-projection — the SAME GemmBf16 the CLI ran +// at append time; the cache stores bf16 exactly as vLLM does) and runs the SAME +// f64 causal-softmax island as MlaSoftmaxIslandInc. +DBuf MlaLayerPagedExact(const Dev& d, const MlaResidentWeights& w, const Tensor& dh, + const KimiLinearParams& p, int64_t T, + const v1::CommonAttentionMetadata& am, + const PagedKvCache& kv, const KimiMlaStep& step) { + const int64_t H = p.hidden_size; + const int64_t nah = p.num_attention_heads; + const int64_t qn = p.qk_nope_head_dim; + const int64_t qr = p.qk_rope_head_dim; + const int64_t qk = qn + qr; + const int64_t vh = p.v_head_dim; + const int64_t L = p.kv_lora_rank; + const int64_t kvw = nah * (qn + vh); + const int64_t head = L + qr; + VT_CHECK(kv.num_kv_heads == 1 && kv.head_size == head, + "kimi paged: the MLA cache must be 1-head, kv_lora+qk_rope wide " + "(MLAAttentionSpec)"); + + DBuf dq(d, DType::kF32, {T, nah * qk}); + GemmBf16(d, dq.t(), dh, w.q_proj, nah * qk, H); + DBuf dlat(d, DType::kF32, {T, L + qr}); + GemmBf16(d, dlat.t(), dh, w.kv_a_proj_with_mqa, L + qr, H); + DBuf dkvc(d, DType::kF32, {T, L}); + DBuf dkpe(d, DType::kF32, {T, qr}); + { + const size_t rl = static_cast(L + qr) * sizeof(float); + const char* src = static_cast(dlat.ptr()); + char* pc = static_cast(dkvc.ptr()); + char* pp = static_cast(dkpe.ptr()); + for (int64_t t = 0; t < T; ++t) { + d.b.Copy(d.q, pc + static_cast(t) * L * sizeof(float), + src + static_cast(t) * rl, + static_cast(L) * sizeof(float)); + d.b.Copy(d.q, pp + static_cast(t) * qr * sizeof(float), + src + static_cast(t) * rl + static_cast(L) * sizeof(float), + static_cast(qr) * sizeof(float)); + } + } + DBuf dkvcn(d, DType::kF32, {T, L}); + vt::RmsNorm(d.q, dkvcn.t(), dkvc.t(), WF32(d, w.kv_a_layernorm, {L}), + vt::RmsNormArgs{p.rms_norm_eps, false}); + + // Write this step's latent rows into the paged cache at slot_mapping (vLLM's + // concat_and_cache_mla order: BEFORE the attention reads). Cache dtype follows + // the spec (bf16 default — vLLM's regime; f32 under VT_KV_CACHE_F32). + Tensor cache_t = MakeTensor(kv.data, kv.dtype, d.q.device, + {kv.num_blocks, kv.block_size, head}); + if (kv.dtype == DType::kF32) { + vt::ConcatAndCacheMla(d.q, dkvcn.t(), dkpe.t(), cache_t, step.slot_mapping); + } else { + DBuf ckv(d, kv.dtype, {T, L}); + DBuf cpe(d, kv.dtype, {T, qr}); + vt::CastBf16(d.q, ckv.t(), dkvcn.t()); + vt::CastBf16(d.q, cpe.t(), dkpe.t()); + vt::ConcatAndCacheMla(d.q, ckv.t(), cpe.t(), cache_t, step.slot_mapping); + } + + // Attention per request over the paged rows (query rows [tok0, tok1) at global + // positions [base, base+Tq)). + DBuf dout(d, DType::kF32, {T, nah * vh}); + const size_t es = vt::SizeOf(kv.dtype); + const size_t row_b = static_cast(head) * es; + std::vector hrows; + for (int r = 0; r < am.num_reqs; ++r) { + const int64_t tok0 = am.query_start_loc[static_cast(r)]; + const int64_t tok1 = am.query_start_loc[static_cast(r) + 1]; + const int64_t Tq = tok1 - tok0; + if (Tq <= 0) continue; + const int64_t base = am.num_computed_tokens_cpu[static_cast(r)]; + const int64_t S = base + Tq; + // Gather the request's S latent rows (block table walk) to host. + hrows.resize(static_cast(S) * row_b); + const int32_t* bt = am.block_table_tensor.data() + + static_cast(r) * am.block_table_num_cols; + for (int64_t s0 = 0; s0 < S; s0 += kv.block_size) { + const int64_t blk = bt[s0 / kv.block_size]; + const int64_t n = std::min(kv.block_size, S - s0); + d.b.Copy(d.q, hrows.data() + static_cast(s0) * row_b, + static_cast(kv.data) + + static_cast(blk) * kv.block_size * row_b, + static_cast(n) * row_b); + } + d.b.Synchronize(d.q); + // Split latent | kpe to f32 host. + std::vector hlat(static_cast(S) * L); + std::vector hkpe(static_cast(S) * qr); + for (int64_t s0 = 0; s0 < S; ++s0) { + const uint8_t* row = hrows.data() + static_cast(s0) * row_b; + if (kv.dtype == DType::kF32) { + std::memcpy(&hlat[static_cast(s0) * L], row, + static_cast(L) * sizeof(float)); + std::memcpy(&hkpe[static_cast(s0) * qr], + row + static_cast(L) * sizeof(float), + static_cast(qr) * sizeof(float)); + } else { + const uint16_t* rb = reinterpret_cast(row); + for (int64_t i = 0; i < L; ++i) + hlat[static_cast(s0 * L + i)] = vt::BF16ToF32(rb[i]); + for (int64_t i = 0; i < qr; ++i) + hkpe[static_cast(s0 * qr + i)] = vt::BF16ToF32(rb[L + i]); + } + } + // Up-project the latent to per-head k_nope|v (the CLI's append-time GemmBf16; + // the bf16 activation cast of the same latent values feeds the same GEMM). + DBuf dlat_r(d, DType::kF32, {S, L}, hlat.data()); + DBuf dkv_r(d, DType::kF32, {S, kvw}); + GemmBf16(d, dkv_r.t(), dlat_r.t(), w.kv_b_proj, kvw, L); + std::vector hkv(static_cast(S) * kvw); + dkv_r.Download(d, hkv.data()); + RoundHostBf16(hkv); + RoundHostBf16(hkpe); + // The SAME f64 causal-softmax island as the CLI (MlaSoftmaxIslandInc). + DBuf dq_r(d, DType::kF32, {Tq, nah * qk}); + d.b.Copy(d.q, dq_r.ptr(), + static_cast(dq.ptr()) + + static_cast(tok0) * nah * qk * sizeof(float), + static_cast(Tq) * nah * qk * sizeof(float)); + DBuf o_r = MlaSoftmaxIslandInc(d, dq_r, hkv, hkpe, p, Tq, base); + d.b.Copy(d.q, + static_cast(dout.ptr()) + + static_cast(tok0) * nah * vh * sizeof(float), + o_r.ptr(), static_cast(Tq) * nah * vh * sizeof(float)); + } + DBuf attn(d, DType::kF32, {T, H}); + GemmBf16(d, attn.t(), dout.t(), w.o_proj, H, nah * vh); + return attn; +} + +// NoPE-MLA layer through mla::ForwardMlaAttentionBlock — vLLM's ACTUAL absorbed- +// MQA decode / FA2 prefill over the paged latent cache (§20.3c / §20.2). Identity +// RoPE (cos=1, sin=0: the GPT-J pair rotation is then the identity — NoPE, no +// positional term), scale qk_head_dim^-0.5, the no-q-lora branch. The block does +// its OWN ConcatAndCacheMla write. +DBuf MlaLayerPagedFa2(const Dev& d, const MlaResidentWeights& w, const Tensor& dh_f32, + const KimiLinearParams& p, int64_t T, const PagedKvCache& kv, + KimiMlaStep& step, const DBuf& kv_a_ln_bf16) { + const int64_t H = p.hidden_size; + const int64_t nah = p.num_attention_heads; + const int64_t qn = p.qk_nope_head_dim; + const int64_t qr = p.qk_rope_head_dim; + const int64_t qk = qn + qr; + const int64_t vh = p.v_head_dim; + const int64_t L = p.kv_lora_rank; + const int64_t head = L + qr; + VT_CHECK(kv.dtype == DType::kBF16, + "kimi paged FA2 MLA: the paged latent cache must be bf16 (unset " + "VT_KV_CACHE_F32 for the FA2 arm)"); + VT_CHECK(!w.w_uk_t.Empty() && !w.w_uv.Empty(), + "kimi paged FA2 MLA: absorbed W_UK_T/W_UV missing (loader absorption)"); + + mla::MlaBlockDims dm; + dm.hidden_size = H; + dm.num_heads = nah; + dm.qk_nope_head_dim = qn; + dm.qk_rope_head_dim = qr; + dm.v_head_dim = vh; + dm.kv_lora_rank = L; + dm.q_lora_rank = 0; + dm.rms_norm_eps = p.rms_norm_eps; + dm.scale = static_cast(std::pow(static_cast(qk), -0.5)); + + mla::MlaBlockWeights mw; + mw.kv_a_proj_with_mqa = ResidentBf16W(d, w.kv_a_proj_with_mqa, {L + qr, H}); + mw.q_proj = ResidentBf16W(d, w.q_proj, {nah * qk, H}); + mw.kv_a_layernorm = kv_a_ln_bf16.t(); + mw.kv_b_proj = ResidentBf16W(d, w.kv_b_proj, {nah * (qn + vh), L}); + mw.w_uk_t = ResidentBf16W(d, w.w_uk_t, {nah, qn, L}); + mw.w_uv = ResidentBf16W(d, w.w_uv, {nah, L, vh}); + mw.o_proj = ResidentBf16W(d, w.o_proj, {H, nah * vh}); + mw.rope_cos_sin_cache = step.rope_cache->t(); + + Tensor cache_t = MakeTensor(kv.data, kv.dtype, d.q.device, + {kv.num_blocks, kv.block_size, head}); + DBuf dh_bf16(d, DType::kBF16, {T, H}); + vt::CastBf16(d.q, dh_bf16.t(), dh_f32); + DBuf attn_bf16(d, DType::kBF16, {T, H}); + Tensor attn_t = attn_bf16.t(); + mla::ForwardMlaAttentionBlock(d, dm, mw, dh_bf16.t(), step.positions, cache_t, + step.slot_mapping, step.meta, step.impl, attn_t); + DBuf attn(d, DType::kF32, {T, H}); + Tensor attn_f = attn.t(); + vt::CastF32(d.q, attn_f, attn_bf16.t()); + return attn; +} + +// The whole paged-runner device forward — DeviceForwardBodyBf16Incremental's +// skeleton with the paged KDA/MLA layer forms and the runner's own metadata. +DBuf DeviceForwardBodyBf16Paged(const Dev& d, const KimiLinearWeights& weights, + const ModelForwardInput& in) { + const KimiLinearResidentWeights& rw = weights.resident; + const KimiLinearParams& p = weights.params; + const int64_t H = p.hidden_size; + const int64_t V = p.vocab_size; + const int64_t L = p.num_hidden_layers; + const int64_t T = static_cast(in.token_ids.size()); + const float eps = p.rms_norm_eps; + VT_CHECK(T > 0, "kimi paged: empty token batch"); + VT_CHECK(rw.resident, "kimi paged: bf16-resident weights required (§13)"); + VT_CHECK(static_cast(rw.layers.size()) == L, + "kimi paged: resident layer count != num_hidden_layers"); + VT_CHECK(in.attn_meta.num_actual_tokens == T, + "kimi paged: attn_meta num_actual_tokens != token count"); + int64_t nkda = 0, nmla = 0; + for (int64_t l = 0; l < L; ++l) (p.is_kda_layer(l) ? nkda : nmla)++; + VT_CHECK(static_cast(in.gdn_state.size()) == nkda, + "kimi paged: one GdnStateCache per KDA layer required"); + VT_CHECK(static_cast(in.attn_kv.size()) == nmla, + "kimi paged: one MLA PagedKvCache per full-attention layer required"); + + const KimiPagedSeg seg = KimiSegment(in.gdn_meta, T); + const bool fa2 = PagedMlaFa2(); + const int64_t block_size = in.attn_kv.empty() ? 0 : in.attn_kv[0].block_size; + KimiMlaStep step; + BuildKimiMlaStep(d, in.positions, in.attn_meta, p, block_size, fa2, step); + // Per-layer bf16 kv_a_layernorm views + the identity rope cache (FA2 arm). + std::vector> kv_a_ln_bf16(rw.layers.size()); + std::unique_ptr rope; + if (fa2) { + const int64_t rows = + std::max(in.attn_meta.max_seq_len + 1, 2); + std::vector ident(static_cast(rows) * p.qk_rope_head_dim); + const uint16_t one = vt::F32ToBF16(1.0f); + const int64_t half = p.qk_rope_head_dim / 2; + for (int64_t rr = 0; rr < rows; ++rr) + for (int64_t i = 0; i < half; ++i) + ident[static_cast(rr * p.qk_rope_head_dim + i)] = one; // cos=1|sin=0 + rope = std::make_unique(d, DType::kBF16, + std::vector{rows, p.qk_rope_head_dim}, + ident.data()); + step.rope_cache = rope.get(); + } + + DBuf hidden(d, DType::kF32, {T, H}); + { + Tensor htab = ResidentBf16W(d, rw.embed_tokens, {V, H}); + Tensor hh = hidden.t(); + if (in.device_token_ids != nullptr) { + // ENG-ASYNC-SCHED W4 (the async device mirror, DEFAULT ON on a real CUDA + // GPU): the runner patched each decode row's sampled token into ITS device + // input-id buffer and deliberately left the host `token_ids` STALE — a + // forward that embeds the host vector reads the previous step's token and + // decodes garbage (the GB10 9/128 divergence this branch was cut from). + // Embed from the device pointer, exactly like qwen3_5's + // DeviceTokenIdsScope consumer. + Tensor ids = MakeTensor(const_cast(in.device_token_ids), + DType::kI32, d.q.device, {T}); + vt::Embedding(d.q, hh, htab, ids); + } else { + DBuf dids(d, DType::kI32, {T}, in.token_ids.data()); + vt::Embedding(d.q, hh, htab, dids.t()); + } + } + DBuf res(d, DType::kF32, {T, H}); + res.Zero(d); + Tensor hcur = hidden.t(); + std::shared_ptr hold; + + int64_t kda_idx = 0, mla_idx = 0; + for (int64_t l = 0; l < L; ++l) { + const KimiLinearLayerResidentWeights& lw = rw.layers[static_cast(l)]; + DBuf dhn(d, DType::kF32, {T, H}); + AddRmsNormS(d, dhn, hcur, lw.input_layernorm, H, res, eps, DType::kF32); + DBuf attn = [&]() -> DBuf { + if (lw.is_kda) { + return KdaLayerPagedBf16(d, lw.kda, dhn.t(), p, T, seg, + in.gdn_state[static_cast(kda_idx++)]); + } + const PagedKvCache& kv = in.attn_kv[static_cast(mla_idx++)]; + if (!fa2) return MlaLayerPagedExact(d, lw.mla, dhn.t(), p, T, in.attn_meta, + kv, step); + std::unique_ptr& ln = kv_a_ln_bf16[static_cast(l)]; + if (!ln) { + DBuf lnf(d, DType::kF32, + {static_cast(lw.mla.kv_a_layernorm.size())}, + lw.mla.kv_a_layernorm.data()); + ln = std::make_unique( + d, DType::kBF16, + std::vector{static_cast(lw.mla.kv_a_layernorm.size())}); + vt::CastBf16(d.q, ln->t(), lnf.t()); + } + return MlaLayerPagedFa2(d, lw.mla, dhn.t(), p, T, kv, step, *ln); + }(); + DBuf dh2(d, DType::kF32, {T, H}); + AddRmsNormS(d, dh2, attn.t(), lw.post_attention_layernorm, H, res, eps, + DType::kF32); + DBuf mlp = lw.is_moe ? MoeBlockDeviceBf16(d, lw.moe, dh2.t(), p, T) + : DenseMlpDeviceBf16(d, lw.dense, dh2.t(), p, T); + auto* held = new DBuf(std::move(mlp)); + hcur = held->t(); + hold = std::shared_ptr(held, [](void* q) { delete static_cast(q); }); + } + + DBuf dnorm(d, DType::kF32, {T, H}); + AddRmsNormS(d, dnorm, hcur, rw.final_norm, H, res, eps, DType::kF32); + + Tensor src = dnorm.t(); + DBuf dgather(d, DType::kF32, + in.logits_indices.empty() + ? std::vector{1, 1} + : std::vector{ + static_cast(in.logits_indices.size()), H}); + if (!in.logits_indices.empty()) { + const size_t rb = static_cast(H) * sizeof(float); + char* dp = static_cast(dgather.ptr()); + const char* sp = static_cast(dnorm.ptr()); + for (size_t i = 0; i < in.logits_indices.size(); ++i) { + const int32_t idx = in.logits_indices[i]; + VT_CHECK(idx >= 0 && idx < T, "kimi paged: logits index out of range"); + d.b.Copy(d.q, dp + i * rb, sp + static_cast(idx) * rb, rb); + } + src = dgather.t(); + } + const int64_t n_out = src.shape[0]; + + const bool tied = p.tie_word_embeddings || rw.lm_head.Empty(); + const OwnedTensor& lm = tied ? rw.embed_tokens : rw.lm_head; + DBuf logits(d, DType::kF32, {n_out, V}); + GemmBf16(d, logits.t(), src, lm, V, H); + return logits; +} + } // namespace // ─── per-op device wrappers (host-in / host-out) — the per-op CPU gates ──────── @@ -1763,6 +2489,15 @@ ForwardLogits KimiLinearModel::ForwardPrefillIncremental( return WrapDeviceLogits(std::move(dlogits), n_out, p.vocab_size); } +// ─── ROW 7 — the shared-paged-runner fold (§20.3) — public entry ─────────────── +ForwardLogits KimiLinearModel::ForwardPaged(const ModelForwardInput& input, + const KimiLinearWeights& weights) { + Dev d{vt::GetBackend(input.queue.device.type), input.queue}; + DBuf dlogits = DeviceForwardBodyBf16Paged(d, weights, input); + const int64_t n_out = dlogits.t().shape[0]; + return WrapDeviceLogits(std::move(dlogits), n_out, weights.params.vocab_size); +} + ForwardLogits KimiLinearModel::ForwardDecodeStepIncremental( int32_t token, int64_t position, const KimiLinearWeights& weights, vt::Queue& queue, KimiDecodeCache& cache) { diff --git a/src/vllm/model_executor/models/kimi_linear_registry.cpp b/src/vllm/model_executor/models/kimi_linear_registry.cpp index 4ec262ae0..3ad2f8c80 100644 --- a/src/vllm/model_executor/models/kimi_linear_registry.cpp +++ b/src/vllm/model_executor/models/kimi_linear_registry.cpp @@ -67,9 +67,15 @@ std::unique_ptr LoadKimiLinearForCausalLM( if (source.safetensors == nullptr) { throw std::runtime_error("safetensors model source is empty"); } + // ROW 7 (§20.3): the ENGINE path loads the bf16-RESIDENT tower — NEVER the + // 183 GiB f32 MaterializeHost, which OOM-reboots the 119 GiB unified pool on + // the full 48.9B checkpoint (§13 pool math). With a load_queue (the engine's + // stage_on_load seam: CUDA context created BEFORE the weights — the GB10 load + // recipe) each large tensor is staged to d_dev and its host mirror released; + // with none (CPU / tests) the host bf16 bytes are kept and aliased. return std::make_unique( - registration, - LoadKimiLinearForCausalLMWeights(*source.safetensors, config)); + registration, LoadKimiLinearResidentBf16Weights(*source.safetensors, config, + source.load_queue)); } void PrepareKimiLinearForCausalLM(LoadedModel& model, const HfConfig& config, @@ -83,6 +89,17 @@ ForwardLogits ForwardKimiLinearForCausalLM(LoadedModel& model, const ModelForwardInput& input) { auto& kl = static_cast(model); const KimiLinearWeights& weights = kl.weights(); + // ROW 7 (§20.3): the RUNNER path — real paged caches supplied (the runner + // always hands its allocated attn_kv + gdn_state groups) with bf16-resident + // weights — routes through the shared-paged-runner fold: KDA state in the + // MambaSpec group, NoPE-MLA latent in the paged MLA group, one forward per + // scheduler step, device-resident logits for the on-GPU sampler. Direct + // callers with no paged caches (the CLI/unit vehicles) keep the historical + // seams below, byte-identical. + if (input.gather_logits && weights.resident.resident && !input.attn_kv.empty() && + !input.gdn_state.empty()) { + return KimiLinearModel::ForwardPaged(input, weights); + } if (input.gather_logits) { return KimiLinearModel::ForwardDevice(input.token_ids, input.positions, input.attn_meta, input.attn_kv, weights, @@ -102,6 +119,9 @@ const ModelFactory kKimiLinearFactory{ .forward = &ForwardKimiLinearForCausalLM, .make_kv_cache = &MakeKimiLinearKVCache, .is_dense_model = false, + // ROW 7 (§20.3): the engine selects the queue BEFORE loading so the 91.5 GiB + // bf16-resident tower stages per tensor into the CUDA context (§13 recipe). + .stage_on_load = true, }; } // namespace @@ -135,7 +155,10 @@ v1::KVCacheConfig MakeKimiLinearKVCache(const HfConfig& config, int block_size, std::vector>{ {p.kda_conv_dim(), p.kda_short_conv_kernel_size - 1}, {p.kda_num_heads, p.kda_head_dim, p.kda_head_dim}}, - std::vector{vt::DType::kBF16, vt::DType::kF32})); + // kda_state_dtype (mamba_utils.py:130-137): conv follows the CACHE + // dtype (model-dtype bf16 default; f32 under VT_KV_CACHE_F32 — the + // fold-identity A/B), recurrent state is always float32. + std::vector{v1::ResolveKvCacheDType(), vt::DType::kF32})); return kv; } diff --git a/src/vllm/model_executor/models/kimi_linear_weights.cpp b/src/vllm/model_executor/models/kimi_linear_weights.cpp index 4709763e4..288cccf5a 100644 --- a/src/vllm/model_executor/models/kimi_linear_weights.cpp +++ b/src/vllm/model_executor/models/kimi_linear_weights.cpp @@ -32,6 +32,7 @@ #include "vllm/model_executor/models/kimi_linear.h" #include +#include #include #include #include @@ -42,6 +43,7 @@ #include "vllm/model_executor/model_loader/safetensors_reader.h" #include "vllm/model_executor/models/dense_weight_loaders.h" // dense_loaders::LoadBf16Direct/MakeOwned +#include "vllm/model_executor/models/mla_attention.h" // mla::AbsorbKvBProjBf16 (ROW 7) #include "vllm/platforms/interface.h" // platforms::GetPlatform (is_cpu) #include "vt/backend.h" // vt::GetBackend (device staging) #include "vt/dtype.h" @@ -513,6 +515,43 @@ KimiLinearWeights LoadKimiLinearForCausalLMWeights( } // ─── bf16-RESIDENT loader + builder + stager (§13) ───────────────────────────── + +// ROW 7 (§20.3c): absorb kv_b_proj into the decode bmm forms W_UK_T/W_UV at LOAD +// time (mla::AbsorbKvBProjBf16 — mla_attention.py:875-962). Must run while the +// kv_b host bf16 bytes are live: the staging path releases them, so the caller +// loads kv_b WITHOUT staging, absorbs here, then stages all three. The absorbed +// forms power the paged-FA2 NoPE-MLA arm (mla::ForwardMlaAttentionBlock). +static void AbsorbKimiMla(MlaResidentWeights& m, const KimiLinearParams& p, + vt::Queue* stage_queue) { + mla::MlaBlockDims dm; + dm.hidden_size = p.hidden_size; + dm.num_heads = p.num_attention_heads; + dm.qk_nope_head_dim = p.qk_nope_head_dim; + dm.qk_rope_head_dim = p.qk_rope_head_dim; + dm.v_head_dim = p.v_head_dim; + dm.kv_lora_rank = p.kv_lora_rank; + dm.q_lora_rank = 0; // the no-q-lora branch (kimi_linear.py:214-215) + dm.rms_norm_eps = p.rms_norm_eps; + // NoPE: plain qk_head_dim^-0.5, no YaRN (mla_use_nope, rotary_emb=None) — + // kimi_linear.py:212 `self.scaling = self.qk_head_dim**-0.5`. + dm.scale = static_cast( + std::pow(static_cast(dm.qk_head_dim()), -0.5)); + VT_CHECK(m.kv_b_proj.HasHostBytes(), + "kimi resident: kv_b_proj host bytes must be live for absorption"); + const auto* w = reinterpret_cast(m.kv_b_proj.bytes.data()); + mla::AbsorbedKvBProj ab = mla::AbsorbKvBProjBf16(w, dm); + const auto own = [&](const std::vector& v, + const std::vector& shape) { + OwnedTensor o = dense_loaders::MakeOwned(vt::DType::kBF16, shape); + std::memcpy(o.bytes.data(), v.data(), v.size() * sizeof(uint16_t)); + if (stage_queue != nullptr) StageKimiResidentBf16(*stage_queue, o); + return o; + }; + m.w_uk_t = + own(ab.w_uk_t, {dm.num_heads, dm.qk_nope_head_dim, dm.kv_lora_rank}); + m.w_uv = own(ab.w_uv, {dm.num_heads, dm.kv_lora_rank, dm.v_head_dim}); +} + void StageKimiResidentBf16(vt::Queue& queue, const OwnedTensor& w) { if (w.d_dev || !w.HasHostBytes()) return; // already staged / nothing to stage // CPU: the device forward aliases the host bytes directly (host-pointer aliasing is @@ -595,7 +634,11 @@ KimiLinearWeights LoadKimiLinearResidentBf16Weights( m.q_proj = big(a + "q_proj.weight"); m.kv_a_proj_with_mqa = big(a + "kv_a_proj_with_mqa.weight"); m.kv_a_layernorm = vecf(a + "kv_a_layernorm.weight"); - m.kv_b_proj = big(a + "kv_b_proj.weight"); + // kv_b: load WITHOUT immediate staging so the host bf16 bytes are live for + // the W_UK_T/W_UV absorption (ROW 7 §20.3c), then stage it like the rest. + m.kv_b_proj = dense_loaders::LoadBf16Direct(get, a + "kv_b_proj.weight"); + AbsorbKimiMla(m, p, stage_queue); + if (stage_queue != nullptr) StageKimiResidentBf16(*stage_queue, m.kv_b_proj); m.o_proj = big(a + "o_proj.weight"); } @@ -679,6 +722,7 @@ KimiLinearResidentWeights BuildKimiResidentFromHost(const KimiLinearHostWeights& rm.kv_b_proj = Bf16OwnedFromF32(hm.kv_b_proj); rm.o_proj = Bf16OwnedFromF32(hm.o_proj); rm.kv_a_layernorm = hm.kv_a_layernorm; + AbsorbKimiMla(rm, p, /*stage_queue=*/nullptr); // ROW 7 §20.3c } rl.is_moe = hl.is_moe; if (hl.is_moe) { diff --git a/src/vllm/transformers_utils/hf_config.cpp b/src/vllm/transformers_utils/hf_config.cpp index b5cf702db..9c9bd9e99 100644 --- a/src/vllm/transformers_utils/hf_config.cpp +++ b/src/vllm/transformers_utils/hf_config.cpp @@ -395,6 +395,60 @@ HfConfig LoadHfConfig(const std::string& path) { cfg.linear_conv_kernel_dim = GetInt(text, "linear_conv_kernel_dim", 0); cfg.mamba_ssm_dtype = GetString(text, "mamba_ssm_dtype"); + // Kimi-Linear (`KimiLinearForCausalLM`) KV enablement for the shared paged + // runner (kimi-linear.md §20.3 B1). Kimi's config carries NO `layer_types` + // and NONE of the explicit qwen3_5-style `linear_*` keys read above: its + // KDA/full-attn split and GDN-group geometry live in the nested + // `linear_attn_config` (upstream transformers_utils/configs/kimi_linear.py + // :34-148; `is_kda_layer(l) := (l+1) in kda_layers` :144-148 — the layer + // lists are 1-INDEXED; `num_heads`/`head_dim`/`short_conv_kernel_size` + // :109-119 are what MambaStateShapeCalculator.kda_state_shape derives the + // conv/recurrent state from, mamba_utils.py:270-294). Synthesize the typed + // runner-facing fields from it so the runner's MambaSpec consistency check + // (runner.cpp `expected_conv_shape`/`expected_ssm_shape`) and its per-layer + // linear-attention/full-attention allocation loop see the same geometry + // `MakeKimiLinearKVCache` declares. ADDITIVE by construction: configs that + // carry the explicit fields (the qwen3_5/qwen3-next family) never enter — + // the explicit branch above already populated them — and configs with no + // `linear_attn_config` (every other arch) skip it entirely. KDA has + // num_k_heads == num_v_heads == num_heads and Dk == Dv == head_dim + // (kimi_gdn_linear_attn.py:120-141), so conv_dim = 2*Hk*Dk + Hv*Dv equals + // kda_state_shape's 3*num_heads*head_dim. + if (cfg.linear_num_key_heads == 0 && text.contains("linear_attn_config") && + text["linear_attn_config"].is_object()) { + const nlohmann::json& lac = text["linear_attn_config"]; + const int64_t kda_heads = GetInt(lac, "num_heads", 0); + const int64_t kda_head_dim = GetInt(lac, "head_dim", 0); + const int64_t kda_conv = GetInt(lac, "short_conv_kernel_size", 0); + const bool has_kda_layers = lac.contains("kda_layers") && + lac["kda_layers"].is_array() && + !lac["kda_layers"].empty(); + if (kda_heads > 0 && kda_head_dim > 0 && kda_conv > 0 && has_kda_layers) { + cfg.linear_num_key_heads = kda_heads; + cfg.linear_num_value_heads = kda_heads; + cfg.linear_key_head_dim = kda_head_dim; + cfg.linear_value_head_dim = kda_head_dim; + cfg.linear_conv_kernel_dim = kda_conv; + if (cfg.layer_types.empty() && cfg.num_hidden_layers > 0) { + std::vector is_kda(static_cast(cfg.num_hidden_layers), + false); + for (const nlohmann::json& e : lac["kda_layers"]) { + if (!e.is_number_integer()) continue; + const int64_t one_indexed = e.get(); + if (one_indexed >= 1 && one_indexed <= cfg.num_hidden_layers) { + is_kda[static_cast(one_indexed - 1)] = true; + } + } + cfg.layer_types.reserve(static_cast(cfg.num_hidden_layers)); + for (int64_t l = 0; l < cfg.num_hidden_layers; ++l) { + cfg.layer_types.push_back(is_kda[static_cast(l)] + ? "linear_attention" + : "full_attention"); + } + } + } + } + // Partial rotary factor. When the key is absent, upstream Qwen-family // config classes default it to 0.25 (qwen3_next.py:240, qwen3_5_moe.py:92); // all other models default to full rotary (1.0). The wrapper carries the diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 4f0610afa..fbfe8319b 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -186,6 +186,8 @@ vllm_cpp_add_test(test_kimi_linear_scaffold vllm/models/test_kimi_linear_scaffol # N tokens with the context advancing). The DEVICE forward stays refuse-by-name # (born-on-runner W6/W7). .agents/specs/kimi-linear.md §5. vllm_cpp_add_test(test_kimi_linear_forward vllm/models/test_kimi_linear_forward.cpp) +vllm_cpp_add_test(test_kimi_linear_paged vllm/models/test_kimi_linear_paged.cpp) +vllm_cpp_add_test(test_kimi_linear_fold_gate vllm/models/test_kimi_linear_fold_gate.cpp) # KDA (Kimi Delta Attention) net-new gated-linear-attention primitives — host # reference + unit gate (KERNEL-KDA-DELTA). Shared unblocker for Kimi-Linear-48B diff --git a/tests/capi/c_header_compile.c b/tests/capi/c_header_compile.c index 68435abdf..e5a332941 100644 --- a/tests/capi/c_header_compile.c +++ b/tests/capi/c_header_compile.c @@ -32,6 +32,13 @@ int vllm_capi_c_header_check(vllm_engine* eng, const char* prompt) { vllm_completion_free(&out); vllm_string_free(out.text); st = vllm_complete_stream(eng, prompt, &sp, cb, /*user_data=*/NULL); + { + const int32_t prompt_ids[1] = {1}; + int32_t out_ids[4]; + int32_t n_out = 0; + st = vllm_complete_tokens(eng, prompt_ids, 1, &sp, out_ids, 4, &n_out, + /*out=*/NULL); + } st = vllm_request_submit(eng, prompt, &sp, cb, /*user_data=*/NULL, &request); if (request != NULL) { diff --git a/tests/capi/test_capi.cpp b/tests/capi/test_capi.cpp index ffa0e2215..96df56eca 100644 --- a/tests/capi/test_capi.cpp +++ b/tests/capi/test_capi.cpp @@ -379,6 +379,72 @@ TEST_CASE("capi: two greedy completions of the same prompt are identical") { vllm_engine_free(eng); } +// ─── (b1b) ABI v13 pre-tokenized completion ────────────────────────────────── +TEST_CASE("capi: vllm_complete_tokens matches the string-prompt completion (ABI v13)") { + vllm_engine* eng = MakeSyntheticEngine(); + REQUIRE(eng != nullptr); + + vllm_sampling_params sp = GreedyParams(6); + // The string leg: "hello" tokenizes to the single id 13 in the synthetic + // tokenizer (see the vllm_complete greedy case above). + vllm_completion via_str; + REQUIRE(vllm_complete(eng, "hello", &sp, &via_str) == VLLM_OK); + + const int32_t prompt[1] = {13}; + int32_t out_tokens[16] = {0}; + int32_t n_out = -1; + vllm_completion via_tok; + const vllm_status st = + vllm_complete_tokens(eng, prompt, 1, &sp, out_tokens, 16, &n_out, &via_tok); + CHECK(st == VLLM_OK); + CHECK(n_out == 6); // greedy max_tokens, all reported + // Hand-pinned synthetic-model greedy stream. This is intentionally + // independent of the string leg: a broken implementation that merely + // reports six zero-initialized buffer entries must not satisfy ABI v12. + const int32_t expected_ids[6] = {22, 12, 14, 9, 13, 2}; + for (int i = 0; i < 6; ++i) { + INFO("generated token index ", i); + CHECK(out_tokens[i] == expected_ids[i]); + } + REQUIRE(via_tok.text != nullptr); + // Same engine, same greedy params, same (single-token) prompt => the SAME + // deterministic completion through both entry points. + CHECK(std::string(via_tok.text) == std::string(via_str.text)); + CHECK(via_tok.prompt_tokens == 1); + CHECK(via_tok.completion_tokens == 6); + REQUIRE(via_tok.finish_reason != nullptr); + CHECK(std::string(via_tok.finish_reason) == "length"); + + // A truncating buffer reports fewer ids but never changes the generation. + int32_t small[2] = {0}; + int32_t n_small = -1; + CHECK(vllm_complete_tokens(eng, prompt, 1, &sp, small, 2, &n_small, nullptr) == + VLLM_OK); + CHECK(n_small == 2); + CHECK(small[0] == out_tokens[0]); + CHECK(small[1] == out_tokens[1]); + + // Null contracts. + CHECK(vllm_complete_tokens(nullptr, prompt, 1, &sp, out_tokens, 16, &n_out, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_complete_tokens(eng, nullptr, 1, &sp, out_tokens, 16, &n_out, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_complete_tokens(eng, prompt, 0, &sp, out_tokens, 16, &n_out, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_complete_tokens(eng, prompt, 1, &sp, nullptr, 16, &n_out, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(vllm_complete_tokens(eng, prompt, 1, &sp, out_tokens, 16, nullptr, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + n_out = -1; + CHECK(vllm_complete_tokens(eng, prompt, 1, &sp, out_tokens, -1, &n_out, + nullptr) == VLLM_ERR_INVALID_ARGUMENT); + CHECK(n_out == 0); + + vllm_completion_free(&via_str); + vllm_completion_free(&via_tok); + vllm_engine_free(eng); +} + // ─── (b2) ABI v8 custom logits processor: forces a token end-to-end ────────── namespace { // Force-a-token processor: state carried through vllm_logits_processor_user_data. @@ -1228,7 +1294,7 @@ TEST_CASE("capi: version and abi-version are exposed") { // slice (vllm_video_*) is ABI v12. The >= pin is the one check that can // catch a WRONG bump: the == VLLM_ABI_VERSION assertions here and in // test_dlopen compare against the same macro and move with it. - CHECK(vllm_abi_version() >= 12); + CHECK(vllm_abi_version() >= 13); } // ─── ABI v11: audio transcription (ARCH-ONE-SURFACE ROW 1) ─────────────────── diff --git a/tests/capi/test_dlopen.cpp b/tests/capi/test_dlopen.cpp index 9bfae52a6..b50e4dccf 100644 --- a/tests/capi/test_dlopen.cpp +++ b/tests/capi/test_dlopen.cpp @@ -39,6 +39,9 @@ using fn_complete = vllm_status (*)(vllm_engine*, const char*, using fn_complete_stream = vllm_status (*)(vllm_engine*, const char*, const vllm_sampling_params*, vllm_token_callback, void*); +using fn_complete_tokens = vllm_status (*)(vllm_engine*, const int32_t*, int32_t, + const vllm_sampling_params*, int32_t*, + int32_t, int32_t*, vllm_completion*); using fn_request_submit = vllm_status (*)(vllm_engine*, const char*, const vllm_sampling_params*, vllm_token_callback, void*, @@ -83,6 +86,7 @@ TEST_CASE("dlopen: libvllm.so resolves the whole C ABI by name and drives it") { auto p_engine_free = Sym(lib, "vllm_engine_free"); auto p_complete = Sym(lib, "vllm_complete"); auto p_complete_stream = Sym(lib, "vllm_complete_stream"); + auto p_complete_tokens = Sym(lib, "vllm_complete_tokens"); auto p_request_submit = Sym(lib, "vllm_request_submit"); auto p_request_cancel = Sym(lib, "vllm_request_cancel"); auto p_request_wait = Sym(lib, "vllm_request_wait"); @@ -103,6 +107,7 @@ TEST_CASE("dlopen: libvllm.so resolves the whole C ABI by name and drives it") { CHECK(sp.repetition_penalty > 0.0f); // a zeroed struct would be invalid. (void)p_complete; (void)p_complete_stream; + (void)p_complete_tokens; (void)p_request_submit; (void)p_request_cancel; (void)p_request_wait; diff --git a/tests/scripts/test_check_surface_coverage.py b/tests/scripts/test_check_surface_coverage.py index cdb484d34..83b8397e5 100644 --- a/tests/scripts/test_check_surface_coverage.py +++ b/tests/scripts/test_check_surface_coverage.py @@ -330,10 +330,10 @@ def test_reaching_count_within_ratchet(self) -> None: reaching |= set(mod.internal_include_dir_grant_units(mod.read(mod.EXAMPLES_CMAKE))) self.assertLessEqual(len(reaching), mod.MAX_INTERNAL_REACHING) - def test_ratchet_ceiling_pinned_at_9(self) -> None: + def test_ratchet_ceiling_pinned_at_8(self) -> None: # EQUALITY pin: a ceiling bump (up OR down) must move this line + the ratchet claims in # the spec/state, so the change is test-visible and reviewed, never silent. - self.assertEqual(mod.MAX_INTERNAL_REACHING, 9) + self.assertEqual(mod.MAX_INTERNAL_REACHING, 8) def test_capability_green(self) -> None: cap_allow, allow_errors = mod.parse_allowlist(mod.read(mod.CAP_ALLOWLIST)) diff --git a/tests/vllm/models/test_kimi_linear_fold_gate.cpp b/tests/vllm/models/test_kimi_linear_fold_gate.cpp new file mode 100644 index 000000000..5191763dc --- /dev/null +++ b/tests/vllm/models/test_kimi_linear_fold_gate.cpp @@ -0,0 +1,240 @@ +// ROW 7 / kimi-linear.md §20.3 — the GB10 fold-gate REFERENCE leg. +// +// The CLI-incremental battery (bf16-resident load + ForwardPrefillIncremental / +// ForwardDecodeStepIncremental greedy over the §12 golden prompts) that +// `examples/kimi_linear_gen` used to run through internal headers. The example +// is now a thin public-ABI client (ONE SURFACE B4), so the REFERENCE leg — the +// stream the engine-paged path must reproduce — lives here, where internal +// headers are legitimate. ENV-GATED: skipped unless VT_KIMI_MODEL_DIR points at +// the real 48.9B snapshot (a 91.5 GiB load has no place in the CPU suite); +// VT_KIMI_GOLDEN_DIR selects the §12 golden and is REQUIRED whenever the model +// gate is enabled. This executable is only the CLI-incremental REFERENCE leg; +// it does not stand in for the still-owed engine/C-ABI golden or HTTP server +// smoke on the 91.5 GiB checkpoint. +// +// GB10 usage (the §19-winning config is the reference regime): with +// VT_KIMI_DEVICE_COMPUTE=1 VT_KIMI_DEVICE_KDA=1 VT_KIMI_DEVICE_KDA_CHUNK=1 +// VT_KIMI_MODEL_DIR=... VT_KIMI_GOLDEN_DIR=... VT_KIMI_STEPS=16 +// run ./test_kimi_linear_fold_gate +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/kimi_linear.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vt/backend.h" +#include "vt/device.h" + +namespace fs = std::filesystem; + +namespace { + +std::vector OpenSafetensorsDir(const std::string& dir) { + std::vector paths; + for (const auto& e : fs::directory_iterator(dir)) + if (e.is_regular_file() && e.path().extension() == ".safetensors") + paths.push_back(e.path().string()); + REQUIRE(!paths.empty()); + std::sort(paths.begin(), paths.end()); + std::vector shards; + shards.reserve(paths.size()); + for (const std::string& p : paths) shards.push_back(vllm::SafetensorsFile::Open(p)); + return shards; +} + +std::vector ReadI32File(const std::string& path) { + std::ifstream f(path, std::ios::binary); + std::vector out; + if (!f) return out; + int32_t v = 0; + while (f.read(reinterpret_cast(&v), 4)) out.push_back(v); + return out; +} + +struct NpyInts { + std::vector shape; + std::vector data; +}; +NpyInts ReadNpyInts(const std::string& path) { + std::ifstream f(path, std::ios::binary); + REQUIRE(static_cast(f)); + char magic[6]; + f.read(magic, 6); + REQUIRE(std::memcmp(magic, "\x93NUMPY", 6) == 0); + unsigned char ver[2]; + f.read(reinterpret_cast(ver), 2); + uint16_t hlen = 0; + f.read(reinterpret_cast(&hlen), 2); + std::string header(hlen, '\0'); + f.read(header.data(), hlen); + const bool i8 = header.find("= dims.size()) break; + size_t j = i; + while (j < dims.size() && dims[j] >= '0' && dims[j] <= '9') ++j; + if (j > i) out.shape.push_back(std::atoll(dims.substr(i, j - i).c_str())); + i = j + 1; + } + int64_t n = 1; + for (int64_t d : out.shape) n *= d; + out.data.resize(static_cast(n)); + for (int64_t k = 0; k < n; ++k) { + if (i8) { + int64_t v = 0; + f.read(reinterpret_cast(&v), 8); + out.data[static_cast(k)] = v; + } else { + int32_t v = 0; + f.read(reinterpret_cast(&v), 4); + out.data[static_cast(k)] = v; + } + } + return out; +} + +} // namespace + +TEST_CASE("kimi fold-gate environment enables model and golden together") { + const bool have_model = std::getenv("VT_KIMI_MODEL_DIR") != nullptr; + const bool have_golden = std::getenv("VT_KIMI_GOLDEN_DIR") != nullptr; + INFO("the 91.5 GiB gate needs both VT_KIMI_MODEL_DIR and VT_KIMI_GOLDEN_DIR"); + CHECK(have_model == have_golden); +} + +TEST_CASE("kimi fold-gate reference: CLI-incremental battery on the real checkpoint" * + doctest::skip(std::getenv("VT_KIMI_MODEL_DIR") == nullptr)) { + const char* model_dir = std::getenv("VT_KIMI_MODEL_DIR"); + REQUIRE(model_dir != nullptr); + const char* golden_dir = std::getenv("VT_KIMI_GOLDEN_DIR"); + const char* steps_env = std::getenv("VT_KIMI_STEPS"); + const char* prompts_env = std::getenv("VT_KIMI_PROMPTS"); + const int steps = steps_env != nullptr ? std::atoi(steps_env) : 16; + const int prompts = prompts_env != nullptr ? std::atoi(prompts_env) : 8; + REQUIRE(golden_dir != nullptr); + REQUIRE(steps > 1); + REQUIRE(prompts > 0); + + // CUDA when available (the GB10 leg; context BEFORE weights — the §13 recipe), + // CPU otherwise. + vt::Queue q{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; + bool cuda = false; + try { + q = vt::GetBackend(vt::DeviceType::kCUDA).CreateQueue(); + cuda = true; + } catch (...) { + } + std::fprintf(stderr, "[fold-gate] device=%s\n", cuda ? "CUDA" : "CPU"); + + const vllm::HfConfig config = + vllm::LoadHfConfig((fs::path(model_dir) / "config.json").string()); + std::vector shards = OpenSafetensorsDir(model_dir); + const auto t0 = std::chrono::steady_clock::now(); + vllm::KimiLinearWeights w = + vllm::LoadKimiLinearResidentBf16Weights(shards, config, cuda ? &q : nullptr); + shards.clear(); + shards.shrink_to_fit(); // release the mmap'd shards (§13) + const auto t1 = std::chrono::steady_clock::now(); + std::fprintf(stderr, "[fold-gate] loaded bf16-resident in %.1fs\n", + std::chrono::duration(t1 - t0).count()); + + const NpyInts gold = + ReadNpyInts((fs::path(golden_dir) / "greedy_ids.npy").string()); + REQUIRE(gold.shape.size() == 2); + REQUIRE(gold.shape[0] >= prompts); + REQUIRE(gold.shape[1] >= steps); + + vt::Backend& be = vt::GetBackend(q.device.type); + const int64_t V = w.params.vocab_size; + auto argmax_row = [&](const vllm::ForwardLogits& fl) { + std::vector row(static_cast(V)); + be.Copy(q, row.data(), fl.device_tensor.data, row.size() * sizeof(float)); + be.Synchronize(q); + int best = 0; + float bv = row[0]; + for (int64_t o = 1; o < V; ++o) + if (row[static_cast(o)] > bv) { + bv = row[static_cast(o)]; + best = static_cast(o); + } + return best; + }; + + int total = 0, matched = 0; + int processed_prompts = 0; + int steady_steps = 0; + double steady_s = 0.0; + for (int pi = 0; pi < prompts; ++pi) { + std::vector prompt; + prompt = ReadI32File( + (fs::path(golden_dir) / ("p" + std::to_string(pi) + "_prompt.i32")).string()); + REQUIRE_FALSE(prompt.empty()); + ++processed_prompts; + + vllm::KimiDecodeCache cache; + std::vector positions(prompt.size()); + for (size_t t = 0; t < prompt.size(); ++t) positions[t] = static_cast(t); + const std::vector li = {static_cast(prompt.size() - 1)}; + vllm::ForwardLogits fl = + vllm::KimiLinearModel::ForwardPrefillIncremental(prompt, positions, w, q, cache, li); + std::vector gen; + int best = argmax_row(fl); + gen.push_back(best); + for (int s = 1; s < steps; ++s) { + const auto ts = std::chrono::steady_clock::now(); + vllm::ForwardLogits d = + vllm::KimiLinearModel::ForwardDecodeStepIncremental(best, cache.seq_len, w, q, cache); + best = argmax_row(d); + const auto te = std::chrono::steady_clock::now(); + steady_s += std::chrono::duration(te - ts).count(); + ++steady_steps; + gen.push_back(best); + } + + std::string got; + for (int t = 0; t < steps; ++t) + got += std::to_string(gen[static_cast(t)]) + (t + 1 < steps ? "," : ""); + const int64_t T = gold.shape[1]; + int row_match = 0; + for (int t = 0; t < steps; ++t) { + ++total; + if (gen[static_cast(t)] == + static_cast(gold.data[static_cast(pi) * T + t])) { + ++matched; + ++row_match; + } + } + std::fprintf(stderr, "[fold-gate] p%d: %d/%d vs golden | got: %s\n", pi, + row_match, steps, got.c_str()); + } + std::fprintf(stderr, "[fold-gate] TOKEN MATCH %d/%d\n", matched, total); + std::fprintf(stderr, "[fold-gate] steady %.3f s/step (%.2f tok/s) over %d steps\n", + steady_s / steady_steps, steady_steps / steady_s, steady_steps); + + REQUIRE(processed_prompts == prompts); + REQUIRE(total == prompts * steps); + REQUIRE(steady_steps == prompts * (steps - 1)); + REQUIRE(steady_s > 0.0); + // Preserve the last independently measured coherent reference floor. For + // the canonical 8x16 battery this is 122/128; scaling the integer inequality + // keeps deliberate smaller diagnostic subsets honest without rounding up. + CHECK(static_cast(matched) * 128 >= + static_cast(total) * 122); +} diff --git a/tests/vllm/models/test_kimi_linear_paged.cpp b/tests/vllm/models/test_kimi_linear_paged.cpp new file mode 100644 index 000000000..5510557c7 --- /dev/null +++ b/tests/vllm/models/test_kimi_linear_paged.cpp @@ -0,0 +1,945 @@ +// ROW 7 / kimi-linear.md §20.3 — the shared-paged-runner fold gates (CPU). +// +// Kimi-Linear folded onto the SHARED paged runner (`ModelRegistry::Forward` → +// `KimiLinearModel::ForwardPaged`): the KDA conv+recurrent state lives in the +// runner's MambaSpec `gdn_state` group and the NoPE-MLA latent-KV in the paged +// MLA `attn_kv` group. Cases: +// (a) B1 KV enablement — a REAL Kimi config.json (no `layer_types`, no +// explicit `linear_*` keys) loads through LoadHfConfig, the registry +// resolves + loads the bf16-resident tower, and GPUModelRunner ALLOCATES +// the two het-KV groups from MakeKimiLinearKVCache without aborting +// (pre-fold: VT_CHECK at runner.cpp `expected_conv_shape` fails on +// {0,0},{0,0,0}). +// (b) FOLD IDENTITY — the paged-runner greedy decode (prefill once + N +// single-token decode steps through execute_model/sample_tokens) emits +// the SAME tokens as the CLI-incremental reference +// (ForwardPrefillIncremental / ForwardDecodeStepIncremental) over the +// same resident weights. Run under VT_KV_CACHE_F32=1 so every paged +// cache (MLA latent rows, kpe, conv taps) carries the f32 values the CLI +// reference carries — any token difference is a state-carry/paging bug, +// not a rounding one. This is the CPU half of the §20.3 Gate A; the GB10 +// real-checkpoint battery is the binding on-box gate. +// (c) SLOT ISOLATION — a 2-request batched decode produces, per request, the +// same tokens as that request's own single-request run (the compact GDN +// state slots + per-request MLA block tables do not cross-talk). +// +// The synthetic 2-layer checkpoint (layer0 KDA+dense, layer1 NoPE-MLA+MoE) +// mirrors tests/vllm/models/test_kimi_linear_forward.cpp's builder. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include + +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vllm/model_executor/models/kimi_linear.h" +#include "vllm/model_executor/models/model_registry.h" +#include "vllm/sampling_params.h" +#include "vllm/transformers_utils/hf_config.h" +#include "vllm/v1/attention/backends/gdn_attn.h" +#include "vllm/v1/core/sched/output.h" +#include "vllm/v1/kv_cache_interface.h" +#include "vllm/v1/worker/gpu/runner.h" +#include "vt/backend.h" +#include "vt/dtype.h" + +using vllm::ForwardLogits; +using vllm::HfConfig; +using vllm::KimiDecodeCache; +using vllm::KimiLinearModel; +using vllm::KimiLinearWeights; +using vllm::LoadKimiLinearResidentBf16Weights; +using vllm::ModelRegistry; +using vllm::ModelSource; +using vllm::SafetensorsFile; +using vllm::SamplingParams; +using vllm::v1::CachedRequestData; +using vllm::v1::GPUModelRunner; +using vllm::v1::KVCacheConfig; +using vllm::v1::MambaSpec; +using vllm::v1::NewRequestData; +using vllm::v1::SchedulerOutput; +using vt::DType; + +namespace { + +// ─── synthetic 2-layer checkpoint (mirror test_kimi_linear_forward.cpp) ─────── +constexpr int H = 32, NAH = 4, V = 8, DENSE_I = 64, MOE_I = 16, E = 2; +constexpr int KV_LORA = 16, QK_NOPE = 8, QK_ROPE = 4, V_HEAD = 8; +constexpr int KDA_NH = 4, KDA_HD = 8, CONV = 4; +constexpr int KDA_PROJ = KDA_NH * KDA_HD; +constexpr int MLA_QK = QK_NOPE + QK_ROPE; + +struct Fx { std::string name, dtype; std::vector shape; std::string bytes; }; +std::string U64Le(uint64_t v) { + std::string s(8, '\0'); + for (int i = 0; i < 8; ++i) s[static_cast(i)] = static_cast((v >> (8 * i)) & 0xff); + return s; +} +int64_t NumEl(const std::vector& s) { + int64_t n = 1; + for (int64_t d : s) n *= d; + return n; +} +std::string Bf16Bytes(size_t n, int seed) { + std::string s(n * 2, '\0'); + uint32_t r = static_cast(seed) * 2654435761u + 1u; + for (size_t i = 0; i < n; ++i) { + r = r * 1664525u + 1013904223u; + const float u = static_cast(r >> 8) / static_cast(1u << 24); + const float f = (u - 0.5f) * 0.25f; + uint16_t bf = static_cast(vt::F32ToBF16(f)); + s[i * 2] = static_cast(bf & 0xff); + s[i * 2 + 1] = static_cast((bf >> 8) & 0xff); + } + return s; +} +std::string F32Bytes(size_t n, int seed) { + std::string s(n * 4, '\0'); + uint32_t r = static_cast(seed) * 2246822519u + 1u; + for (size_t i = 0; i < n; ++i) { + r = r * 1664525u + 1013904223u; + const float u = static_cast(r >> 8) / static_cast(1u << 24); + const float f = (u - 0.5f) * 0.25f; + std::memcpy(&s[i * 4], &f, 4); + } + return s; +} +Fx Bf16(const std::string& n, std::vector sh, int seed) { + return {n, "BF16", sh, Bf16Bytes(static_cast(NumEl(sh)), seed)}; +} +Fx F32(const std::string& n, std::vector sh, int seed) { + return {n, "F32", sh, F32Bytes(static_cast(NumEl(sh)), seed)}; +} +std::string BuildSt(const std::vector& ts) { + nlohmann::json hdr = nlohmann::json::object(); + std::string data; + for (const Fx& t : ts) { + const size_t start = data.size(); + data += t.bytes; + hdr[t.name] = {{"dtype", t.dtype}, {"shape", t.shape}, + {"data_offsets", {start, data.size()}}}; + } + const std::string header = hdr.dump(); + return U64Le(header.size()) + header + data; +} +class TempFile { + public: + explicit TempFile(const std::string& bytes, const char* ext = ".safetensors") { + // Unique PER PROCESS (pid + counter): two concurrent ctest processes with a + // fixed /tmp/kimi_paged_ name would overwrite each other's fixture + // mid-read (the cross-process collision hazard). + static int c = 0; + path_ = (std::filesystem::temp_directory_path() / + ("kimi_paged_" + std::to_string(::getpid()) + "_" + + std::to_string(c++) + ext)).string(); + std::ofstream out(path_, std::ios::binary); + out.write(bytes.data(), static_cast(bytes.size())); + } + ~TempFile() { std::remove(path_.c_str()); } + const std::string& path() const { return path_; } + private: + std::string path_; +}; + +std::vector BuildTensors() { + std::vector v; + int s = 1; + v.push_back(Bf16("model.embed_tokens.weight", {V, H}, s++)); + v.push_back(Bf16("model.norm.weight", {H}, s++)); + v.push_back(Bf16("lm_head.weight", {V, H}, s++)); + const std::string a0 = "model.layers.0."; + v.push_back(Bf16(a0 + "input_layernorm.weight", {H}, s++)); + v.push_back(Bf16(a0 + "post_attention_layernorm.weight", {H}, s++)); + const std::string k = a0 + "self_attn."; + v.push_back(Bf16(k + "q_proj.weight", {KDA_PROJ, H}, s++)); + v.push_back(Bf16(k + "k_proj.weight", {KDA_PROJ, H}, s++)); + v.push_back(Bf16(k + "v_proj.weight", {KDA_PROJ, H}, s++)); + v.push_back(Bf16(k + "f_a_proj.weight", {KDA_HD, H}, s++)); + v.push_back(Bf16(k + "f_b_proj.weight", {KDA_PROJ, KDA_HD}, s++)); + v.push_back(Bf16(k + "b_proj.weight", {KDA_NH, H}, s++)); + v.push_back(Bf16(k + "g_a_proj.weight", {KDA_HD, H}, s++)); + v.push_back(Bf16(k + "g_b_proj.weight", {KDA_PROJ, KDA_HD}, s++)); + v.push_back(Bf16(k + "o_proj.weight", {H, KDA_PROJ}, s++)); + v.push_back(F32(k + "q_conv1d.weight", {KDA_PROJ, 1, CONV}, s++)); + v.push_back(F32(k + "k_conv1d.weight", {KDA_PROJ, 1, CONV}, s++)); + v.push_back(F32(k + "v_conv1d.weight", {KDA_PROJ, 1, CONV}, s++)); + v.push_back(F32(k + "dt_bias", {KDA_PROJ}, s++)); + v.push_back(F32(k + "A_log", {KDA_NH}, s++)); + v.push_back(Bf16(k + "o_norm.weight", {KDA_HD}, s++)); + v.push_back(Bf16(a0 + "mlp.gate_proj.weight", {DENSE_I, H}, s++)); + v.push_back(Bf16(a0 + "mlp.up_proj.weight", {DENSE_I, H}, s++)); + v.push_back(Bf16(a0 + "mlp.down_proj.weight", {H, DENSE_I}, s++)); + const std::string a1 = "model.layers.1."; + v.push_back(Bf16(a1 + "input_layernorm.weight", {H}, s++)); + v.push_back(Bf16(a1 + "post_attention_layernorm.weight", {H}, s++)); + const std::string m = a1 + "self_attn."; + v.push_back(Bf16(m + "q_proj.weight", {NAH * MLA_QK, H}, s++)); + v.push_back(Bf16(m + "kv_a_proj_with_mqa.weight", {KV_LORA + QK_ROPE, H}, s++)); + v.push_back(Bf16(m + "kv_a_layernorm.weight", {KV_LORA}, s++)); + v.push_back(Bf16(m + "kv_b_proj.weight", {NAH * (QK_NOPE + V_HEAD), KV_LORA}, s++)); + v.push_back(Bf16(m + "o_proj.weight", {H, NAH * V_HEAD}, s++)); + const std::string mo = a1 + "block_sparse_moe."; + v.push_back(Bf16(mo + "gate.weight", {E, H}, s++)); + v.push_back(F32(mo + "gate.e_score_correction_bias", {E}, s++)); + v.push_back(Bf16(mo + "shared_experts.gate_proj.weight", {MOE_I, H}, s++)); + v.push_back(Bf16(mo + "shared_experts.up_proj.weight", {MOE_I, H}, s++)); + v.push_back(Bf16(mo + "shared_experts.down_proj.weight", {H, MOE_I}, s++)); + for (int e = 0; e < E; ++e) { + const std::string ep = mo + "experts." + std::to_string(e) + "."; + v.push_back(Bf16(ep + "w1.weight", {MOE_I, H}, s++)); + v.push_back(Bf16(ep + "w2.weight", {H, MOE_I}, s++)); + v.push_back(Bf16(ep + "w3.weight", {MOE_I, H}, s++)); + } + return v; +} + +// The REAL Kimi config shape: NO layer_types, NO explicit linear_* keys — the +// KDA split lives only in linear_attn_config (B1's synthesis is what makes the +// runner allocation possible). +std::string TinyConfigJson() { + nlohmann::json j = { + {"model_type", "kimi_linear"}, + {"architectures", {"KimiLinearForCausalLM"}}, + {"hidden_size", H}, + {"num_hidden_layers", 2}, + {"vocab_size", V}, + {"num_attention_heads", NAH}, + {"num_key_value_heads", NAH}, + {"intermediate_size", DENSE_I}, + {"rms_norm_eps", 1e-5}, + {"tie_word_embeddings", false}, + {"num_experts", E}, + {"num_experts_per_token", 1}, + {"num_shared_experts", 1}, + {"moe_intermediate_size", MOE_I}, + {"first_k_dense_replace", 1}, + {"moe_layer_freq", 1}, + {"moe_router_activation_func", "sigmoid"}, + {"routed_scaling_factor", 2.446}, + {"kv_lora_rank", KV_LORA}, + {"qk_nope_head_dim", QK_NOPE}, + {"qk_rope_head_dim", QK_ROPE}, + {"v_head_dim", V_HEAD}, + {"mla_use_nope", true}, + {"max_position_embeddings", 64}, + {"linear_attn_config", + {{"kda_layers", nlohmann::json::array({1})}, + {"full_attn_layers", nlohmann::json::array({2})}, + {"num_heads", KDA_NH}, + {"head_dim", KDA_HD}, + {"short_conv_kernel_size", CONV}}}, + }; + return j.dump(); +} + +constexpr int kBlockSize = 8; +constexpr int kNumBlocks = 8; +constexpr int kMaxModelLen = 32; + +vt::Queue Q() { return vt::Queue{vt::Device{vt::DeviceType::kCPU, 0}, nullptr}; } + +SamplingParams Greedy() { + SamplingParams sp; + sp.temperature = 0.0; + sp.PostInit(); + return sp; +} + +NewRequestData MakeNewReq(const std::string& id, std::vector prompt, + std::vector fa_blocks, int gdn_block) { + NewRequestData nr; + nr.req_id = id; + nr.prompt_token_ids = prompt; + nr.sampling_params = Greedy(); + // Kimi group order (MakeKimiLinearKVCache): group 0 = MLA latent, group 1 = KDA. + nr.block_ids = {std::move(fa_blocks), std::vector{gdn_block}}; + nr.num_computed_tokens = 0; + nr.prefill_token_ids = std::move(prompt); + return nr; +} + +SchedulerOutput NewStep(std::vector new_reqs, + std::map scheduled) { + SchedulerOutput so; + so.scheduled_cached_reqs = CachedRequestData::make_empty(); + so.scheduled_new_reqs = std::move(new_reqs); + int total = 0; + for (const auto& [id, n] : scheduled) total += n; + so.num_scheduled_tokens = std::move(scheduled); + so.total_num_scheduled_tokens = total; + return so; +} + +SchedulerOutput DecodeStep(const std::vector& ids, + const std::vector& num_computed, + const std::vector& num_output) { + SchedulerOutput so; + CachedRequestData cached; + cached.req_ids = ids; + for (size_t i = 0; i < ids.size(); ++i) { + cached.num_computed_tokens.push_back(num_computed[i]); + cached.num_output_tokens.push_back(num_output[i]); + cached.new_block_ids.emplace_back(std::nullopt); + } + so.scheduled_cached_reqs = std::move(cached); + for (const std::string& id : ids) so.num_scheduled_tokens[id] = 1; + so.total_num_scheduled_tokens = static_cast(ids.size()); + return so; +} + +// Greedy tokens from the paged runner for one request set. Returns tokens per +// request id, in `prompts` order. +std::vector> RunnerGreedy( + const HfConfig& cfg, vllm::LoadedModel& model, + const std::vector>& prompts, int steps) { + const vllm::ModelRegistration& reg = model.registration(); + KVCacheConfig kv = reg.factory->make_kv_cache(cfg, kBlockSize, kNumBlocks); + GPUModelRunner runner(cfg, model, kv, Q(), /*max_num_reqs=*/4, kMaxModelLen, + /*max_num_batched_tokens=*/64); + + // Admit all prompts in one prefill step (disjoint FA blocks + GDN slots). + std::vector reqs; + std::map sched; + for (size_t i = 0; i < prompts.size(); ++i) { + const std::string id = "R" + std::to_string(i); + const int b0 = static_cast(i) * 2; + reqs.push_back(MakeNewReq(id, prompts[i], {b0, b0 + 1}, static_cast(i))); + sched[id] = static_cast(prompts[i].size()); + } + SchedulerOutput s1 = NewStep(std::move(reqs), std::move(sched)); + CHECK_FALSE(runner.execute_model(s1).has_value()); + vllm::v1::ModelRunnerOutput m1 = runner.sample_tokens(std::nullopt); + REQUIRE(m1.sampled_token_ids.size() == prompts.size()); + + std::vector> out(prompts.size()); + std::vector computed(prompts.size()), outputs(prompts.size(), 1); + std::vector ids; + for (size_t i = 0; i < prompts.size(); ++i) { + // sample order == admission order in this single-batch harness. + REQUIRE(m1.sampled_token_ids[i].size() == 1); + out[i].push_back(m1.sampled_token_ids[i][0]); + computed[static_cast(i)] = static_cast(prompts[i].size()); + ids.push_back("R" + std::to_string(i)); + } + for (int s = 1; s < steps; ++s) { + SchedulerOutput sd = DecodeStep(ids, computed, outputs); + CHECK_FALSE(runner.execute_model(sd).has_value()); + vllm::v1::ModelRunnerOutput md = runner.sample_tokens(std::nullopt); + REQUIRE(md.sampled_token_ids.size() == prompts.size()); + for (size_t i = 0; i < prompts.size(); ++i) { + REQUIRE(md.sampled_token_ids[i].size() == 1); + out[i].push_back(md.sampled_token_ids[i][0]); + computed[i] += 1; + outputs[i] += 1; + } + } + return out; +} + +// The CLI-incremental reference leg (the §19 18.9 tok/s vehicle): prefill once, +// then N carried decode steps, greedy host argmax. +std::vector CliIncrementalGreedy(const KimiLinearWeights& w, + const std::vector& prompt, + int steps) { + vt::Queue q = Q(); + vt::Backend& be = vt::GetBackend(q.device.type); + const int64_t vocab = w.params.vocab_size; + auto argmax_row = [&](const ForwardLogits& fl) { + std::vector row(static_cast(vocab)); + be.Copy(q, row.data(), fl.device_tensor.data, row.size() * sizeof(float)); + be.Synchronize(q); + int best = 0; + float bv = row[0]; + for (int64_t o = 1; o < vocab; ++o) + if (row[static_cast(o)] > bv) { + bv = row[static_cast(o)]; + best = static_cast(o); + } + return best; + }; + KimiDecodeCache cache; + std::vector positions(prompt.size()); + for (size_t t = 0; t < prompt.size(); ++t) positions[t] = static_cast(t); + const std::vector li = {static_cast(prompt.size() - 1)}; + ForwardLogits fl = + KimiLinearModel::ForwardPrefillIncremental(prompt, positions, w, q, cache, li); + std::vector gen; + int best = argmax_row(fl); + gen.push_back(best); + for (int s = 1; s < steps; ++s) { + ForwardLogits dfl = KimiLinearModel::ForwardDecodeStepIncremental( + best, cache.seq_len, w, q, cache); + best = argmax_row(dfl); + gen.push_back(best); + } + return gen; +} + +struct Fixture { + std::unique_ptr st; + std::unique_ptr cfg_json; + std::vector shards; + HfConfig cfg; + Fixture() { + st = std::make_unique(BuildSt(BuildTensors())); + cfg_json = std::make_unique(TinyConfigJson(), ".json"); + shards.push_back(SafetensorsFile::Open(st->path())); + cfg = vllm::LoadHfConfig(cfg_json->path()); + } +}; + +} // namespace + +// ─── (a) B1 — the runner allocates Kimi's het-KV groups (no abort) ──────────── +TEST_CASE("kimi paged: runner allocates MLA latent + KDA state groups from a real config") { + Fixture fx; + // The B1 synthesis populated the runner-facing geometry. + REQUIRE(fx.cfg.layer_types.size() == 2); + CHECK(fx.cfg.layer_types[0] == "linear_attention"); + CHECK(fx.cfg.layer_types[1] == "full_attention"); + CHECK(fx.cfg.linear_num_key_heads == KDA_NH); + CHECK(fx.cfg.linear_conv_kernel_dim == CONV); + + const vllm::ModelRegistration& reg = ModelRegistry::Resolve(fx.cfg); + std::unique_ptr model = + ModelRegistry::Load(fx.cfg, ModelSource::FromSafetensors(fx.shards)); + KVCacheConfig kv = reg.factory->make_kv_cache(fx.cfg, kBlockSize, kNumBlocks); + REQUIRE(kv.kv_cache_groups.size() == 2); + + // Pre-fold this constructor ABORTED (runner.cpp MambaSpec shape check against + // config-derived {0,0},{0,0,0}); with B1 it allocates. + GPUModelRunner runner(fx.cfg, *model, kv, Q(), /*max_num_reqs=*/4, kMaxModelLen, + /*max_num_batched_tokens=*/64); + CHECK(runner.full_attn_group_id() == 0); + CHECK(runner.gdn_group_id() == 1); + // 1 MLA layer page (num_kv_heads==1, 576-analog width) + 1 KDA state group. + REQUIRE(runner.attn_kv().size() == 1); + CHECK(runner.attn_kv()[0].num_kv_heads == 1); + CHECK(runner.attn_kv()[0].head_size == KV_LORA + QK_ROPE); + REQUIRE(runner.gdn_state().size() == 1); + CHECK(runner.gdn_state()[0].conv_state.shape[1] == 3 * KDA_PROJ); + CHECK(runner.gdn_state()[0].conv_state.shape[2] == CONV - 1); + CHECK(runner.gdn_state()[0].ssm_state.shape[1] == KDA_NH); + CHECK(runner.gdn_state()[0].ssm_state.shape[2] == KDA_HD); + CHECK(runner.gdn_state()[0].ssm_state.shape[3] == KDA_HD); +} + +// ─── (b) FOLD IDENTITY — paged-runner greedy == CLI-incremental greedy ──────── +TEST_CASE("kimi paged: runner decode tokens == CLI incremental reference (f32 caches)") { + // f32 caches: the paged rows carry the same f32 values the CLI reference + // carries, so identity is exact-by-construction (any diff = a paging bug). + // The EXACT MLA arm is the fold-identity vehicle (the production default is + // the FA2 arm, whose GB10 profile gate lives in §21). + setenv("VT_KV_CACHE_F32", "1", 1); + setenv("VT_KIMI_PAGED_MLA_FA2", "0", 1); + Fixture fx; + std::unique_ptr model = + ModelRegistry::Load(fx.cfg, ModelSource::FromSafetensors(fx.shards)); + + const std::vector prompt = {5, 1, 2, 7, 3}; + const int steps = 8; + const std::vector> paged = + RunnerGreedy(fx.cfg, *model, {prompt}, steps); + + KimiLinearWeights cli_w = + LoadKimiLinearResidentBf16Weights(fx.shards, fx.cfg, nullptr); + const std::vector cli = CliIncrementalGreedy(cli_w, prompt, steps); + + REQUIRE(paged.size() == 1); + REQUIRE(paged[0].size() == static_cast(steps)); + REQUIRE(cli.size() == static_cast(steps)); + for (int s = 0; s < steps; ++s) { + INFO("step ", s, " paged=", paged[0][static_cast(s)], + " cli=", cli[static_cast(s)]); + CHECK(paged[0][static_cast(s)] == cli[static_cast(s)]); + } + unsetenv("VT_KV_CACHE_F32"); + unsetenv("VT_KIMI_PAGED_MLA_FA2"); +} + +// The production regime (bf16 latent/conv caches — vLLM's own cache dtype): the +// tiny random model's argmax margins survive the cache rounding, so the token +// stream still matches the CLI reference. +TEST_CASE("kimi paged: runner decode tokens == CLI reference (production bf16 caches)") { + Fixture fx; + std::unique_ptr model = + ModelRegistry::Load(fx.cfg, ModelSource::FromSafetensors(fx.shards)); + const std::vector prompt = {2, 6, 4, 1}; + const int steps = 6; + const std::vector> paged = + RunnerGreedy(fx.cfg, *model, {prompt}, steps); + KimiLinearWeights cli_w = + LoadKimiLinearResidentBf16Weights(fx.shards, fx.cfg, nullptr); + const std::vector cli = CliIncrementalGreedy(cli_w, prompt, steps); + REQUIRE(paged.size() == 1); + for (int s = 0; s < steps; ++s) { + INFO("step ", s); + CHECK(paged[0][static_cast(s)] == cli[static_cast(s)]); + } +} + +// B3's shared MLA arm must execute, not merely compile behind a default-off +// switch. The first half runs the real absorbed-MQA/FA2 block over bf16 pages +// and requires its greedy stream to agree with the exact-island fold vehicle. +// The second half deliberately gives that arm an f32 page: its arm-specific +// dtype contract must reject it, so mutating the dispatch back to the exact +// island cannot leave this test green. +TEST_CASE("kimi paged: shared MLA arm runs and preserves the tiny greedy stream") { + Fixture fx; + std::unique_ptr model = + ModelRegistry::Load(fx.cfg, ModelSource::FromSafetensors(fx.shards)); + const std::vector prompt = {2, 6, 4, 1}; + const int steps = 4; + + unsetenv("VT_KIMI_PAGED_MLA_FA2"); + unsetenv("VT_KV_CACHE_F32"); + const std::vector> exact = + RunnerGreedy(fx.cfg, *model, {prompt}, steps); + + setenv("VT_KIMI_PAGED_MLA_FA2", "1", 1); + const std::vector> shared = + RunnerGreedy(fx.cfg, *model, {prompt}, steps); + CHECK(shared == exact); + + // The f32-page rejection probes ForwardPaged DIRECTLY: RunnerGreedy's internal + // REQUIREs throw doctest's own failure exception (not a std::runtime_error), + // which would mask the guard's type. + setenv("VT_KV_CACHE_F32", "1", 1); + { + KimiLinearWeights w = + LoadKimiLinearResidentBf16Weights(fx.shards, fx.cfg, nullptr); + vt::Queue q = Q(); + const int64_t head = KV_LORA + QK_ROPE; + const int64_t conv_dim = 3 * KDA_PROJ; + std::vector mla_page(static_cast(2 * 8 * head), 0.0f); + std::vector conv_buf(static_cast(conv_dim) * (CONV - 1), 0.0f); + std::vector ssm_buf(static_cast(KDA_NH) * KDA_HD * KDA_HD, 0.0f); + vllm::PagedKvCache kv; + kv.data = mla_page.data(); + kv.dtype = DType::kF32; + kv.num_blocks = 2; + kv.block_size = 8; + kv.num_kv_heads = 1; + kv.head_size = head; + std::vector attn_kv{kv}; + vllm::GdnStateCache gs; + gs.conv_state = vt::Tensor::Contiguous(conv_buf.data(), DType::kF32, q.device, + {1, conv_dim, CONV - 1}); + gs.ssm_state = vt::Tensor::Contiguous(ssm_buf.data(), DType::kF32, q.device, + {1, KDA_NH, KDA_HD, KDA_HD}); + std::vector gdn_state{gs}; + const std::vector toks = {2, 6}; + vllm::v1::CommonAttentionMetadata am; + am.query_start_loc = {0, 2}; + am.query_start_loc_cpu = am.query_start_loc; + am.seq_lens = {2}; + am.seq_lens_cpu = am.seq_lens; + am.num_computed_tokens_cpu = {0}; + am.num_reqs = 1; + am.num_actual_tokens = 2; + am.max_query_len = 2; + am.max_seq_len = 2; + am.block_table_tensor = {0, 1}; + am.block_table_num_cols = 2; + am.slot_mapping = {0, 1}; + vllm::v1::CommonAttentionMetadata gdn_cam = am; + gdn_cam.block_table_tensor = {0}; + gdn_cam.block_table_num_cols = 1; + vllm::v1::GDNAttentionMetadataBuilder builder; + vllm::v1::GDNAttentionMetadata gm = builder.build(0, gdn_cam); + const std::vector positions = {0, 1}; + const std::vector li = {1}; + vllm::ModelForwardInput in{ + .token_ids = toks, + .positions = positions, + .attn_meta = am, + .gdn_meta = gm, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = fx.cfg, + .queue = q, + .logits_indices = li, + .num_reqs = 1, + }; + CHECK_THROWS_WITH_AS((void)KimiLinearModel::ForwardPaged(in, w), + doctest::Contains("paged latent cache must be bf16"), + std::runtime_error); + } + unsetenv("VT_KV_CACHE_F32"); + unsetenv("VT_KIMI_PAGED_MLA_FA2"); +} + +// ─── (b2) LOGITS-EXACT — ForwardPaged logits == CLI incremental logits ──────── +// The runner-level token gates above are necessary but weak on a tiny random +// model (an argmax over V=8 can survive a corrupted state). This case drives +// KimiLinearModel::ForwardPaged DIRECTLY over hand-allocated paged caches with +// the REAL GDNAttentionMetadataBuilder segmentation, and requires the [1,V] +// logits row to be byte-equal to the CLI incremental leg at EVERY step — so a +// dropped state scatter, a stale conv tap, or a mis-indexed latent row all go +// RED (verified by mutation). +TEST_CASE("kimi paged: ForwardPaged logits byte-equal the CLI incremental logits") { + setenv("VT_KV_CACHE_F32", "1", 1); + setenv("VT_KIMI_PAGED_MLA_FA2", "0", 1); // the EXACT arm is the identity vehicle + Fixture fx; + KimiLinearWeights w = LoadKimiLinearResidentBf16Weights(fx.shards, fx.cfg, nullptr); + vt::Queue q = Q(); + vt::Backend& be = vt::GetBackend(q.device.type); + + const std::vector prompt = {5, 1, 2, 7, 3}; + const int P = static_cast(prompt.size()); + const int steps = 8; + const int64_t vocab = w.params.vocab_size; + + // Paged caches: 1 MLA layer page (f32, 2 blocks x 8) + 1 KDA state group. + const int64_t bs = 8, nblocks = 2; + const int64_t head = KV_LORA + QK_ROPE; + const int64_t conv_dim = 3 * KDA_PROJ; + std::vector mla_page(static_cast(nblocks * bs * head), 0.0f); + std::vector conv_buf(static_cast(conv_dim) * (CONV - 1), 0.0f); + std::vector ssm_buf(static_cast(KDA_NH) * KDA_HD * KDA_HD, 0.0f); + vllm::PagedKvCache kv; + kv.data = mla_page.data(); + kv.dtype = DType::kF32; + kv.num_blocks = nblocks; + kv.block_size = bs; + kv.num_kv_heads = 1; + kv.head_size = head; + std::vector attn_kv{kv}; + vllm::GdnStateCache gs; + gs.conv_state = vt::Tensor::Contiguous(conv_buf.data(), DType::kF32, q.device, + {1, conv_dim, CONV - 1}); + gs.ssm_state = vt::Tensor::Contiguous(ssm_buf.data(), DType::kF32, q.device, + {1, KDA_NH, KDA_HD, KDA_HD}); + std::vector gdn_state{gs}; + + // One step through ForwardPaged: `toks` at positions [pos0, pos0+n), context + // pos0 tokens already absorbed. Returns the gathered [1,V] logits row. + auto paged_step = [&](const std::vector& toks, int pos0) { + const int n = static_cast(toks.size()); + vllm::v1::CommonAttentionMetadata am; + am.query_start_loc = {0, n}; + am.query_start_loc_cpu = am.query_start_loc; + am.seq_lens = {pos0 + n}; + am.seq_lens_cpu = am.seq_lens; + am.num_computed_tokens_cpu = {pos0}; + am.num_reqs = 1; + am.num_actual_tokens = n; + am.max_query_len = n; + am.max_seq_len = pos0 + n; + am.block_table_tensor = {0, 1}; + am.block_table_num_cols = 2; + for (int t = 0; t < n; ++t) am.slot_mapping.push_back(pos0 + t); + vllm::v1::CommonAttentionMetadata gdn_cam = am; + gdn_cam.block_table_tensor = {0}; // GDN state slot 0 + gdn_cam.block_table_num_cols = 1; + vllm::v1::GDNAttentionMetadataBuilder builder; + vllm::v1::GDNAttentionMetadata gm = builder.build(0, gdn_cam); + std::vector positions(toks.size()); + for (int t = 0; t < n; ++t) positions[static_cast(t)] = pos0 + t; + const std::vector li = {n - 1}; + vllm::ModelForwardInput in{ + .token_ids = toks, + .positions = positions, + .attn_meta = am, + .gdn_meta = gm, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = fx.cfg, + .queue = q, + .logits_indices = li, + .num_reqs = 1, + }; + ForwardLogits fl = KimiLinearModel::ForwardPaged(in, w); + std::vector row(static_cast(vocab)); + be.Copy(q, row.data(), fl.device_tensor.data, row.size() * sizeof(float)); + be.Synchronize(q); + return row; + }; + + // The CLI leg, capturing per-step logits. + KimiDecodeCache cache; + std::vector positions(prompt.size()); + for (size_t t = 0; t < prompt.size(); ++t) positions[t] = static_cast(t); + const std::vector li = {P - 1}; + auto row_of = [&](const ForwardLogits& fl) { + std::vector row(static_cast(vocab)); + be.Copy(q, row.data(), fl.device_tensor.data, row.size() * sizeof(float)); + be.Synchronize(q); + return row; + }; + auto argmax = [&](const std::vector& row) { + int best = 0; + for (int64_t o = 1; o < vocab; ++o) + if (row[static_cast(o)] > row[static_cast(best)]) + best = static_cast(o); + return best; + }; + + std::vector cli_row = row_of(KimiLinearModel::ForwardPrefillIncremental( + prompt, positions, w, q, cache, li)); + std::vector paged_row = paged_step(prompt, 0); + CHECK(std::any_of(ssm_buf.begin(), ssm_buf.end(), + [](float x) { return x != 0.0f; })); + int mismatches = 0; + for (int64_t o = 0; o < vocab; ++o) + if (cli_row[static_cast(o)] != paged_row[static_cast(o)]) + ++mismatches; + CHECK(mismatches == 0); + + int tok = argmax(cli_row); + CHECK(argmax(paged_row) == tok); + int pos0 = P; + for (int s = 1; s < steps; ++s) { + cli_row = row_of( + KimiLinearModel::ForwardDecodeStepIncremental(tok, cache.seq_len, w, q, cache)); + const std::vector state_before = ssm_buf; + paged_row = paged_step({tok}, pos0); + // A decode step must commit the advanced recurrent state back to the + // runner-owned slot. Logit equality alone is too weak on this tiny model: + // deleting the decode GdnStateScatter used to leave its argmax unchanged. + CHECK(ssm_buf != state_before); + mismatches = 0; + for (int64_t o = 0; o < vocab; ++o) + if (cli_row[static_cast(o)] != paged_row[static_cast(o)]) + ++mismatches; + INFO("decode step ", s); + CHECK(mismatches == 0); + const int nt = argmax(cli_row); + CHECK(argmax(paged_row) == nt); + tok = nt; + pos0 += 1; + } + unsetenv("VT_KV_CACHE_F32"); + unsetenv("VT_KIMI_PAGED_MLA_FA2"); +} + +// Two fresh prefills in one batch must populate the two state slots named by +// GDNAttentionMetadata. This pins the slot mapping directly: token-level +// isolation can survive a mutation that sends every fresh prefill to slot 0 on +// a tiny random model. +TEST_CASE("kimi paged: batched prefill writes each request's distinct KDA state slot") { + setenv("VT_KV_CACHE_F32", "1", 1); + setenv("VT_KIMI_PAGED_MLA_FA2", "0", 1); // f32 pages => the exact arm + Fixture fx; + KimiLinearWeights w = LoadKimiLinearResidentBf16Weights(fx.shards, fx.cfg, nullptr); + vt::Queue q = Q(); + const std::vector pa = {5, 1, 2, 7, 3}; + const std::vector pb = {6, 6, 0, 4}; + std::vector toks = pa; + toks.insert(toks.end(), pb.begin(), pb.end()); + const int na = static_cast(pa.size()); + const int nb = static_cast(pb.size()); + const int nt = na + nb; + const int64_t head = KV_LORA + QK_ROPE; + const int64_t conv_dim = 3 * KDA_PROJ; + const size_t ssm_row = static_cast(KDA_NH) * KDA_HD * KDA_HD; + + std::vector mla_page(static_cast(4 * 8 * head), 0.0f); + std::vector conv_buf(static_cast(2 * conv_dim) * (CONV - 1), 0.0f); + std::vector ssm_buf(2 * ssm_row, 0.0f); + vllm::PagedKvCache kv; + kv.data = mla_page.data(); + kv.dtype = DType::kF32; + kv.num_blocks = 4; + kv.block_size = 8; + kv.num_kv_heads = 1; + kv.head_size = head; + std::vector attn_kv{kv}; + vllm::GdnStateCache gs; + gs.conv_state = vt::Tensor::Contiguous(conv_buf.data(), DType::kF32, q.device, + {2, conv_dim, CONV - 1}); + gs.ssm_state = vt::Tensor::Contiguous(ssm_buf.data(), DType::kF32, q.device, + {2, KDA_NH, KDA_HD, KDA_HD}); + std::vector gdn_state{gs}; + + vllm::v1::CommonAttentionMetadata am; + am.query_start_loc = {0, na, nt}; + am.query_start_loc_cpu = am.query_start_loc; + am.seq_lens = {na, nb}; + am.seq_lens_cpu = am.seq_lens; + am.num_computed_tokens_cpu = {0, 0}; + am.num_reqs = 2; + am.num_actual_tokens = nt; + am.max_query_len = std::max(na, nb); + am.max_seq_len = std::max(na, nb); + am.block_table_tensor = {0, 1, 2, 3}; + am.block_table_num_cols = 2; + for (int t = 0; t < na; ++t) am.slot_mapping.push_back(t); + for (int t = 0; t < nb; ++t) am.slot_mapping.push_back(2 * 8 + t); + vllm::v1::CommonAttentionMetadata gdn_cam = am; + gdn_cam.block_table_tensor = {0, 1}; + gdn_cam.block_table_num_cols = 1; + vllm::v1::GDNAttentionMetadataBuilder builder; + vllm::v1::GDNAttentionMetadata gm = builder.build(0, gdn_cam); + REQUIRE(gm.non_spec_state_indices_tensor.has_value()); + CHECK(*gm.non_spec_state_indices_tensor == std::vector{0, 1}); + + std::vector positions(static_cast(nt)); + for (int t = 0; t < na; ++t) positions[static_cast(t)] = t; + for (int t = 0; t < nb; ++t) positions[static_cast(na + t)] = t; + const std::vector li = {na - 1, nt - 1}; + vllm::ModelForwardInput in{ + .token_ids = toks, + .positions = positions, + .attn_meta = am, + .gdn_meta = gm, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = fx.cfg, + .queue = q, + .logits_indices = li, + .num_reqs = 2, + }; + (void)KimiLinearModel::ForwardPaged(in, w); + + const auto slot_nonzero = [&](size_t slot) { + const auto first = ssm_buf.begin() + static_cast(slot * ssm_row); + return std::any_of(first, first + static_cast(ssm_row), + [](float x) { return x != 0.0f; }); + }; + CHECK(slot_nonzero(0)); + CHECK(slot_nonzero(1)); + CHECK_FALSE(std::equal(ssm_buf.begin(), ssm_buf.begin() + ssm_row, + ssm_buf.begin() + ssm_row)); + unsetenv("VT_KV_CACHE_F32"); + unsetenv("VT_KIMI_PAGED_MLA_FA2"); +} + +// ─── (b3) ASYNC DEVICE-MIRROR input ids (ENG-ASYNC-SCHED W4) ────────────────── +// On the GB10 served path the async device mirror patches each decode row's +// sampled token into the RUNNER's device input-id buffer and deliberately +// leaves the host `token_ids` STALE — a model that ignores +// `ModelForwardInput::device_token_ids` embeds garbage (the 9/128 divergence +// this case was cut from). ForwardPaged must embed from the device pointer +// when it is non-null. On CPU a host pointer IS device-addressable, so the +// contract is directly testable: hand the RIGHT ids only through +// device_token_ids (host ids deliberately wrong) and require the logits to +// match a normal run with the right host ids. +TEST_CASE("kimi paged: ForwardPaged embeds device_token_ids over stale host ids") { + setenv("VT_KV_CACHE_F32", "1", 1); + setenv("VT_KIMI_PAGED_MLA_FA2", "0", 1); + Fixture fx; + KimiLinearWeights w = LoadKimiLinearResidentBf16Weights(fx.shards, fx.cfg, nullptr); + vt::Queue q = Q(); + vt::Backend& be = vt::GetBackend(q.device.type); + const int64_t vocab = w.params.vocab_size; + const int64_t conv_dim = 3 * KDA_PROJ; + const int64_t head = KV_LORA + QK_ROPE; + + // Two independent single-request paged contexts (fresh caches each). + auto run_prefill = [&](const std::vector& host_ids, + const int32_t* device_ids) { + std::vector mla_page(static_cast(2 * 8 * head), 0.0f); + std::vector conv_buf(static_cast(conv_dim) * (CONV - 1), 0.0f); + std::vector ssm_buf(static_cast(KDA_NH) * KDA_HD * KDA_HD, 0.0f); + vllm::PagedKvCache kv; + kv.data = mla_page.data(); + kv.dtype = DType::kF32; + kv.num_blocks = 2; + kv.block_size = 8; + kv.num_kv_heads = 1; + kv.head_size = head; + std::vector attn_kv{kv}; + vllm::GdnStateCache gs; + gs.conv_state = vt::Tensor::Contiguous(conv_buf.data(), DType::kF32, q.device, + {1, conv_dim, CONV - 1}); + gs.ssm_state = vt::Tensor::Contiguous(ssm_buf.data(), DType::kF32, q.device, + {1, KDA_NH, KDA_HD, KDA_HD}); + std::vector gdn_state{gs}; + const int n = static_cast(host_ids.size()); + vllm::v1::CommonAttentionMetadata am; + am.query_start_loc = {0, n}; + am.query_start_loc_cpu = am.query_start_loc; + am.seq_lens = {n}; + am.seq_lens_cpu = am.seq_lens; + am.num_computed_tokens_cpu = {0}; + am.num_reqs = 1; + am.num_actual_tokens = n; + am.max_query_len = n; + am.max_seq_len = n; + am.block_table_tensor = {0, 1}; + am.block_table_num_cols = 2; + for (int t = 0; t < n; ++t) am.slot_mapping.push_back(t); + vllm::v1::CommonAttentionMetadata gdn_cam = am; + gdn_cam.block_table_tensor = {0}; + gdn_cam.block_table_num_cols = 1; + vllm::v1::GDNAttentionMetadataBuilder builder; + vllm::v1::GDNAttentionMetadata gm = builder.build(0, gdn_cam); + std::vector positions(host_ids.size()); + for (int t = 0; t < n; ++t) positions[static_cast(t)] = t; + const std::vector li = {n - 1}; + vllm::ModelForwardInput in{ + .token_ids = host_ids, + .positions = positions, + .attn_meta = am, + .gdn_meta = gm, + .attn_kv = attn_kv, + .gdn_state = gdn_state, + .config = fx.cfg, + .queue = q, + .logits_indices = li, + .num_reqs = 1, + }; + in.device_token_ids = device_ids; + ForwardLogits fl = KimiLinearModel::ForwardPaged(in, w); + std::vector row(static_cast(vocab)); + be.Copy(q, row.data(), fl.device_tensor.data, row.size() * sizeof(float)); + be.Synchronize(q); + return row; + }; + + const std::vector right = {5, 1, 2, 7, 3}; + const std::vector stale = {0, 0, 0, 0, 0}; // deliberately wrong host ids + const std::vector want = run_prefill(right, nullptr); + const std::vector via_device = run_prefill(stale, right.data()); + int mismatches = 0; + for (int64_t o = 0; o < vocab; ++o) + if (want[static_cast(o)] != via_device[static_cast(o)]) + ++mismatches; + CHECK(mismatches == 0); + unsetenv("VT_KV_CACHE_F32"); + unsetenv("VT_KIMI_PAGED_MLA_FA2"); +} + +// ─── (c) SLOT ISOLATION — batched decode == per-request single runs ─────────── +TEST_CASE("kimi paged: 2-request batched decode matches each single-request run") { + setenv("VT_KV_CACHE_F32", "1", 1); + setenv("VT_KIMI_PAGED_MLA_FA2", "0", 1); + Fixture fx; + std::unique_ptr model = + ModelRegistry::Load(fx.cfg, ModelSource::FromSafetensors(fx.shards)); + const std::vector pa = {5, 1, 2, 7, 3}; + const std::vector pb = {6, 6, 0, 4}; + const int steps = 6; + + const std::vector> batched = + RunnerGreedy(fx.cfg, *model, {pa, pb}, steps); + const std::vector> only_a = + RunnerGreedy(fx.cfg, *model, {pa}, steps); + const std::vector> only_b = + RunnerGreedy(fx.cfg, *model, {pb}, steps); + + REQUIRE(batched.size() == 2); + for (int s = 0; s < steps; ++s) { + INFO("step ", s); + CHECK(batched[0][static_cast(s)] == only_a[0][static_cast(s)]); + CHECK(batched[1][static_cast(s)] == only_b[0][static_cast(s)]); + } + unsetenv("VT_KV_CACHE_F32"); + unsetenv("VT_KIMI_PAGED_MLA_FA2"); +} diff --git a/tests/vllm/test_hf_config.cpp b/tests/vllm/test_hf_config.cpp index c8bb59b2d..16d828fd1 100644 --- a/tests/vllm/test_hf_config.cpp +++ b/tests/vllm/test_hf_config.cpp @@ -10,6 +10,7 @@ #include #include #include +#include #include "vllm/transformers_utils/hf_config.h" @@ -389,6 +390,116 @@ TEST_CASE("LoadHfConfig mirrors sliding_window normalization from the text confi } } +// ROW 7 / kimi-linear.md §20.3 B1 — KV enablement for the shared paged runner. +// Kimi-Linear's config carries NO `layer_types` and NONE of the qwen3_5-style +// explicit `linear_*` GDN-geometry keys: its KDA/full-attn split lives in the +// nested `linear_attn_config` (transformers_utils/configs/kimi_linear.py:34-148, +// `is_kda_layer(l) := (l+1) in kda_layers` :144-148 — the lists are 1-INDEXED). +// LoadHfConfig must SYNTHESIZE the runner-facing fields from it so the runner's +// MambaSpec shape check (runner.cpp:489-493) and the per-layer KDA-state / +// MLA-page allocation loop (runner.cpp:625-) see the same geometry +// MakeKimiLinearKVCache declares — instead of {0,0},{0,0,0} and an abort. +TEST_CASE("LoadHfConfig synthesizes layer_types + GDN geometry from linear_attn_config") { + TempJson f(R"({ + "model_type": "kimi_linear", + "architectures": ["KimiLinearForCausalLM"], + "hidden_size": 2304, + "num_hidden_layers": 8, + "vocab_size": 163840, + "num_attention_heads": 32, + "intermediate_size": 9216, + "rms_norm_eps": 1e-05, + "kv_lora_rank": 512, + "qk_nope_head_dim": 128, + "qk_rope_head_dim": 64, + "v_head_dim": 128, + "mla_use_nope": true, + "linear_attn_config": { + "kda_layers": [1, 2, 3, 5, 6, 7], + "full_attn_layers": [4, 8], + "num_heads": 32, + "head_dim": 128, + "short_conv_kernel_size": 4 + }, + "max_position_embeddings": 1048576 + })"); + vllm::HfConfig cfg = vllm::LoadHfConfig(f.path()); + + // GDN-group geometry sourced from linear_attn_config (num_k == num_v == the + // KDA num_heads; Dk == Dv == head_dim; conv kernel == short_conv_kernel_size), + // so the runner derives conv_dim = 2*Hk*Dk + Hv*Dv = 3*32*128 = 12288 and the + // ssm shape {32,128,128} — exactly MakeKimiLinearKVCache's MambaSpec. + CHECK(cfg.linear_num_key_heads == 32); + CHECK(cfg.linear_num_value_heads == 32); + CHECK(cfg.linear_key_head_dim == 128); + CHECK(cfg.linear_value_head_dim == 128); + CHECK(cfg.linear_conv_kernel_dim == 4); + + // layer_types synthesized from the 1-indexed kda_layers list. + REQUIRE(cfg.layer_types.size() == 8); + const std::vector expect = { + "linear_attention", "linear_attention", "linear_attention", + "full_attention", "linear_attention", "linear_attention", + "linear_attention", "full_attention"}; + for (size_t i = 0; i < expect.size(); ++i) CHECK(cfg.layer_types[i] == expect[i]); + + // The raw doc still carries linear_attn_config for ParseKimiLinearParams. + CHECK(cfg.raw.contains("linear_attn_config")); +} + +// The synthesis must be ADDITIVE: a config that carries the explicit qwen3_5 +// fields (kHybridJson has both layer_types AND linear_*) is untouched by it — +// that path is byte-identical (asserted by the hybrid TEST_CASE above), and a +// config with NEITHER (plain dense llama) stays all-zero / empty. +TEST_CASE("LoadHfConfig linear_attn_config synthesis leaves non-Kimi configs untouched") { + TempJson f(kLlamaJson); + vllm::HfConfig cfg = vllm::LoadHfConfig(f.path()); + CHECK(cfg.linear_num_key_heads == 0); + CHECK(cfg.linear_num_value_heads == 0); + CHECK(cfg.linear_conv_kernel_dim == 0); + CHECK(cfg.layer_types.empty()); +} + +// The PRIORITY conjunct (`cfg.linear_num_key_heads == 0`): a config carrying +// BOTH the explicit qwen3_5-style keys AND a linear_attn_config keeps the +// EXPLICIT values — the synthesis never overwrites them. Drop the conjunct and +// this case REDs (num_value_heads would become 4 == num_heads, key_head_dim 16, +// conv 4). This makes B1 additive-by-construction, pinned rather than assumed. +TEST_CASE("LoadHfConfig explicit linear_* fields WIN over linear_attn_config") { + TempJson f(R"({ + "model_type": "qwen3_5_moe", + "architectures": ["Qwen3NextForCausalLM"], + "hidden_size": 64, + "num_hidden_layers": 3, + "vocab_size": 128, + "num_attention_heads": 4, + "layer_types": ["linear_attention", "full_attention", "linear_attention"], + "linear_num_key_heads": 2, + "linear_num_value_heads": 8, + "linear_key_head_dim": 32, + "linear_value_head_dim": 64, + "linear_conv_kernel_dim": 3, + "linear_attn_config": { + "kda_layers": [1, 2, 3], + "full_attn_layers": [], + "num_heads": 4, + "head_dim": 16, + "short_conv_kernel_size": 4 + } + })"); + vllm::HfConfig cfg = vllm::LoadHfConfig(f.path()); + // The explicit (qwen3_5-style, asymmetric) geometry survives verbatim ... + CHECK(cfg.linear_num_key_heads == 2); + CHECK(cfg.linear_num_value_heads == 8); + CHECK(cfg.linear_key_head_dim == 32); + CHECK(cfg.linear_value_head_dim == 64); + CHECK(cfg.linear_conv_kernel_dim == 3); + // ... and the explicit layer_types are NOT resynthesized from kda_layers + // (which would flip index 1 to linear_attention). + REQUIRE(cfg.layer_types.size() == 3); + CHECK(cfg.layer_types[1] == "full_attention"); +} + TEST_CASE("LoadHfConfig throws when required fields are missing") { SUBCASE("missing model_type") { TempJson f(R"({"hidden_size": 64, "num_hidden_layers": 2})");