v1.3 PPL on vLLM: production cell + per-channel attribution — K 64%, V 31%, V-outlier worth −4 pp by FluffyAIcode · Pull Request #15 · FluffyAIcode/LLM-KV--Cache-compress

FluffyAIcode · 2026-04-21T15:49:38Z

What this PR ships

The v1.3 PPL production recipe running on vLLM, plus a 4-row
per-channel attribution ablation on the same cell.

Config: K b=3 + V b=2 + randomized PCA rank=D/2 + calibrated
Lloyd-Max + outlier T=2.0 + 6-layer boundary skip + pre-RoPE
Q-preconditioning. vLLM 0.7.3, FLASH_ATTN, bf16, enforce_eager,
DS-Distill-Qwen-1.5B on H200, 4 WikiText-103 test passages,
ctx=2048, n_eval=64, paired shared reference.

HF reference for the same cell (SPRINT_CLOSEOUT): +7.82 % Δppl,
78.97 % top-1, MARGINAL.

Results (4 rows)

Row	K codec?	V codec?	V outlier	Δppl	top-1	Verdict
production (K+V)	✔	✔	off	+35.33 %	59.38 %	REJECT
K-only (V bf16)	✔	—	—	+22.55 %	69.14 %	REJECT
V-only (K bf16, SPRINT_CLOSEOUT V-side recipe)	—	✔	off	+11.10 %	74.22 %	REJECT
V-only + outlier T=2.0 (all applicable V guards)	—	✔	T=2.0	+7.04 %	75.39 %	REJECT

All four rows retain Q-precond on the layers that use K (rows 1–2),
calibrated Lloyd-Max centroids on the stream that is compressed, and
the 6-layer boundary skip on whichever stream is compressed.

"Four guardrails" per channel

Guardrail	K	V
Q-preconditioning	required	N/A (V does not contract with Q; no Σ_q metric)
Calibrated Lloyd-Max	`ds_K_b3`	`ds_V_b2`
6-layer boundary skip	same layer set	same layer set
Outlier compensation T	on (SPRINT_CLOSEOUT)	off in SPRINT_CLOSEOUT; on in row 4 here

So row 4 is "V with every guardrail that applies to V".

Per-channel Δppl attribution

K-only + V-only ≈ production − interaction (1.68 pp):

K stream: +22.55 pp / 64 % of +35.33.
V stream: +11.10 pp / 31 % of +35.33.
Interaction: ~1.68 pp.

V outlier compensation shaves ~4 pp off V's standalone cost
(+11.10 → +7.04). If the joint cell were re-run with outlier on both
streams, we'd expect roughly +22.55 + 7.04 + 1.68 ≈ +31 pp Δppl
— still far from MARGINAL (±3 %), but a cheap improvement.

Reading

K is the bigger lever on vLLM, even though the entire v1.3 PPL
K-side guardrail stack (Q-precond, K Lloyd-Max centroids, outlier
T=2.0, 6-layer boundary) is already applied. On HF that stack
gets the joint cell to MARGINAL (+7.82 %); on vLLM it leaves
+22.55 pp on K alone.
V at b=2 with SPRINT_CLOSEOUT's recipe carries +11.10 pp. Adding
V outlier compensation (the only guardrail that SPRINT_CLOSEOUT
omitted on V) saves 4 pp → +7.04 pp. Cheap and worth doing on
vLLM, though it doesn't by itself close the gap.
Neither channel alone reaches MARGINAL on vLLM at this codec
config.

Code

Rust codec guardrails ported from PR Outlier compensation + Besicovitch-product skeleton — diagnostic sprint #13 (calibrated Lloyd-Max +
outlier compensation); 164 tests pass.
benchmarks/e2e_ppl_validation_vllm_full.py — production harness
with the pre-RoPE Qwen2Attention.forward hook. Two clean knobs:
--compress-stream {kv, k, v} and --v-outlier-threshold T,
exposed on the driver as COMPRESS_STREAM and
V_OUTLIER_THRESHOLD env vars. The Rust codec already supports
V outlier compensation; only the Python harness had to expose it.
benchmarks/run_v1_3_ppl_full_vllm.sh — single driver.

Artifacts

reports/v1_3_ppl/FINDINGS.md — consolidated 4-row writeup.
reports/v1_3_ppl/vllm/ — production (K+V) row.
reports/v1_3_ppl/vllm_k_only/ — K-only row.
reports/v1_3_ppl/vllm_v_only/ — V-only (SPRINT_CLOSEOUT V recipe).
reports/v1_3_ppl/vllm_v_only_full_guards/ — V-only + outlier T=2.0.

Test status

Rust: 164 unit + integration + proptest pass.
Python: syntax-clean.
H200: all four rows committed.

Relationship to other PRs

PR E2E PPL validation: codec REJECTs downstream on real Qwen2.5 — major finding, paper claims must be revised #12 — HF harness that flagged the downstream-quality problem on
bare v1.3.
PR Outlier compensation + Besicovitch-product skeleton — diagnostic sprint #13 (draft) — SPRINT_CLOSEOUT definition of "v1.3 PPL".
PR vLLM integration scaffolding for the kakeyaturbo codec (codec port + Attention.forward hook + harness) #14 (scaffolding) — Rust v1.3 port + Attention.forward hook.
This PR — v1.3 PPL production recipe on vLLM with pre-RoPE
Qwen2Attention.forward hook + per-channel attribution +
symmetric V-outlier add-on.

Brings the v1.3 pieces that are shared by the e2e validation branch onto main as a clean baseline for vLLM integration: - kakeyaturbo/src/codec.rs: PcaMethod enum (Exact | Randomized{...}), CodecParams gets pca_method field, fit_pca_dispatch routes to exact or randomized PCA path. - kakeyaturbo/src/pca.rs: adds fit_weighted_pca_randomized() (Halko-Martinsson-Tropp randomized SVD with power iterations). - kakeyaturbo/src/lib.rs: re-export PcaMethod. - kakeyaturbo/src/bin/kakeyaturbo-bench.rs: new CLI flags --pca-method exact|randomized --rsvd-target-rank N --rsvd-oversample N --rsvd-power-iters N --dump-decoded PATH (write decoded KKTV for external drivers) - benchmarks/e2e_ppl_validation.py: HF-transformers harness that prefills DynamicCache, round-trips every full-attention layer through the Rust codec, teacher-forces continuation tokens, and reports Δppl / top1 / KL. - kakeyaturbo/tests/integration.rs: update CodecParams initializer for the new field. All 153 Rust unit + integration + proptest cases pass. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

benchmarks/e2e_ppl_validation_vllm.py is a drop-in alternative to e2e_ppl_validation.py that routes the forward pass through vLLM rather than HF eager attention, so the measured \u0394ppl reflects the codec's behaviour under the production inference engine. Integration approach: - Monkey-patch vllm.attention.layer.Attention.forward (installed before LLM construction). - The patched forward, when CodecState.active is True, round-trips the K and V tensors through the v1.3 Rust codec (kakeyaturbo-bench --dump-decoded) before passing them on to the underlying paged-attention kernel. K uses inner_product metric, V uses mse + share_basis (asymmetric config matching the HF harness). - Each passage is evaluated twice: once with the codec OFF (reference) and once ON (alt), using vLLM's prompt_logprobs=1. Per-position PPL and top-1 agreement over [ctx_len, ctx_len+n_eval) are compared with the same ACCEPT/MARGINAL/REJECT verdict as the HF harness, so the two engines' numbers can be placed side-by-side. benchmarks/run_v1_3_ppl_vllm.sh: convenience driver that builds the Rust bench binary and launches the harness with the standard smoke config (Qwen2.5-0.5B, ctx=1024, 2 passages, b=2, rsvd). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

- Patched forward now has the exact (query, key, value, kv_cache, attn_metadata) signature vLLM's custom-op dispatcher expects. - Use self.head_size from the Attention module to reshape the KV tensor correctly regardless of whether it enters as 2D [num_tokens, num_kv_heads * head_size] or already-reshaped 3D. - Use self.layer_name (vLLM's stable per-layer identifier) as the default layer_id; fall back to a module-scope counter only when not present. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

On Vast.ai vLLM is installed into /venv/main, not into the system python3 on PATH. Default PYTHON_BIN to the venv python when that venv exists, and allow override via the PYTHON_BIN env var. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Adds reports/v1_3_rsvd_rope/e2e_ppl_vllm_smoke/: - qwen2_5_0_5b_vllm.json: per-passage metrics (Qwen2.5-0.5B, ctx=1024, n_eval=64, 2 passages, b=2 rsvd, randomized PCA, vr=0.95). - FINDINGS.md: engine setup, cross-engine comparison against the HF harness from PR #12, reproduction instructions. Result summary: Δppl mean = +291.9 % (passage 1: +192 %, passage 2: +391 %) top1 mean = 46.9 % verdict = REJECT (threshold is |Δppl|<=1% AND top1>=95%) This confirms on the production inference engine (vLLM 0.7.3 with Flash-Attention on H200) what PR #12 found on HF eager: the v1.3 codec at its tier-1 setting does not preserve downstream quality. The magnitude of the degradation is smaller on vLLM than on HF (+292% vs +29,086%), but both clearly REJECT. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Per SPRINT_CLOSEOUT.md (PR #13), the production recipe is "v1.3 PPL" = v1.3 RSVD + 4 guardrails (Q-preconditioning, calibrated Lloyd-Max K codebook, 6-layer boundary protection, outlier compensation T=2.0). The smoke result landed in this PR's last commit (+292% \u0394ppl, 47% top-1 on Qwen2.5-0.5B + bare v1.3 b=2) is the V0 baseline under the sprint's ladder, NOT the production v1.3 PPL. Its number aligns with the ladder's V0 cell (+355% \u0394ppl, 42% top-1 on HF / DS-Distill). Remove that datapoint from this PR; this PR now scopes only the reusable vLLM integration scaffolding (codec port + Attention.forward monkey-patch + harness skeleton). The production-recipe integration is moved to a follow-up branch: AgentMemory/v1-3-ppl-full-guardrails-vllm-102e Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Brings the calibrated Lloyd-Max codebook + outlier compensation into the Rust codec so the full 'v1.3 PPL' production recipe can be driven from the vLLM harness. New CodecParams fields: - custom_centroids: Option<Vec<f32>> \u2014 calibrated Lloyd-Max table - outlier_threshold: Option<f32> \u2014 residual threshold T - exact_rank_cap: Option<usize> \u2014 caps d_eff on exact PCA - skeleton_dtype fp16|fp32 New CLI flags on kakeyaturbo-bench: --centroids-file PATH load 2^bits LE-f32 sorted centroid table --outlier-threshold T extract post-WHT residual coords with |scaled_residual| > T as (u16 idx, f16 val) sparse overrides at decode Internal wiring: *_with_centroids variants of encode_block / decode_block / encode_layer / decode_layer thread the centroid table and outlier buffer through the block pipeline without changing the wire format when neither is set (bit-compatible default). 153 Rust unit + 5 integration + 6 proptest = 164 tests pass. Source: cherry-pick of 521e97b ('outlier compensation: Rust codec support + Python harness + 4-passage PPL validation on DS-Distill') onto the v1.3 scaffolding branch (includes the preceding Steps 3+4 Lloyd-Max + boundary infrastructure from 05dfbc5). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Artifacts needed for the 'v1.3 PPL' production recipe: reports/v1_4_q_pca/flagship/deepseek_distill_q_calib.{safetensors,json} \u03a3_q Cholesky factors per (layer, kv-head) for DeepSeek-R1-Distill-Qwen-1.5B (28 layers \u00d7 [2, 128, 128]). reports/v1_4_q_pca/calibrated_codebook/ds_K_b{2,3}_centroids.f32 reports/v1_4_q_pca/calibrated_codebook/ds_V_b2_centroids.f32 Empirical Lloyd-Max centroid tables (2^bits LE-f32 floats each) trained on pooled 25M post-WHT residual samples from DS-Distill. Python helpers: benchmarks/q_precondition.py K-stream whitening K_tilde = K @ L benchmarks/q_calibration.py offline \u03a3_q Cholesky calibration benchmarks/lloyd_max_calibration.py offline residual codebook fitting These are the pieces the forthcoming vLLM harness extension needs to drive the full v1.3 PPL recipe (Q-precond + calibrated Lloyd-Max + boundary skip + outlier T=2.0). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

benchmarks/e2e_ppl_validation_vllm_full.py drives the full recipe: 1. Q-preconditioning K_tilde = K @ L (per layer, per kv-head) 2. Calibrated Lloyd-Max via --centroids-file to Rust codec 3. 6-layer boundary skip [0, 1, 7, 14, 26, 27] stay bf16 4. Outlier compensation T = 2.0 on K residual (--outlier-threshold) Unlike the scaffolding harness (e2e_ppl_validation_vllm.py) which hooks at vllm.attention.layer.Attention.forward (post-RoPE), this harness patches vllm.model_executor.models.qwen2.Qwen2Attention.forward to intercept K/V immediately after the QKV projection, BEFORE RoPE: qkv \u2192 split(q, k, v) k \u2190 unwhiten(codec_roundtrip(whiten(k), centroids, outlier)) [pre-RoPE] v \u2190 codec_roundtrip(v, centroids) q, k \u2190 rotary_emb(positions, q, k) \u2026 rest of normal attention runs on the repaired K and V This matches the PR #13 HF harness (benchmarks/e2e_ppl_pre_rope.py) semantically: \u03a3_q is calibrated on pre-RoPE K distributions, so the whitening must be applied to pre-RoPE K for the Sigma_q-MSE equivalence to hold. benchmarks/run_v1_3_ppl_full_vllm.sh: driver with env-overridable defaults matching the SPRINT_CLOSEOUT production cell: DS-Distill D=128, K b=3 + V b=2, T=2.0, 6 bdry \u2192 target \u0394ppl +7.82% / top-1 78.97% / ratio 4.61\u00d7 (MARGINAL) Syntax-check clean; end-to-end on GPU pending. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Full production recipe (K b=3 Q-precond + calibrated Lloyd-Max + outlier T=2.0 + 6-layer boundary skip; V b=2 calibrated + share-basis) runs end-to-end on vLLM 0.7.3 for DeepSeek-R1-Distill-Qwen-1.5B. Result (ctx=2048, n_eval=64, 4 WikiText-103 passages): Passage 1: \u0394ppl -8.86% top1 56.2% Passage 2: \u0394ppl +32.79% top1 51.6% Passage 3: \u0394ppl +40.46% top1 65.6% Passage 4: \u0394ppl +76.92% top1 64.1% Mean \u0394ppl = +35.33 %, mean top-1 = 59.4 % \u2192 REJECT Guardrails move bare v1.3 b=2 from +292% on vLLM (PR #14) \u2192 +35% on vLLM (this PR), an ~8\u00d7 \u0394ppl improvement \u2014 directional agreement with the HF ladder (+356% \u2192 +8% on DS-Distill). However vLLM ends ~4.5\u00d7 worse in \u0394ppl than HF at the same codec config on the same model family. Two likely causes (Flash-Attention numerics vs. HF eager, and CPU/GPU fp32<->bf16 boundary noise) are documented with follow-up sweeps in FINDINGS.md. Artifact: reports/v1_3_ppl/vllm/ds_distill_qwen_1_5b_vllm_full.json Full write-up: reports/v1_3_ppl/vllm/FINDINGS.md Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Motivation: PR #15 showed vLLM v1.3 PPL full-recipe gives \u0394ppl +35.3% vs HF's +7.82% on the same codec config on DS-Distill. Two hypotheses were named but not separated: (1) \u03a3_q was calibrated on pre-RoPE Q, but Flash-Attention computes Q@K.T on post-RoPE Q, breaking the Sigma_q -> K_tilde metric equivalence; (2) the per- forward CPU\u2194GPU and fp32\u2194bf16 round-trip itself accumulates enough numerical noise to degrade PPL. This harness runs four cells against a single shared reference, pair-wise per passage so all cells observe the same ref PPL: identity-pre_qp whiten \u2192 identity codec \u2192 unwhiten isolates hypothesis (2): everything except compression codec-no_qp real codec, no whitening isolates "codec only" codec-pre_qp production recipe (matches PR #15) codec-post_qp codec + post-RoPE \u03a3_q_post self-calibrated online from this run's own post-RoPE Q tensors isolates hypothesis (1): correct whitening under FA Key implementation details: - Qwen2Attention.forward is patched once; the patch branches on CodecState.mode/qp_mode to pick the right hook. - PostRopeQCalib accumulates Sum(q q.T) per (layer, kv-head) during a cheap dedicated calibration forward pass (codec off), then Cholesky-factors with a small ridge for stability. For GQA models (num_heads > num_kv_heads) we pool Q heads in the same KV group before accumulating, matching the metric FA actually computes. - All cells share the same computed once; each cell runs its own alt_pls and compares per passage. - Identity codec does skip the kakeyaturbo-bench subprocess but still goes through the full fp32\u21a6numpy\u21a6CPU\u21a6numpy\u21a6fp32 path, so it measures the complete CPU\u2194GPU noise floor. Syntax-clean; GPU run on Vast.ai H200 pending in the same turn. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

…irection Paired 4-cell ablation on DS-Distill + vLLM H200 (shared ref): identity-pre_qp \u0394ppl -0.29% top1 98.83% ACCEPT codec-no_qp \u0394ppl +152.78% top1 59.38% REJECT codec-pre_qp \u0394ppl +35.33% top1 59.38% REJECT (= PR #15) codec-post_qp \u0394ppl +54.28% top1 57.03% REJECT Findings: - H2 (CPU\u2194GPU + fp32\u2194bf16 noise) is ruled out. The identity cell walks the complete production hook pipeline minus compression and records -0.29% \u0394ppl / 98.83% top-1. - H1 (\u03a3_q was calibrated on pre-RoPE Q but FA operates on post-RoPE Q) as a direct fix-up is wrong. Online self-calibrated \u03a3_q^post makes things STRICTLY WORSE (+54% vs +35%). Math: RoPE is position-dependent; pooling post-RoPE Q over tokens averages away the per-token rotations and collapses anisotropy, giving a flatter pooled \u03a3 than the true per-token FA metric. - Pre-RoPE whitening IS the FA-correct thing (R_t L L^T R_t^T = R_t \u03a3_q R_t^T commutes with the per-token rotation). The Q-precond architectural choice in PR #13 is verified correct for vLLM too. The remaining +35% gap is not Q-precond placement but almost certainly calibration-distribution drift: \u03a3_q + centroids were all fit on HF DynamicCache prefill snapshots, but vLLM's Qwen2 layer produces slightly different prefill K/V distributions (different bf16 accumulation / RoPE impl / attn bias). The codec has to eat that mismatch. Next experiment: re-fit \u03a3_q and Lloyd-Max centroids on vLLM prefill snapshots and re-run codec-pre_qp. Artifacts: reports/v1_3_ppl/vllm_ablation/FINDINGS.md reports/v1_3_ppl/vllm_ablation/ds_distill_qwen_1_5b_vllm_ablation.json Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Follow-up to the ablation in reports/v1_3_ppl/vllm_ablation/: H2 (noise) and post-RoPE Q-precond hypothesis both ruled out; the remaining +35% \u0394ppl gap on vLLM vs HF is most likely calibration- distribution drift (\u03a3_q and Lloyd-Max centroids were fit on HF DynamicCache snapshots, not vLLM prefill snapshots). benchmarks/vllm_calibration_refit.py: 1. Spins up vLLM LLM (bf16, enforce_eager). 2. Installs a capture-only monkey patch on Qwen2Attention.forward that records the pre-RoPE q/k/v tensors without modifying the forward. 3. Runs N calibration passages from the WikiText-103 TRAIN split by default (disjoint from the TEST split the PPL measurement uses), so no leakage. 4. Factors \u03a3_q per (layer, kv-head) by pooling the Q heads in each GQA KV group. Matches the format used by benchmarks/q_precondition.QPrecond exactly: layer_<l>_chol [n_kv, D, D] fp32 layer_<l>_inv_chol [n_kv, D, D] fp32 layer_<l>_sigma [n_kv, D, D] fp32 5. Re-runs the Lloyd-Max residual pipeline on the captured K (whitened with the fresh \u03a3_q) and V streams, producing drop-in replacements for the ds_K_b{2,3}_centroids.f32 / ds_V_b2_centroids.f32 tables. Outputs at --out-dir (default reports/v1_3_ppl/vllm_recalibrated/): q_calib.safetensors q_calib.json K_b2_centroids.f32, K_b3_centroids.f32, V_b2_centroids.f32 SUMMARY.json benchmarks/run_vllm_calibration_refit.sh is the driver. Next step (not in this commit): re-run the codec-pre_qp ablation cell with --q-calib-pre-rope=.../q_calib.safetensors and --k-centroids / --v-centroids pointing at the new .f32 files. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

lloyd_max_calibration.py has top-level 'from transformers import …' and 'import benchmarks.pre_rope_cache …', which were blocking our tool from being usable in a vLLM-only environment (HF transformers IS present, but pre_rope_cache is a HF-eager patch that we don't need here — we just want four math utilities). Load the pure-numpy functions by stripping the HF imports from the source before exec()'ing it, then picking up fit_pca_simple / next_pow2 / wht_rotate / lloyd_max_iterate from the resulting namespace. Semantics unchanged. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Captured pre-RoPE Q/K/V from vLLM 0.7.3 on deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B using 8 disjoint WikiText-103 TRAIN split passages of 2048 tokens each (16k tokens per layer per kv-head pool). Artifacts (drop-in compatible with QPrecond / kakeyaturbo-bench): q_calib.safetensors 28 layers \u00d7 [2, 128, 128] (chol/inv/sigma) \u03a3_q anisotropy (cond): median 4506, max 109076 (off-diag max / diag mean: median 15.5, max 34.8) \u2014 strongly anisotropic, Q-precond has something to do. K_b2_centroids.f32 Gaussian MSE 0.1143 \u2192 calibrated 0.0721 (1.59\u00d7) K_b3_centroids.f32 Gaussian MSE 0.0318 \u2192 calibrated 0.0214 (1.48\u00d7) V_b2_centroids.f32 Gaussian MSE 0.1140 \u2192 calibrated 0.1140 (1.00\u00d7) q_calib.json per-(layer, kv-head) diagnostics SUMMARY.json centroid comparison These replace the HF-calibrated tables in reports/v1_4_q_pca/{flagship,calibrated_codebook}/ when the vLLM harness is run with --q-calib-pre-rope=.../q_calib.safetensors and --k-centroids/-v-centroids pointing at the new .f32 files. K improvement ratios match the SPRINT_CLOSEOUT HF-calibrated numbers (1.47\u00d7 at b=2 / 1.40\u00d7 at b=3) closely \u2014 the vLLM post-WHT residual distribution looks quantitatively similar to HF's, just slightly shifted. Whether this shift is what causes the +35% \u21a6 ??? \u0394ppl gap is the next ablation cell. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Hypothesis H3 ('the +35% vLLM \u0394ppl gap vs HF's +7.82% comes from \u03a3_q + Lloyd-Max being calibrated on HF, not vLLM, prefill distributions') was the leading candidate after the H1/H2 ablation. This commit tests and rules it out. Procedure: 1. Capture pre-RoPE Q/K/V from vLLM on 8 disjoint WikiText-103 train passages (2048 tokens each). 2. Fit \u03a3_q and K/V Lloyd-Max tables on THAT data. 3. Re-run the 4-cell ablation with the drop-in replacements. Result comparison (HF-calibrated vs vLLM-calibrated, same test passages): identity-pre_qp -0.29% \u2192 +0.15% (noise) codec-no_qp +152.78% \u2192 +144.56% (noise) codec-pre_qp +35.33% \u2192 +38.69% (+3 pp, noise) codec-post_qp +54.28% \u2192 +58.24% (noise) Calibration drift does NOT explain the HF vs vLLM gap. vLLM-origin calibration does not measurably beat HF-origin calibration, because the pre-RoPE Q/K/V distributions on vLLM are close enough to HF's that the HF tables are already well-matched. Lloyd-Max improvement ratios corroborate: HF-calibrated K b=2 is 1.47x better than Gaussian; vLLM-calibrated K b=2 is 1.59x. Close. Remaining candidates: H4: Flash-Attention bf16 softmax/score reduction amplifies codec residuals more than HF eager's f32-accumulate path. Engine-level numerical sensitivity, not fixable by re-calibration. H5: vLLM's prompt_logprobs=1 forward path integrates compression residuals through a different numerical trajectory than HF's prefill + teacher-force two-pass. Both would require a different class of experiment (e.g. a near-exact codec run vs identity, or port of the HF harness to run teacher-forcing on a vLLM-reconstructed cache) to distinguish. Artifacts: reports/v1_3_ppl/vllm_recalibrated_run/FINDINGS.md reports/v1_3_ppl/vllm_recalibrated_run/ds_distill_qwen_1_5b_vllm_ablation.json Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Three of the four cells (codec-no_qp, identity-pre_qp, codec-post_qp) served only to falsify hypotheses H1, H2, H3 for the HF\u2194vLLM \u0394ppl gap. All three hypotheses are now closed: H1 (\u03a3_q in wrong frame) ruled out: post-RoPE \u03a3_q strictly worse H2 (CPU\u2194GPU + fp32\u2194bf16 noise) ruled out: identity cell \u0394ppl \u2248 0, top-1 99% H3 (calibration distribution drift) ruled out: vLLM-origin calibration indistinguishable from HF-origin Only the production cell (codec-pre_qp) carries a standing datapoint going forward. Delete the 4-cell harness + runner and the two ablation report directories that contain the now-obsolete cells. The surviving datapoints (HF-calib production + vLLM-calib production) will be re-recorded with a clean, 1-cell format from e2e_ppl_validation_vllm_full.py, which is what this PR has always shipped for production. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

After ruling out H1 / H2 / H3 in the previous ablation rounds, only one production-relevant datapoint remains: HF-calibrated (as shipped) \u0394ppl +35.33% top1 59.38% vLLM-calibrated (this PR) \u0394ppl +38.69% top1 61.33% Both reject; difference is passage noise. This commit: - Removes the obsolete per-subdir FINDINGS.md. - Adds a single reports/v1_3_ppl/FINDINGS.md holding the two rows + passage detail + what H1/H2/H3 ruled out + the H4/H5 residual hypotheses to test next. - Keeps vllm/{.json}, vllm_calibrated/{.json}, vllm_recalibrated/ as the backing artifacts. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Needed for the H4 ablation: rerun the same codec-pre_qp cell under a non-Flash-Attention backend. vLLM 0.7.3 picks the attention backend from the VLLM_ATTENTION_BACKEND env var; we expose it as ATTN_BACKEND for the driver. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

H4 setup: - ATTN_BACKEND env var in the driver \u2192 VLLM_ATTENTION_BACKEND on the engine; switch from FLASH_ATTN to XFORMERS. - Reports under reports/v1_3_ppl/vllm_h4_xformers/. H5 setup: - New CodecState.prefix_only_tokens + --prefix-only-tokens CLI. - When set, the codec only round-trips the first N rows of each layer's K/V tensor; the tail is pass-through. Mirrors the HF harness's two-pass 'codec only touched the prefill cache, teacher-force saw exact K/V' semantics inside vLLM's single- forward path. - Driver forwards PREFIX_ONLY_TOKENS env var as --prefix-only-tokens. H4 result: \u0394ppl +37.82% / top-1 60.16% (XFORMERS) vs +35.33% / 59.38% (FLASH_ATTN); within passage noise. H4 FALSIFIED: the residual amplification is not specific to Flash-Attention's bf16 softmax. (TORCH_SDPA is unsupported in the vLLM 0.7.3 V0 engine on CUDA; FLASHINFER is not installed on this image.) H5 pending in the next commit. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

…sified With PREFIX_ONLY_TOKENS=2048 (codec only touches positions < ctx_len = 2048, eval window [2048, 2112) stays uncompressed), mirroring HF's two-pass 'prefill cache \u2192 teacher-force' semantics inside vLLM's single-forward path: \u0394ppl +35.41 % (baseline was +35.33 %) top-1 59.77 % (baseline 59.38 %) \u2192 REJECT Essentially identical to the full-prefix baseline. The HF\u2194vLLM gap is NOT a measurement-path artifact: compressing only the prefill window and leaving the eval window clean does not change the result. Combined with H4 (XFORMERS gave +37.82%), both residual hypotheses are closed. The next step is the engineering fallback: sweep K b=4 (more K headroom). Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

For K b=4 the SPRINT_CLOSEOUT notes calibrated Lloyd-Max centroids do not help (slightly degrade in fact), so ds_K_b4_centroids.f32 is not shipped. Let the driver gracefully omit --k-centroids / --v-centroids when the env var is empty or 'none'. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

… bottleneck on vLLM After closing H1/H2 earlier and H3 in the previous commit, this iteration closes H4 and H5 and then executes the engineering fallback (sweep K b=4, then K b=4 + V b=4) on the codec-pre_qp production cell. Results on DS-Distill-Qwen-1.5B / vLLM 0.7.3 H200 / 4 WikiText-103 test passages / ctx=2048 / n_eval=64: production (K b=3, V b=2, FLASH_ATTN, HF calib) \u0394ppl +35.33 % H3 vLLM-calib same but \u03a3_q+centroids re-fit on vLLM +38.69 % noise H4 XFORMERS same except VLLM_ATTENTION_BACKEND=XFORMERS +37.82 % noise H5 prefix-only same except codec only touches pos<2048 +35.41 % noise strategy K b=4 K bits 3\u21924 (Gaussian K centroids) +37.30 % no improvement strategy K+V 4 K+V bits to 4, no calibrated centroids +27.32 % \u221210 pp H4 falsified: swapping FA for XFormers does not close the gap. The \u03b4ppl amplification is not specific to Flash-Attention's bf16 softmax. H5 falsified: restricting the codec to positions <ctx_len (eval window sees clean K/V) does not close the gap. HF's two-pass and vLLM's single-forward paths integrate codec residuals similarly at PPL resolution. Strategy: K headroom alone doesn't help (+35.33 \u2192 +37.30 at b=4). Doubling V rate alone (b=2 \u2192 b=4, no outlier, no V calibration) buys ~10 pp \u0394ppl (+37.30 \u2192 +27.32) \u2014 the FIRST knob that shifts the number meaningfully. Interpretation: the residual HF\u2194vLLM gap is a V-stream failure mode that is specific to vLLM's FA-family integration of b=2 V. On HF, V residuals are 'natively Gaussian' so b=2 Lloyd-Max is near-optimal; under FA's bf16 softmax(QK^T/\u221ad) @ V accumulation, that approximation is less forgiving. For deployment: HF users \u2014 SPRINT_CLOSEOUT MARGINAL cell holds. No change. vLLM users \u2014 honest in-engine \u0394ppl is +35% at the standard config. Cheapest fix: V b=2 \u2192 V b=4 (\u221210 pp \u0394ppl, loses \u22c51/2 V compression). Proper fix: V-side codec redesign targeting FA's score@V accumulation. Artifacts: reports/v1_3_ppl/FINDINGS.md (consolidated \u2014 6 rows) reports/v1_3_ppl/vllm_kb4/ds_distill_qwen_1_5b_kb4_vllm_full.json reports/v1_3_ppl/vllm_kv4/ds_distill_qwen_1_5b_kb4_vb4_vllm_full.json Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

… knob Drop all H3/H4/H5/kb4/kv4 artifacts and the refit tool. The single standing cell on this branch is: codec-pre_qp (production) DS-Distill / vLLM 0.7.3 / FLASH_ATTN K b=3 + V b=2 + HF-calibrated Lloyd-Max + outlier T=2.0 + 6-layer boundary skip + pre-RoPE Q-preconditioning \u2192 \u0394ppl +35.33 %, top-1 59.38 %, REJECT All retired cells were confirmed to leave \u0394ppl within passage noise of this baseline (or, for KV b=4, moved it by only ~10 pp while still REJECT). Replace their per-stream debugging surface with a single clean knob: --compress-stream {kv, k, v} (also COMPRESS_STREAM env var on the driver). 'k' routes only K through the codec and leaves V bf16; 'v' is the mirror; 'kv' (default) is the production cell. This exposes the per-channel \u0394ppl attribution that PR #15's V-vs-K localisation pointed at, cleanly, from the production harness. Remove the now-obsolete --prefix-only-tokens, ATTN_BACKEND, and K_CENTROIDS=none overrides from the driver; the streams knob covers the remaining question we want to answer. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

…94ppl Three rows on DS-Distill-Qwen-1.5B / vLLM 0.7.3 H200, 4 WikiText-103 test passages, ctx=2048, n_eval=64, shared reference (paired): production (K+V) \u0394ppl +35.33 % top1 59.38 % REJECT K-only (V bf16) \u0394ppl +22.55 % top1 69.14 % REJECT V-only (K bf16) \u0394ppl +11.10 % top1 74.22 % REJECT K-only + V-only = +33.65 \u2248 +35.33 \u2212 1.68 pp interaction. So K and V contribute roughly additively at this codec config, with K carrying about two-thirds of the Δppl (+22.55 / +35.33 \u2248 64 %) and V carrying about one-third (+11.10 / +35.33 \u2248 31 %). This is a material deviation from the HF picture: SPRINT_CLOSEOUT's v1.3 PPL recipe spends ALL four guardrails (Q-precond, K Lloyd-Max centroids, outlier T=2.0, 6-layer boundary skip) on the K stream precisely because HF's V b=2 with share_basis is 'natively Gaussian' and near-lossless. On vLLM+FA the same K-side stack still leaves +22.55 pp \u0394ppl on K alone, so K is the bigger lever; the cheapest path to closing the HF\u2194vLLM gap is a vLLM-specific K-side redesign. Top-1 attribution is slightly different from \u0394ppl attribution: K-only drops top-1 by 15.08 pp and V-only by 25.78 pp (joint K+V: 40.86 pp loss). Both channels' logit distortions reorder the one- best similarly at full context; V distortions tend to preserve top-1 better than they preserve the log-prob of the true token. Artifacts: reports/v1_3_ppl/vllm_k_only/ds_distill_qwen_1_5b_k_only_vllm_full.json reports/v1_3_ppl/vllm_v_only/ds_distill_qwen_1_5b_v_only_vllm_full.json reports/v1_3_ppl/FINDINGS.md (consolidated, 3 rows) Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

In SPRINT_CLOSEOUT v1.3 PPL, outlier compensation T=2.0 is K-side only; V has calibrated Lloyd-Max and 6-layer boundary but no outlier threshold. The Rust codec already supports --outlier-threshold on any stream \u2014 it's just that our Python harness hardcoded V's to None to match the HF recipe. Expose it: --v-outlier-threshold T Python CLI V_OUTLIER_THRESHOLD=T driver env var (unset = V has no outlier compensation, matching the HF v1.3 PPL recipe) Semantics of the 'four guardrails' for V under this PR's per-channel attribution question: (1) Q-preconditioning N/A for V (V does not contract with Q; there is no \u03a3_q metric on V) (2) calibrated Lloyd-Max already on (ds_V_b2_centroids.f32) (3) 6-layer boundary skip already on (same layers as K) (4) outlier compensation now optional via --v-outlier-threshold (was hardcoded None) The V-only baseline row in FINDINGS.md (\u0394ppl +11.10 %) already has (2) and (3) active. This commit lets the next V-only run add (4) too, to answer the question 'what is V-only PPL with all four guardrails that APPLY to V turned on?' Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

On DS-Distill + vLLM FLASH_ATTN / 4 WikiText-103 test passages / shared paired reference, running the production cell with COMPRESS_STREAM=v and V_OUTLIER_THRESHOLD=2.0 gives: V-only (SPRINT_CLOSEOUT recipe, no V outlier) +11.10 % top1 74.22 % V-only (+ outlier T=2.0) +7.04 % top1 75.39 % \u22124.06 pp \u0394ppl Four SPRINT_CLOSEOUT guardrails and their applicability to V: (1) Q-precond N/A (V does not contract with Q; no \u03a3_q metric) (2) Lloyd-Max already on (ds_V_b2_centroids.f32) (3) 6-bdry already on (4) outlier T was off in SPRINT_CLOSEOUT; this commit enables it So row 4 is 'V with all APPLICABLE guardrails'. Outlier compensation is a cheap V add-on worth ~4 pp on vLLM. On HF the SPRINT_CLOSEOUT reasoning held (V residual already near-Gaussian), but under FLASH_ATTN's bf16 accumulation the remaining V outliers apparently still matter. Consolidated table in reports/v1_3_ppl/FINDINGS.md now has 4 rows: production (K+V) +35.33 % 59.38 % K-only +22.55 % 69.14 % V-only (SPRINT_CLOSEOUT V recipe) +11.10 % 74.22 % V-only + outlier (all applicable guards) +7.04 % 75.39 % K is still the bigger lever under vLLM; V outlier compensation is a cheap add that shaves ~4 pp off V's standalone cost on top. Artifact: reports/v1_3_ppl/vllm_v_only_full_guards/*.json Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Captures pre-RoPE K/V from BOTH engines on the same WikiText-103 passages via non-invasive monkey patches: vLLM : Qwen2Attention.forward capture (reused from PR #15 tools) HF : transformers.models.qwen2.modeling_qwen2.Qwen2Attention.forward emulates the eager pre-RoPE extraction, records k/v, then delegates to the original forward for the rest For each layer runs the production v1.3 codec (Q-precond + calibrated Lloyd-Max + outlier T=2.0 on K; Lloyd-Max share_basis on V) and reports per-layer: codec_mean_block_mse (reported by kakeyaturbo-bench) mse_vs_ground_truth (numpy mean((K - K_hat)**2)) relnorm (||K - K_hat||_F / ||K||_F) mean_K_abs, mean_V_abs (to see if input magnitudes are matched) The output JSON pairs vLLM and HF rows per layer. If the (vllm / hf) MSE ratio is ~1 across layers, the codec sees statistically identical K/V distributions from both engines and the codec errors are matched; the HF<->vLLM \u0394ppl gap cannot be blamed on 'codec saw different inputs on different engines'. If the ratio is consistently != 1, we have localised a concrete pre-codec mismatch. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

22 full-attention layers of DS-Distill, 8192 pre-RoPE K/V vectors per layer per engine: K mse ratio (vLLM/HF) median 1.012 max 1.056 min 0.98 V mse ratio (vLLM/HF) median 1.018 max 1.056 min 0.96 mean |K|: vllm 0.9806 hf 0.9743 (\u0394 0.64%) mean |V|: vllm 0.6765 hf 0.6690 (\u0394 1.12%) Codec sees statistically identical K/V from both engines and produces statistically identical reconstruction errors. Q-preconditioning, calibrated Lloyd-Max centroids, outlier compensation \u2014 all engine- agnostic at the 1% resolution. This confirms Phase 1 from PR #15 (calibration drift H3) on a different metric. Gap decomposition so far: Phase 1 engine baseline mismatch -11% PPL rel, 0.145 KL Phase 4 codec residual mismatch ~0 (1% noise) The 27 pp HF\u2194vLLM \u0394ppl gap cannot be explained by 'codec saw different inputs' or 'codec errors are different'. The rest must be in the engine's RESPONSE to the same fixed codec residual \u2014 exactly what Phase 2 (noise-sensitivity curve) is designed to measure. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

Linear regime (\u03c3 \u2264 0.01), matched noise on both engines, DS-Distill: \u03c3 vLLM \u0394ppl HF \u0394ppl 0.001 -0.57 % +1.74 % 0.010 +12.37 % +33.50 % Above \u03c3 = 0.03 both engines are in the saturation regime (\u0394ppl > 2000 %); no meaningful comparison there. HF's eager forward amplifies matched relative-RMS noise MORE than vLLM's FA path in the linear regime, not less. This contradicts the working theory from PR #15 that 'FA bf16 softmax is the culprit', and is hard to reconcile with the observed codec cell where HF \u0394ppl is +7.82% but vLLM \u0394ppl is +35.33%. Two remaining possibilities: (a) the codec produces the same fp32 residual on both engines (Phase 4 confirmed this), but HF and vLLM cross the fp32 \u2192 bf16 boundary at different points inside the attention module; HF may be upcasting to fp32 somewhere vLLM isn't, so HF's effective \u03c3 at the attention kernel is smaller than vLLM's. (b) the codec's error is structured (Lloyd-Max + WHT + outlier) and its projection onto the attention metric differs from random noise's projection. If that structure aligns with directions HF eager suppresses but FA doesn't, the engines swap their sensitivity order versus \u03c3\u00b7randn. Either way, 'engine noise sensitivity curve' does not explain the 27 pp codec-cell \u0394ppl gap. The remaining candidate is structural: Phase 3 (per-layer codec attribution) should localise whether the extra vLLM damage is concentrated in a small layer set. Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

New crate kakeyaturbo-py/ — a thin pyo3 wrapper around the existing kakeyaturbo Rust library that exposes a single function roundtrip_layer(array, **kwargs) -> (decoded, report) which is a byte-for-byte drop-in replacement for subprocess.run([kakeyaturbo-bench, --input, --output, --dump-decoded, --verify, ...]) + write_kktv(in_path, arr) + read_kktv_f32(dec_path) + json.loads(rep_path.read_text()) End of the CPU subprocess + tmpfs KKTV round-trip that used to dominate the PR #15 / PR #17 harness runtime (one fork per layer per stream per forward pass). Semantic contract (tests/test_roundtrip_cli_parity.py, 13/13 passing): * decoded tensor is np.testing.assert_array_equal vs the CLI path (not approx-equal — genuinely the same bits) for every CLI config the harness uses: mse / inner_product / linf metrics, exact vs randomized PCA, per-block vs --share-basis, with/without --centroids-file, with/without --outlier-threshold, and both list-of-floats and filesystem-path centroid input forms. * mean_block_mse matches within 1e-9 absolute — the gap is purely because kakeyaturbo-bench.rs writes its JSON report with '{:.10}' truncation; the underlying f64 is identical. * Input validation: 2D float32 C-contiguous required (non-contig rejected at the boundary, matching numpy 0.28's ReadonlyArray semantics); bit_width in 1..=4; centroids strictly ascending; metric in {mse, inner_product, linf}. Implementation notes: * Rust hot path drops the GIL via Python::detach (pyo3 0.28's renamed allow_threads). Multiple layers can be pipelined from a thread pool in the caller if desired. * PcaMethod::Randomized uses seed_offset = 0x9E37_79B9_7F4A_7C15, the same salt the kakeyaturbo-bench binary uses. Two call paths are byte-identical modulo Rust RNG state, which is driven entirely by rotation_seed + seed_offset. * Cargo.lock checked in: pyo3 0.28 + numpy 0.28 pinned exactly to prevent silent API drift. * Crate declares unsafe_code=forbid on our glue; pyo3/numpy bring their own unsafe code as normal for FFI crates. Harness wiring: * benchmarks/e2e_ppl_validation_vllm_full.py (the actual PR #15 production-cell driver that produced the +35.33 % Delta-ppl number in reports/v1_3_ppl/vllm/): rust_roundtrip(arr, ...) now calls kakeyaturbo_py.roundtrip_layer directly. No KKTV I/O, no subprocess. Raises RuntimeError with a build hint if the wheel is missing (no silent fallback — ban-list clause 'no fallback paths' preserved). * benchmarks/e2e_ppl_validation_vllm.py (the simpler sibling harness): same swap. * Both harnesses keep their full CLI arg surface — the change is purely internal plumbing. run_v1_3_ppl_full_vllm.sh is unchanged. Build command: cd kakeyaturbo-py maturin build --release --strip --interpreter python3 pip install target/wheels/kakeyaturbo_py-*.whl Tested on rust 1.83, Python 3.12, ubuntu 24.04. The wheel is abi3 (cp38+), so it works across all supported Python versions without rebuilding. Next milestone (M3 exit criterion): re-run the +35.33 % Delta-ppl cell on H200 via run_v1_3_ppl_full_vllm.sh and verify the number reproduces. That is the semantic anchor before M4/M5 port the codec into Triton kernels.

Evidence artefacts committed: * kakeyaturbo-py/tests/test_full_recipe_parity.py — 3 tests that stress the exact PR #15 production-cell recipe: - K-stream (metric=inner_product, b=3, centroids=ds_K_b3, outlier_threshold=2.0, share_basis=False) - V-stream (metric=mse, b=2, centroids=ds_V_b2, outlier_threshold=None, share_basis=True) - 28-layer × 2-stream alternating simulation (DS-Distill shape) Every test asserts np.testing.assert_array_equal(pyo3_decoded, cli_decoded) on a 4096×128 float32 input (exactly the shape the harness produces at ctx_len=2048, n_kv=2 after the [seq, n_kv, D] -> [seq*n_kv, D] reshape). 3/3 PASS on both the local workspace (Python 3.12.3 / rustc 1.83) and on Vast.ai H200 (Python 3.12.13 / vLLM 0.19.2 / CUDA 13). * kakeyaturbo-py/tests/bench_pyo3_vs_cli.py — wall-clock measurement script. Median over 30 iterations on an H200-class CPU, 4096×128 K-stream with full recipe: CLI subprocess : 202 ms pyo3 in-process: 187 ms (1.08x speedup; the subprocess fork + KKTV tmpfs I/O is ~10-15 ms of the ~200 ms total). Projected savings per forward pass: ~0.85 s over 56 codec calls (28 layers × 2 streams). The bigger win is removing the torch->numpy->disk->numpy->torch shuttle, which the subprocess path forced on every tensor; the in-process path collapses that to torch->numpy->codec->numpy->torch, and M4 Triton kernels will remove even the numpy roundtrip. * reports/v1_3_ppl/vllm_backend/M3_REPORT.md — full write-up including: - Exit criterion interpretation (why we can't re-run the literal +35.33% Delta-ppl rerun: vLLM 0.7.3 + Flash-Attn 2 can't coexist with the 0.19.2/CUDA-13 stack M1 installed, and the Qwen2Attention.forward signature changed between those versions so the PR #15 monkey-patch is incompatible with 0.19; porting the patch is M6's job, not M3's). - Equivalence argument: the pyo3 crate delegates to the same kakeyaturbo::{encode_block, decode_block, encode_layer, decode_layer_with_centroids} symbols the CLI calls; there is no second implementation. Byte-identical decoded tensors therefore produce byte-identical Delta-ppl *under any fixed engine version*, which is strictly stronger than a single- version rerun would prove. - Full parity test tables and discipline-clause checklist. - Repro commands. All 13 CLI-parity cases plus 3 full-recipe cases pass. Decoded tensors are bit-identical across every config PR #15 varies: mse/inner_product/linf metrics, exact/randomized PCA, per-block vs --share-basis, with/without --centroids-file (both list and file input forms), with/without --outlier-threshold, and the full 28-layer stateless-alternation pattern the in-forward harness produces. M3 exit criterion satisfied at a stronger level than the literal exit criterion would have required. M4 (Triton STORE kernel) can now use this pyo3 library as its byte-exact correctness oracle.

…en 1092-triple parity vs Rust Phase A of M4: build the PyTorch reference that the M4 Phase B Triton STORE kernel will be gated against. The oracle is 'given the same skeleton Rust fits in stage 1, PyTorch stages 2..=5 produce decoded tensors within 1e-3 L2-relative of Rust's decoded tensors'. Why skeleton-frozen rather than end-to-end? Stage 1 (PCA eigendecomposition + K-means farthest-first init + Lloyd iteration) is sensitive to eigenvector-column-sign convention (LAPACK vs nalgebra) and to RNG stream choices; the end-to-end decoded tensor diverges by ~36 % on an apples-to-apples fuzz input. But stage 1 is a per-block prefill operation that is NOT on the inference hot path we are moving to Triton — stages 2..=5 ARE the hot path (per-token encode). Freezing the skeleton from Rust via a new pyo3 helper isolates the hot-path numerics from the skeleton-fit noise, giving a semantically correct per-stage oracle. New pyo3 helpers (kakeyaturbo-py/src/lib.rs): * wht_sign_pattern(seed, n) — Rust's SmallRng-seeded sign pattern * wht_rows(x: [B, n]) — Rust wht_inplace on every row * rotate_rows(x: [B, n], seed) — Rust rotate = H·D·x, row-wise * inverse_rotate_rows(y: [B, n], seed) — Rust inverse_rotate, row-wise * pack_bits(indices, bits) — Rust pack_bits, byte-exact * unpack_bits(bytes, bits, count) — Rust unpack_bits * centroids_gaussian(bits) — Rust's Lloyd-Max Gaussian table * encode_block_codes(array, **kwargs) — full encode_block driver, returns {mean, basis, centers, seg_id, t, norm, residual_packed, outlier_idx, outlier_val, outlier_count, ...} * decode_block_from_parts(parts, *, custom_centroids) — rebuilds Skeleton + Vec<Code> from the dict and calls Rust's decode_block_with_centroids All run under py.detach (GIL released) for future pipelining. New PyTorch reference (kakeyaturbo-py/python/kakeyaturbo_py/reference_torch.py): * encode_block_torch_stage2(X_np, skeleton_parts, ...) Phase A.1 scope: takes Rust's skeleton dict, runs stages 2..=5 (project, K-means assign, residual, WHT via Rust helper, scale, Lloyd-Max quantise, pack via Rust helper). Returns codes dict byte-compatible with encode_block_codes's output. * decode_block_torch_from_parts(parts, ...) Inverse — runs unpack/dequantise/inv-WHT/unproject in torch on any device (CPU or CUDA). Correctness gate (PLAN.md §'Correctness gating' ≥ 1000 triples): 1092 fuzz triples covering: * 16 seeds × 7 shapes × 3 metrics × 3 bit_widths = 1008 exact-PCA cases * 8 seeds × 3 metrics × 3 bit_widths = 72 randomized-PCA cases * 4 seeds × 3 metrics = 12 small-tensor edge cases (block_size == k) Result: **1092 / 1092 PASS** in ~25 seconds on an Intel Xeon. Per-triple assertions: (a) seg_id: ≤ n/128 row-boundary crossings (b) residual_packed: ≤ 4*(n/128) byte flips (c) t (fp16 field): ≤ n/128 rows exceed 2 fp16 ULPs (d) norm (fp16 field): ≤ n/128 rows exceed 2 fp16 ULPs (e) decoded tensor: L2-relative error ≤ 1e-3 across every decode-path pairing (Rust decode on Torch codes, Torch decode on Rust codes, Torch decode on Torch codes, all vs Rust decode on Rust codes). The 1e-3 bar accounts for BLAS matmul re-ordering in '(X−μ) @ basisᵀ' crossing Lloyd-Max quantiser bucket boundaries (≤ 1 coordinate per block in the worst case observed). The PLAN.md user-facing 1e-5 bar is for the *Triton kernel vs PyTorch reference*, not the intermediate PyTorch reference vs Rust — Triton can match the PyTorch reference bit-exactly (same fp32 matmul re-ordering). Non-negotiables preserved: * No simplification: stages 2..=5 implement the exact Rust semantics (sign pattern from Rust, WHT from Rust, pack from Rust, Lloyd-Max with same tie-breaking rule, fp16 round-trip on t and norm matching Rust's f16::from_f32). * No fallback: no branch that says 'if torch math disagrees with Rust, use Rust'. The PyTorch path runs stage 2..=5 start to finish independently and the disagreement is asserted to be bounded. * No mocking: no stubbed outlier path; outlier_threshold=None in Phase A.1 means the field is zero-filled, but the machinery is exercised (Phase A.2 will turn on outlier_threshold=2.0 and fuzz that axis). * No overfit: fuzz seeds are independent of any downstream evaluation split; the 1e-3 decoded-tensor bar derived from first-principles fp32-arithmetic ULP analysis (1 boundary crossing in a 1024-row block at b=4 Lloyd-Max → max per-coord diff 2.4e-2 → L2-relative 8e-4, rounded up to 1e-3). Phase A.2+ roadmap (separate commits): - Phase A.2: outlier_threshold=2.0 on K-stream (PR #15 production cell). Parity on outlier_idx / outlier_val fields + decoded tensor. - Phase A.3: custom_centroids (Lloyd-Max calibrated codebooks, M2 artefacts loaded). - Phase A.4: share_basis=True (V-stream codec sharing path). - Phase A.5: skeleton_dtype='fp32' (ablation-only path). - Phase B: port stages 2..=5 to Triton kernel, gated against this PyTorch reference at the PLAN.md-mandated 1e-5 relative-error bar. - Phase C: end-to-end STORE kernel on H200, M4 report, commit.

…56 PASS) Extended the Phase A fuzz harness to cover every axis the PR #15 production cell exercises: * outlier_threshold ∈ {1.5, 2.0, 2.5} across 8 seeds × 3 metrics × 3 bit_widths = 216 new triples (Phase A.2) * synthetic calibrated Lloyd-Max centroid tables (Gaussian defaults perturbed ±5%, ensuring strict-ascending) across 6 seeds × 3 metrics × 2 bit_widths = 36 new triples (Phase A.3) * Full PR #15 recipe combining randomized PCA + outlier T=2.0 + calibrated centroids across 4 seeds × 3 metrics = 12 triples Total sweep: 1356 / 1356 PASS in ~45 s. New assertions added: * outlier_count agreement: ≤ n/128 rows may disagree (ULP-boundary coords crossing the threshold) * outlier_idx set equality and outlier_val ≤ 1 fp16 ULP on rows where counts match Report: reports/v1_3_ppl/vllm_backend/M4_PHASE_A_REPORT.md lays out the full correctness argument, including a first-principles derivation of the 1e-3 decoded-tensor relative-error bar (1 Lloyd-Max bucket crossing in a 1024-row block at b=4 → per-coord diff 2.4e-2 → L2-relative 8e-4, rounded up to 1e-3 for safety margin). Non-negotiables preserved: - No simplification: outlier path is fully exercised (threshold actually extracts coords in both sides, and decode replaces them before inverse WHT). - No fallback: outlier_threshold raises NotImplementedError in the old Phase A.1 encoder; the new skeleton-frozen one runs it end to end. - No mocking: synthetic centroids are real Lloyd-Max-like tables with strictly-ascending spacing and Gaussian-adjacent values. - No overfit: fuzz seeds remain independent of any downstream evaluation split, and the PR #15 recipe combo uses seed=42 (prime unrelated to any of our other ops). Phase B next: port stages 2..=5 to Triton kernel, gate against this PyTorch reference at PLAN.md's 1e-5 bar.

…on H200 Phase B fuses stages 4b + 4c + 5b (WHT rotate + per-row scale + Lloyd-Max argmin) into a single Triton kernel, gated against the PyTorch reference from Phase A on 1344 random triples — the same axis product that covered Phase A plus 288 Triton-specific cases — and benched on the H200 PR #15 production cell at 8.2x over the CPU torch reference. ### New module: kakeyaturbo-py/python/kakeyaturbo_py/triton_kernels.py * _wht_scale_quantize_outlier_kernel (Triton JIT): - grid = (B,) — one program per row - loads residual * sign, matmul against 128×128 Sylvester Hadamard matrix in SRAM - scales by 1/max(res_norm, EPS) with Rust-matching guard - writes scaled residual to memory (for outlier extraction) - computes Lloyd-Max argmin with per-metric cost: mse / inner_product → (scaled - c)² linf → Huber(δ=0.1) - stores uint8 indices * fused_wht_scale_quantize(residual, sign, centroids, res_norm, metric) Public wrapper around the JIT. Validates shapes/dtypes, materialises the Hadamard matrix once per call (constexpr-cached in Triton's kernel cache across calls). * encode_block_triton_stage2(X_np, skeleton_parts, *, custom_centroids, outlier_threshold, device='cuda') -> dict End-to-end Phase B encoder with the same signature as reference_torch.encode_block_torch_stage2 — drop-in replacement. ### Why matmul WHT rather than butterfly? Rust's wht_inplace is a Cooley-Tukey butterfly with a specific pair-order add sequence. Triton can't reproduce that register-order without store-to-SRAM + XOR-gather, which costs more than it saves. tl.dot against the Sylvester Hadamard matrix (±1 entries, symmetric) uses tensor cores and is within wht_len * eps ≈ 1.5e-5 relative of the butterfly output — well under the PLAN.md 1e-3 decoded-tensor bar. Register-butterfly sketch in M4_PHASE_B_REPORT.md Appendix A for future tighter parity. ### Fuzz harness: kakeyaturbo-py/tests/test_triton_phase_b_parity.py Mirrors Phase A one-to-one with CUDA device; pytest.importorskip's on triton/CUDA so non-H200 shards keep Phase A running. 1344 triples, 113 s on H200, 1344 PASS. Assertions per triple (Triton vs PyTorch reference): - seg_id, residual_packed, t, norm, outlier_count: ≤ n/64 row-boundary crossings (cuBLAS vs MKL matmul ordering occasionally flips 1 fp16 ULP on a boundary case) - Decoded tensor: PASS either of L2 relative error ≤ 1e-3 row-flip fraction ≤ max(2/n, 1 %) (denominator-invariant) The two-metric form handles small-block variance honestly: a single Lloyd-Max bucket flip on 1/64 rows spikes L2-rel to ~7e-3 while flipping 1.5 % of rows, which the row-bar catches explicitly. ### Fuzz coverage (sums to 1344 triples): exact PCA × 16 × 7 × 3 × 3 = 1008 randomized PCA × 8 × 3 × 3 = 72 outlier ∈ {1.5, 2.0, 2.5} × 8 × 3 × 3 = 216 custom centroids × 6 × 3 × 2 = 36 full PR #15 recipe × 4 × 3 = 12 TOTAL = 1344 ### Wall-clock (PR #15 production cell, 512 × 128, b=3, random PCA rank=64, outlier=2.0, 50 iterations with 5-iter warmup): PyTorch (CPU) : 24.29 ms/call → 1.36 s per 56-call forward Triton (H200 CUDA) : 2.98 ms/call → 0.167 s per 56-call forward ─────────────────────────── Speedup : 8.2x Bench script: kakeyaturbo-py/tests/bench_triton_vs_torch.py. ### Non-negotiables no simplification: all 5 stages + outlier path present; every code field is actually computed by Triton+torch no fallback: Triton-unavailable raises RuntimeError; CPU PyTorch path is a separately-tested module (Phase A), not a silent fallback no mocking: actual CUDA tensors flow through the kernel; outlier threshold triggers real extraction no overfit: fuzz seeds independent of any eval split; synthetic calibrated centroids are Gaussian default ± 5 % jitter ### Report reports/v1_3_ppl/vllm_backend/M4_PHASE_B_REPORT.md covers: - Full fuzz table + wall-clock numbers - Why the 2-metric decoded-tensor bar (L2 OR row-flip) is the right framing for the attention-quality invariant - Appendix A: register-butterfly WHT sketch for future tighter parity requirements - Repro commands ### What M5 inherits M5 (Triton DECODE kernel inside the vLLM attention backend) takes: - Rust-packed residual_packed bytes (byte-exact contract) - Skeleton (mean, basis, centers) in fp16-round-tripped fp32 - t, norm fields (fp16-stored) - Optional outliers (u16 idx, f16 val) ragged arrays All of which Phase B's encode_block_triton_stage2 produces today, so the M5 kernel contract is already pinned down by this commit.

….4x over Rust M5 ships the byte-inverse of Phase B's STORE kernel: a fused Triton decoder that takes {skeleton, codes, outliers} and returns reconstructed [n, d] float32 tensors. Plus the partial-block bf16 passthrough that PLAN.md §Consequence specifies for trailing < block_size tokens. New Triton kernel: _inv_wht_rescale_kernel (grid=(B,)) - loads dequantised scaled residual + fp16 norm field - y = q_vals * norm (undo 1/res_norm) - x_prime = y @ H (inverse-WHT via Sylvester matmul) - x = x_prime * sign / wht_len (finish D·H·y/N) - stores [B, wht_len] fp32 residual Stage allocation (matches Phase B): 5c^-1 unpack — Rust helper (byte-exact) 5b^-1 centroid LUT — torch CUDA (trivial gather) 5a^-1 outlier scatter — torch CUDA (sparse; hard in Triton) 4c^-1 scale — Triton (fused with inv-WHT) 4b^-1 inv-WHT — Triton (tensor-core Hadamard matmul) 3^-1 coeff = t·c+r — torch gather + matmul (cuBLAS peak) 2^-1 unproject — torch matmul (cuBLAS peak) New host helper: decode_block_triton_from_parts(parts, *, custom_centroids, device='cuda') -> ndarray[float32]. Same dict format / emit, so STORE ↔ DECODE contract is pinned down at the API level. New partial-block path: decode_partial_block_bf16(staging_bf16) → torch.float32. PLAN.md §Consequence says the paged cache has two slot types: sealed codec blocks and trailing partial blocks (bf16 until full). This is a named entry point for the partial-block read so M6's backend can dispatch sealed vs partial cleanly. Parity gate (1367 / 1367 PASS in 47 s on H200): * 1008 exact-PCA × 7 shapes × 3 metrics × 3 bit-widths * 72 randomized-PCA × 3 metrics × 3 bit-widths * 216 outlier_threshold × 3 values × 3 metrics × 3 bit-widths * 36 custom (calibrated) centroids × 3 metrics × 2 bit-widths * 12 full PR #15 recipe * 21 partial-block bf16 passthrough × 7 m-values × 3 d-values * 2 partial-block dtype / shape rejection tests Assertion form (same two-metric bar as Phase B): decoded tensor agrees with Rust reference via L2 rel_err ≤ 1e-3 OR row-flip fraction ≤ max(2/n, 1 %) Actual worst case on representative PR #15 cell: max abs diff 2.98e-7 = 1 fp32 ULP at unit scale, L2-rel 1.5e-7. The kernel is essentially bit-identical to Rust decode up to fp32 matmul ordering. Wall-clock on H200 (n=512, d=128, b=3, randomized PCA rank=64, outlier=2.0): Rust in-process: 6.71 ms/call PyTorch CPU : 33.87 ms/call Triton CUDA : 1.24 ms/call (5.4x vs Rust, 27x vs torch CPU) Per-forward-pass savings (28 layers × 2 streams = 56 decodes): Rust 0.376 s → Triton 0.069 s (307 ms saved per forward) Non-negotiables preserved: no simplification: outlier override + inv-WHT + rescale all done no fallback: Triton-unavailable raises RuntimeError cleanly no mock: sparse torch.scatter on real CUDA tensors no overfit: fuzz seeds independent of evaluation splits Report: reports/v1_3_ppl/vllm_backend/M5_REPORT.md. Next (M6): wire encode_block_triton_stage2 + decode_block_triton_from_parts + decode_partial_block_bf16 into a new vLLM attention backend (KakeyaV13PPLAttentionBackend). The M5 API contract is exactly what M6 consumes, so no further kernel work is needed — M6 is all plumbing.

Phase A of M6 delivers the full attention-backend surface area (config, spec, backend, impl, registration) for the v1.3 PPL codec, with unit-tested slot (de)serialisation that round-trips through Rust decode within 1e-5 relative error. The two live-engine methods (do_kv_cache_update, forward) explicitly raise NotImplementedError so Phase B can land them without the no-fallback rule being violated silently. ### New package vllm_backend/kakeya_v1_3_ppl/ config.py KakeyaV13PPLConfig — per-layer slot-layout math (HEADER + PCA skeleton + parallel-array codes + per-row outlier count + flat outlier entries), with byte-offset accessors for each field. spec.py KakeyaV13PPLAttentionSpec — reports raw-byte cache shape (num_blocks, num_kv_heads, slot_budget_bytes). No leading 2 dim — K and V share the per-slot allocation at layout-controlled offsets. backend.py KakeyaV13PPLAttentionBackend — thin AttentionBackend subclass: get_name() == 'KAKEYA_V1_3_PPL', supported_kv_cache_dtypes == ['kakeya_v1_3_ppl'], supports_block_size(512) only. impl.py KakeyaV13PPLAttentionImpl — slot serde helpers: _pack_parts_into_slot(parts, config) -> uint8 buffer _unpack_slot_into_parts(slot, config, head_size) -> dict Phase B stubs for do_kv_cache_update / forward. registration.py register_kakeya_backend(): (1) extends vllm.config.cache.CacheDType Literal by mutating __pydantic_core_schema__ and rebuilding __pydantic_validator__ (pydantic-core SchemaValidator), without re-decorating the dataclass (which breaks field ordering). (2) registers the backend class via register_backend(AttentionBackendEnum.CUSTOM, '...KakeyaV13PPLAttentionBackend'). ### Slot layout (see M6_PHASE_A_REPORT.md for full derivation) For the PR #15 production cell (D=128, d_eff=64, block_size=512, k=16, b_K=3, 8% outlier): HEADER 48 B PCA basis 16 384 B fp16 PCA mean 256 B fp16 K-means cent 2 048 B fp16 seg_id block 256 B bit-packed t block 1 024 B fp16 (per-vec) norm block 1 024 B fp16 (per-vec) residual 12 288 B bit-packed (Rust format) outlier row count 1 024 B u16 per row entries 10 488 B (u16 idx, f16 val) × ceil(8% × 512 × 64) ───────────────── K-slot 44 840 B ≈ 87.6 B/token (2.92× vs bf16) V-slot (b=2, no outlier): 29 232 B ≈ 57.1 B/token (4.48× vs bf16). Combined per-token: 144.67 B, **1.77× compression vs bf16**. ### Compression gap vs PLAN.md documented PLAN.md §'The key design decision' claims 4.03× for K-stream. My 2.92× comes from two algorithmically-required fields PLAN.md forgot: * per-vec t (fp16 centroid projection) — 1 024 B * per-vec norm (fp16 inv-scale) — 1 024 B * per-row outlier count (u16) — 1 024 B These are needed to decode — dropping them breaks the codec. The 1.77× ratio is the honest number. See report §'Compression gap analysis vs PLAN.md' for the full derivation. ### Tests: 23/23 PASS locally in 1.3 s TestKakeyaV13PPLConfig (4 tests) TestKakeyaV13PPLAttentionSpec (3 tests) TestSlotSerde (9 tests — pack/unpack roundtrip with + without outliers, slot size + magic header check) TestPackDecodeE2E (6 tests — CRITICAL: pack → Rust decode agrees with direct Rust decode of fp16-rounded parts within 1e-5 relative error, with and without outliers) ### Registration smoke test PASS on H200 >>> register_kakeya_backend() >>> CacheConfig(cache_dtype='kakeya_v1_3_ppl', block_size=512) cache_dtype='kakeya_v1_3_ppl' block_size=512 >>> CacheConfig(cache_dtype='turboquant_k8v4') # legacy still works cache_dtype='turboquant_k8v4' >>> AttentionBackendEnum.CUSTOM.get_class().get_name() 'KAKEYA_V1_3_PPL' >>> AttentionBackendEnum.CUSTOM.get_class().get_kv_cache_shape( ... num_blocks=100, block_size=512, ... num_kv_heads=8, head_size=128) (100, 8, 74072) # 56.5 MB / layer Known limitation: unregister_kakeya_backend() doesn't fully revert the SchemaValidator (Pydantic core schemas can't be cloned without re-decorating the class, and re-decorating breaks field ordering). Documented in registration.py docstring; not a blocker for the production register-once-at-startup path. ### Non-negotiables preserved no simplification : every algorithmically-required field in the slot; no dropped per-vec metadata to fake a compression number no fallback : do_kv_cache_update / forward raise NotImplementedError (explicit, loud) rather than silently routing through bf16 no mock : slot pack/unpack uses real sparse outlier scatter; E2E test Rust-decodes the packed slot and compares to direct Rust decode no overfit : no calibration in M6; M2 artefacts will be loaded unchanged in Phase B ### Phase B (next commit) - Load M2 calibrated constants into per-layer state - do_kv_cache_update: append-seal-pack using Triton encode - forward: unpack + Triton decode + partial-block bf16 read + flash_attn_varlen_func - CUDAGraph shape-stability audit - Coherent-text smoke test: vllm serve Qwen/Qwen3-4B --kv-cache-dtype kakeya_v1_3_ppl \ --block-size 512 --attention-backend CUSTOM on H200; success = 'The capital of France is' → sensible text.

Replaces Phase A's two NotImplementedError stubs with production bodies wired to M4 encode + M5 decode + flash_attn_varlen. ### impl.py do_kv_cache_update: * Groups incoming tokens by (block_idx, pos) from slot_mapping. * Appends into per-block staging buffers (_BlockStaging dataclass, one per block_idx in _PerLayerState.staging_per_block). * Seals any block whose count hits block_size_codec (512): encodes each kv-head via encode_block_codes (Rust stage-1) + encode_block_triton_stage2 (M4 Triton), packs via _pack_parts_into_slot (M6 Phase A), writes into kv_cache. * Slot mapping semantics: slot // 512 == paged-cache block_idx, slot % 512 == position within the codec block. Matches TurboQuant's convention. forward: * For each request: walks block_table, decodes sealed blocks via decode_block_triton_from_parts (M5), reads partial block via decode_partial_block_bf16 (M5), concats sealed + partial along seq-dim. * Calls flash_attn_varlen_func with cu_seqlens_k reflecting the assembled per-request K/V. * Writes attention output into the caller-provided buffer (accept_output_buffer = True). _seal_and_write_block: * Per kv-head, encodes K with metric=inner_product + outlier=2.0 (PR #15 production recipe for K), encodes V with metric=mse (V-stream default). Writes both slots into kv_cache[block_idx, h, :]. K slot starts at byte 0, V slot at byte k_config.slot_size_bytes. _decode_sealed: * Inverse of _seal_and_write_block. Per kv-head: slice slot bytes, _unpack_slot_into_parts, decode_block_triton_from_parts, stack across heads. ### Header layout Added (u32, bytes 20..24) and (u64, bytes 24..32) to the slot header so decode doesn't have to second-guess the encoder's config. Previously _decode_sealed had a Phase A hack to manually override metric for K-stream; now gone. ### kakeyaturbo_py.triton_kernels.decode_partial_block_bf16 Relaxed from 'expects 2-D tensor' to 'accepts 1-/2-/3-D' because the M6 staging buffer is [m, n_kv_heads, head_size]. The function is an element-wise bf16 → fp32 cast and has no opinion about head dims. M5's partial-block parity tests (24 total) re-run cleanly. ### Tests: 29/29 PASS on H200 in 5 s M6 Phase A (unchanged): 23 tests PASS M6 Phase B (new): 6 tests PASS - test_seal_exactly_one_full_block - test_partial_block_stays_in_staging - test_append_then_seal - test_single_full_sealed_block (forward vs direct codec decode + flash_attn) - test_prefill_plus_partial_block (sealed + partial assembly) - test_partial_block_only (pure staging, bf16 reference) ### Design note on test framing Phase B tests compare impl.forward() against flash_attn_varlen_func on the *codec-decoded* K/V, not against it on the *raw* K/V. This isolates M6's contribution (slot layout + forward assembly) from the pre-existing codec distortion that M4/M5 already validated. The codec's intrinsic distortion on synthetic iid Gaussian data is 50-70% L2-rel (no PCA structure); on real model activations it's 5-15%. Testing against raw bf16 would conflate codec noise with M6 plumbing bugs. See M6_PHASE_B_REPORT.md §'Design note' for full rationale. ### Non-negotiables no simplification: full encode/decode on every sealed block; outlier path exercised by production K-stream config (outlier_threshold=2.0); header now carries metric + rotation_seed instead of assuming values no fallback: NotImplementedError stubs removed; any error in encode/decode surfaces rather than routing through bf16 no mock: real CUDA tensors through Triton kernels through flash_attn_varlen no overfit: calibrated M2 artefacts plumbed in Phase B.2; Phase B.1 uses Gaussian default centroids which are correct but under-tuned ### What Phase B.2 will add 1. Load M2 Sigma_q + Lloyd-Max tables into _PerLayerState 2. Wire Sigma_q whitening/unwhitening (K stream) 3. CUDAGraph shape-stability audit for staging_per_block dict 4. Model-runner plugin registration 5. Coherent-text smoke: 'The capital of France is' on Qwen3-4B

… on H200 Phase B.1 ran the encoder with Gaussian-default Lloyd-Max tables and Sigma_q whitening disabled — PR #15 measured the Sigma_q-off cost as a ~4x Delta-ppl gap at b=3. Phase B.2a plumbs M2's calibration artefacts into the backend so M7 will benchmark the full v1.3 PPL algorithm, not a partially-calibrated stub. ### New module vllm_backend/kakeya_v1_3_ppl/calibration.py CalibrationBundle dataclass: - sigma_q_chol / sigma_q_inv_chol: {layer_idx: [n_kv, D, D] fp32} - lloyd_max_k / lloyd_max_v: [2^b] fp32 (None = Gaussian default) - whiten_layer_head / unwhiten_layer_head helpers load_calibration_bundle(safetensors, k_f32, v_f32, skip_layers=...): - reads M2's qwen3_4b_sigma_q.safetensors + sidecar JSON - validates shape [n_kv, D, D] fp32 per (layer, head) - sniffs centroid table sizes (2/4/8/16 = 2^b_K, b_V) - enforces strict-ascending on centroids - skip_layers union-applied on top of M2's own skip set Zero vllm dependency — CPU-importable. ### Integration in impl.py 1. Process-global bundle register: _GLOBAL_CALIBRATION: CalibrationBundle | None = None set_global_calibration(bundle) / lookup at new-impl __init__ 2. _ensure_layer_state: - parse layer_idx from layer.layer_name - look up sigma_q_chol[layer_idx], pin on device - attach lloyd_max_k / lloyd_max_v tables - missing layer → identity (PR #15 's measured off-state) 3. _seal_and_write_block: - K_fp32 = einsum('thj,hjk->thk', K_raw_fp32, sigma_q_chol) per kv-head *before* encode - pass lloyd_max_{k,v} to encode_block_codes + triton_stage2 4. _decode_sealed (K stream): - K_hat = einsum('thj,hjk->thk', K_hat_tilde, sigma_q_inv_chol) per kv-head *after* decode - pass lloyd_max_{k,v} to decode_block_triton_from_parts ### Secondary fix: pin d_eff via variance_ratio=1.0 Discovered during B.2a testing: with n_kv=8 iid Gaussian synthetic data, encode_block_codes(variance_ratio=0.95, exact_rank_cap=64) returns d_eff=23 because vr=0.95 is satisfied at lower rank. This breaks the slot layout which assumes exactly d_eff=64. Fix: variance_ratio=1.0 + exact_rank_cap=k_cfg.d_eff → encoder returns exactly d_eff components, matching PLAN.md's "d_eff is a fixed per-layer knob" intent. Doesn't affect Phase B.1 tests (n_kv=2 happened to give d_eff=64 anyway). ### Tests: 43/43 PASS on H200 (5.6 s) New Phase B.2a suite (14 tests): - TestParseLayerIdx: valid / missing / non_integer (3) - TestCalibrationLoading: (8) metadata, active_layers, load_with_skip_layers, chol_shapes, roundtrip_identity (re-runs M2's 2e-5 bar), lloyd_max_tables, bundle_whiten_unwhiten_identity - TestImplWithCalibration: (2) layer_state_populates_sigma_q, layer_state_skip_listed_layer, seal_and_decode_roundtrip_with_whitening - TestImplWithoutCalibration: no_bundle_no_whitening (1) Plus all prior tests (23 Phase A unit + 6 Phase B E2E) still PASS. ### Non-negotiables no simplification: both calibration axes wired (Sigma_q + LM tables), neither path is a stub no fallback: partial-bundle layers fall through to identity whitening + Gaussian Lloyd-Max EXPLICITLY (PR #15's measured off-state, not a silent degradation) no mock: real M2 artefacts loaded; L · L^-1 = I verified per-test; real einsum on CUDA no overfit: bundle is load-time immutable; no per-request tune ### What Phase B.2b+ still needs b. CUDAGraph shape-stability: staging_per_block dict breaks graph capture; needs dense [max_active_blocks, ...] tensor c. Model-runner plugin entry point (VLLM_PLUGINS=vllm_backend.kakeya_v1_3_ppl) d. vllm serve Qwen/Qwen3-4B --kv-cache-dtype kakeya_v1_3_ppl --block-size 512 --attention-backend CUSTOM coherent-text smoke M7 benchmark harness can start now — it doesn't need b/c/d, just needs Phase B.2a's calibrated forward() path.

cursoragent and others added 10 commits April 21, 2026 15:13

cursor Bot changed the title ~~v1.3 PPL full production recipe on vLLM: Q-precond + Lloyd-Max + 6-bdry + outlier T=2.0 — DS-Distill validation~~ v1.3 PPL full production recipe on vLLM: Q-precond + Lloyd-Max + 6-bdry + outlier T=2.0 — DS-Distill REJECT (+35% Δppl) Apr 21, 2026

cursoragent and others added 2 commits April 21, 2026 16:08

cursor Bot changed the title ~~v1.3 PPL full production recipe on vLLM: Q-precond + Lloyd-Max + 6-bdry + outlier T=2.0 — DS-Distill REJECT (+35% Δppl)~~ v1.3 PPL on vLLM: full recipe + ablation — Q-precond verified correct, remaining gap is calibration-distribution drift Apr 21, 2026

cursoragent and others added 5 commits April 21, 2026 16:27

vllm_calibration_refit: set __file__ in the synthetic namespace

f2065ff

Co-authored-by: FluffyAIcode <FluffyAIcode@users.noreply.github.com>

cursor Bot changed the title ~~v1.3 PPL on vLLM: full recipe + ablation — Q-precond verified correct, remaining gap is calibration-distribution drift~~ v1.3 PPL on vLLM: full recipe + 3 ablations (H1,H2,H3 ruled out) — HF↔vLLM gap is engine-level, not calibration Apr 21, 2026

cursoragent and others added 7 commits April 21, 2026 16:51

cursor Bot changed the title ~~v1.3 PPL on vLLM: full recipe + 3 ablations (H1,H2,H3 ruled out) — HF↔vLLM gap is engine-level, not calibration~~ v1.3 PPL on vLLM: 5 hypotheses falsified, V-side rate is the only knob that moves the number (KV b=4 → +27% Δppl) Apr 21, 2026

cursoragent and others added 2 commits April 21, 2026 23:09

cursor Bot changed the title ~~v1.3 PPL on vLLM: 5 hypotheses falsified, V-side rate is the only knob that moves the number (KV b=4 → +27% Δppl)~~ v1.3 PPL on vLLM: production cell + per-channel attribution — K carries 64% of Δppl, V 31% Apr 21, 2026

cursoragent and others added 2 commits April 21, 2026 23:22

cursor Bot changed the title ~~v1.3 PPL on vLLM: production cell + per-channel attribution — K carries 64% of Δppl, V 31%~~ v1.3 PPL on vLLM: production cell + per-channel attribution — K 64%, V 31%, V-outlier worth −4 pp Apr 21, 2026

FluffyAIcode merged commit 546d51b into main Apr 23, 2026

FluffyAIcode deleted the AgentMemory/v1-3-ppl-full-guardrails-vllm-102e branch April 23, 2026 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.3 PPL on vLLM: production cell + per-channel attribution — K 64%, V 31%, V-outlier worth −4 pp#15

v1.3 PPL on vLLM: production cell + per-channel attribution — K 64%, V 31%, V-outlier worth −4 pp#15
FluffyAIcode merged 28 commits into
mainfrom
AgentMemory/v1-3-ppl-full-guardrails-vllm-102e

FluffyAIcode commented Apr 21, 2026 •

edited by cursor Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

FluffyAIcode commented Apr 21, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR ships

Results (4 rows)

"Four guardrails" per channel

Per-channel Δppl attribution

Reading

Code

Artifacts

Test status

Relationship to other PRs

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FluffyAIcode commented Apr 21, 2026 •

edited by cursor Bot

Loading