Kakeya vs TurboQuant+ head-to-head on the same open-source models

[FluffyAIcode]

Summary

Byte-for-byte comparison of Kakeya KV cache compression vs
TurboQuant+ (PolarQuant + WHT + scalar quantization, from
TheTom/turboquant_plus)
on the same captured KV tensors, same models, same context
lengths.

Both methods were run on the same machine (CPU-only, 15 GiB RAM, BF16,
eager attention) using the same harness. TurboQuant+ was installed
from its official Python prototype; Kakeya uses the codec and standard
preset already in this repo (block_size=512, residual_length=256, d_res=8, K=16, variance_ratio=0.95).

Headline (bf16 baseline, total cache)

TurboQuant+'s ratio is context-independent (per-token scalar
quantization). Kakeya's ratio grows with context (block skeleton
amortizes). Both facts are visible in the same table:

2k tokens

Model	Kakeya bf16	turbo4	turbo3	turbo2
qwen2_5_0_5b (hd=64)	1.68×	3.77×	4.92×	7.11×
smollm2_1_7b (hd=64)	1.72×	3.77×	4.92×	7.11×
qwen3_0_6b (hd=128)	2.37×	3.88×	5.12×	7.53×
deepseek_r1_distill_qwen_1_5b (hd=128)	2.10×	3.88×	5.12×	7.53×
glm_edge_1_5b (hd=128)	2.21×	3.88×	5.12×	7.53×
glm_edge_4b (hd=128)	2.27×	3.88×	5.12×	7.53×
gemma4_e2b (hybrid)	1.58×	3.96×	5.26×	7.84×

128k tokens (Kakeya = byte-exact extrapolation)

Model	Kakeya bf16	turbo4	turbo3	turbo2
qwen2_5_0_5b	2.15×	3.77×	4.92×	7.11×
smollm2_1_7b	2.25×	3.77×	4.92×	7.11×
qwen3_0_6b	4.51×	3.88×	5.12×	7.53×
deepseek_r1_distill_qwen_1_5b	3.41×	3.88×	5.12×	7.53×
glm_edge_1_5b	3.70×	3.88×	5.12×	7.53×
glm_edge_4b	3.88×	3.88×	5.12×	7.53×
gemma4_e2b	4.29×	3.97×	5.27×	7.86×

Bold Kakeya entries = it surpasses turbo4 there. Kakeya never surpasses turbo3.

Where Kakeya is better (reconstruction quality on K)

Measured on the same captured KV tensors — first full-attention layer,
first compressed block (512 rows):

Model	head_dim	Kakeya K MSE	turbo3 K MSE	Ratio
qwen2_5_0_5b	64	2.6e+01	2.07e+02	8× better
qwen3_0_6b	128	4.3e+00	8.99e+01	21× better
deepseek_r1_distill_qwen_1_5b	128	3.5e+00	1.44e+03	411× better
gemma4_e2b	512	1.3e-03	2.96e-03	2× better

On the exact models TurboQuant+'s own README flags for the q8_0-K + turbo-V rescue config, Kakeya handles K cache reconstruction orders of magnitude better than symmetric turbo3. The reason is simple: TurboQuant+'s codebooks are calibrated for unit-norm vectors, and the K norms in the Qwen family are large and heterogeneous. Kakeya's PCA on raw values absorbs that variance without per-vector renormalization.

On V cache, TurboQuant+ is 2–10× lower MSE — a consequence of its per-coordinate bit-depth doing well on near-Gaussian rotated V values, while Kakeya's d_res=8 sparse residual discards per-row information V cache actually uses.

Architecture observations

• TurboQuant+ compresses within a single KV vector (per-coord scalar quantization after WHT rotation).
• Kakeya compresses across neighboring KV vectors (per-block PCA + spherical K-means + top-k residual).
• They target orthogonal sources of redundancy and compose naturally. An obvious next step is:
  • K cache: Kakeya (wins big on Qwen family MSE)
  • V cache: TurboQuant+ (wins cleanly on V MSE)
    This should land around 8–10× combined ratio with better quality than either alone.

What's in this PR

File	Purpose
compare_kakeya_vs_turboquant.py	Loads any HF model → captures KV → runs both codecs on the same tensors → writes byte-accurate JSON
run_comparison_matrix.sh	Orchestrator: runs the 7-model × {2k, 4k, 8k} sweep
reports/compare/<model>/compare_<ctx>.json	Per-row byte-level reports (14 files)
reports/compare/SUMMARY.md	Side-by-side tables, MSE comparison, takeaways
.gitignore	Excludes the cloned turboquant_plus/ from this repo

Reproducing

# 1. Clone TurboQuant+ and install its Python prototype
git clone https://github.com/TheTom/turboquant_plus /workspace/turboquant_plus
pip install -e /workspace/turboquant_plus

# 2. Run a single comparison row
python3 compare_kakeya_vs_turboquant.py \
  --model-path models/Qwen3-0.6B \
  --model-name qwen3_0_6b \
  --context-tokens 8192 \
  --out reports/compare/qwen3_0_6b/compare_8192.json

# 3. Reproduce the full matrix
./run_comparison_matrix.sh

Limitations

• Memory/MSE only. End-to-end perplexity and NIAH would require a shared evaluation protocol; TurboQuant+'s published PPL numbers (+0.23% for turbo4, +1.06% for turbo3 on wikitext-2) and Kakeya's (identical greedy decode at 2k on Gemma 4) are not directly comparable.
• Per-token vs per-block. TurboQuant+ compresses from token 1; Kakeya needs block_size + residual_length tokens to begin compressing. For prompts shorter than ~768 tokens Kakeya is effectively DynamicCache.
• Python prototype only. TurboQuant+ has a production-grade llama.cpp fork across Metal/CUDA/HIP; Kakeya is a pure-Python transformers.Cache subclass. The "zero kernel work" advantage of Kakeya is real but comes with a CPU-side runtime cost the TurboQuant+ C port does not pay.

Environment

CPU-only x86_64, 15 GB RAM, BF16, eager attention, torch==2.11.0+cu130, transformers==5.5.4. TurboQuant+ Python prototype version captured at clone time.