Follow-up to #61 / #65.
Quality (mean KL vs FP32) was measured only at ~300 tokens; decode cost was measured out to kv_len ~3072 (where INT4 is net-faster). What's unverified is whether INT4 / INT4+ROT stays near-lossless at long context — accumulated quantization error over thousands of positions.
Task: extend tests/bench_kv_quality (or a variant) to teacher-force a long document (2–4k tokens) and report mean KL per mode at, say, the last 512 positions vs the first 512. Rotation is position-independent so quality should hold, but it's an assumption.
Impact: moderate — confirms (or refutes) that the "near-lossless" claim holds in the regime packed INT4 is actually for (long chats, RAG, agents).
Follow-up to #61 / #65.
Quality (mean KL vs FP32) was measured only at ~300 tokens; decode cost was measured out to kv_len ~3072 (where INT4 is net-faster). What's unverified is whether INT4 / INT4+ROT stays near-lossless at long context — accumulated quantization error over thousands of positions.
Task: extend
tests/bench_kv_quality(or a variant) to teacher-force a long document (2–4k tokens) and report mean KL per mode at, say, the last 512 positions vs the first 512. Rotation is position-independent so quality should hold, but it's an assumption.Impact: moderate — confirms (or refutes) that the "near-lossless" claim holds in the regime packed INT4 is actually for (long chats, RAG, agents).