fix: passthrough BF16 for vocab tensors + Lance upgrade roadmap + WAV validity test

[AdaWorldAPI]

Summary

Follow-up to #177 (merged) with the correctness fix for the text-embedding collapse, a planning doc for Lance 2 → 4/5 migration, and an end-to-end validity test producing actual WAV audio.

SHA	What
3b084d1	Passthrough BF16 for n_rows > 8192 — HCLAM 256×256 (landed in #177) collapsed WORSE than RVQ on the vocab embedding (cos=0.004 vs 0.054). Root cause: vocab rows are near-orthogonal in 2048-d; single-centroid tree quantization cannot synthesize directions. Skip compression on the one vocab-sized tensor and ship it as BF16. Result: codec token match 225/225 = 100.0% on Qwen3-TTS-0.6B.
f6ed834	docs/LANCE_UPGRADE_ROADMAP.md (161 L) — Planning doc for Lance 2→4/5 migration. 9 features relevant (IVF_RQ, IVF multi-split PR #6423, HNSW fp16 partition assignment, CacheBackend, distributed segment builds, BF16 PyTorch ingest, pre-transposed PQ SIMD, file format 2.3, Hamming HNSW). DataFusion 51→52.1 is the primary blocker. 5-phase plan.
6e8ee98	tts_full_inference mkdir_p fix — example panicked on fresh-clone because codebooks/ parent dir didn't exist. Now creates it. Surfaced while running the end-to-end validity test that produced the WAV.

End-to-end validity test (new signal)

The #177 "225/225 codec token match" only proved raw inference == RVQ-reconstructed inference — stable codec, didn't prove raw itself was correct. A single-token-constant bug would still show 225/225.

This session ran tts_full_inference.rs end-to-end on Qwen3-TTS-0.6B with prompt "Hello world, this is a test of text to speech synthesis using a compressed neural network running entirely in Rust.":

[1] Tokenize: 22 tokens
[2] Embed: RMS=0.013 in 1.38s
[3] 28 talker layers: RMS climbs 0.66 → 1.01 (expected growth)
[3.5] codec_head + re-embed: RMS=0.026
[4] 5 code_predictor layers: RMS 0.23 → 1.37
[5] 128 autoregressive steps in 50.12s
    Step 0:   tokens=[68, 151, 102, 40]
    Step 32:  tokens=[254, 183, 154, 199]
    Step 64:  tokens=[92, 48, 158, 42]
    Step 96:  tokens=[31, 81, 196, 221]
    Step 127: tokens=[142, 251, 253, 178]
[6] Decoder latent RMS=2.29
[7] WAV: /home/user/models/tts_real_speech.wav (65580 bytes, 1.37s)

WAV statistics (mono 24 kHz 16-bit, 32,768 samples):

Metric	Value	Interpretation
RMS	0.0865 (−21 dB)	normal speech loudness
Peak	1.0000	hits limiter (tanh/snake)
Zero-crossings	11,968 /s	high — fricatives / unvoiced
Energy by time decile	0.097, 0.092, 0.066, 0.087, 0.068, 0.059, 0.131, 0.075, 0.067, 0.096	varies — real amplitude modulation

Not silence, not constant, not noise. Codec tokens vary across all 128×16 positions. Wiring is real — 33 transformer layers + codec head + 5 code_predictor layers + RVQ dequant + conv decoder all work together and emit dynamic audio.

Storage note (deferred)

With the passthrough fix active, total storage is still 1:1.39 — net loss because per-tensor RVQ codebooks at k=[256, 512, 1024, 4096] are individually larger than the BF16 tensors they compress. The correctness gate passed; storage ratio is a separate optimization track — see follow-up directions below.

Follow-up directions (out of scope for this PR)

• BGZ-HHTL-D schema extension — user flagged it. Current Slot D (16-bit tree) + Slot V (16-bit BF16 scalar residual) is 4 B/row but can't point in specific directions on 1024-d rows. Proposed: add Slot L (8 bytes of i8 or TurboQuant on shared SVD basis) → 12 B/row total, ρ > 0.99 per row. Reuses matryoshka.rs SVD basis already in repo. Would give ~50:1 ratio on the vocab embedding (1.8 MB vs 622 MB BF16) at cos > 0.98.
• Family codebook restructure — the existing 2+4+8 = 14-bit address space already supports 16,384 effective entries if HIP-indexed sub-palettes replace the single flat 256-entry palette. Zero wire-format change.
• Lance 4.0 IVF_RQ + multi-split adoption — see docs/LANCE_UPGRADE_ROADMAP.md. Multi-split (PR #6423) is a candidate fix for the skewed-partition problem we hit. Vendor the algorithm without the full Lance migration as a first step.
• Python HF reference comparison — listen to the WAV manually, also diff our codec tokens against transformers.AutoModelForCausalLM output on the same prompt for token-exact wiring proof.

Test plan

• cargo build --release --example tts_rvq_e2e — clean
• cargo build --release --example tts_full_inference — clean
• Passthrough run: 478/478 cos = 1.0000, codec token match 225/225 = 100%
• Full inference run: 22-token prompt → 128 frames → 1.37s 24 kHz WAV; RMS / ZC / energy stats confirm real audio
• Human listen-check of the WAV (not done on this remote VM)
• Compare Rust codec tokens to HF Python reference (next session)
• Apply proposed BGZ-HHTL-D schema extension to close the storage-ratio gap (next session)

https://claude.ai/code/session_01NYGrxVopyszZYgLBxe4hgj

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6e8ee987b8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Use source dtype for passthrough byte accounting

The passthrough branch charges codebook_bytes as n_rows * n_cols * 2 for every tensor with n_rows > 8192, but this dispatch is shape-based and can include F32 tensors. In that case the script underreports storage by 2x, so the printed compressed-size and ratio metrics become incorrect and can mislead experiment conclusions about whether this path actually saves space.

Useful? React with 👍 / 👎.