feat(embed): Phase 1 — static token embedding engine

[ajianaz]

Phase 1: Static Token Embedding (Brain Mode Roadmap)

Dual-backend code embedding for Brain Mode.

What's included

File	Purpose
src/embed/tokens.rs	Hashing-trick backend (256-dim, zero-dep) + tokenizer with camelCase/snake_case splitting
src/embed/token_vocab.rs	Pre-trained nomic-embed-code backend (768-dim, real vectors)
src/embed/mod.rs	Module root, re-exports both backends
vendored/nomic/	30MB int8 blob (40,856 tokens × 768 dims) + vocab + Apache-2.0 license
scripts/extract_code_tokens.py	Reference extraction script (for re-generating vectors)
THIRD_PARTY.md	License attribution for nomic-embed-code

Architecture

tokenize_code(code) → HashMap<String, u32>
  ├─→ embed() → TokenEmbedding (256-dim, hashing trick, fast)
  └─→ embed_pretrained() → PretrainedEmbedding (768-dim, real vectors)

Tokenizer improvements

• camelCase splitting: calculateHash → [calculate, hash]
• snake_case splitting: hello_world → [hello, world]
• Acronym grouping: HTTPServer → [HTTP, Server]
• Multi-char operators: ->, ::, += as single tokens
• Stop punctuation filtering, number normalization

Trade-offs

• Bag-of-tokens = no positional context (acceptable for code search)
• 30MB binary size increase (vendored vectors)
• Hashing trick is noisy for cross-language comparison (expected)

Testing

• 36 embed-specific tests (all pass)
• 695 total tests (0 failures)
• Binary format verification, normalization checks, similarity quality

Roadmap context

Phase 0 (rename) ✅ → Phase 1 (this PR) → Phase 2 (tree-sitter + knowledge graph) → Phase 3 (brain mode commands) → Phase 4 (uteke integration) → Phase 5 (Voyage-4-Nano ONNX)