feat(indexing): port CspIndex orchestrator (fromPath/fromGit/search/findRelated/save/load)

[amondnet]

Summary

Port src/semble/index/{create,index,types}.py to TypeScript:

• src/indexing/types.ts — PersistencePath class with chunks, bm25Index, semanticIndex, metadata fields, nonExisting() instance method, and static fromPath(base) factory matching semble's layout (chunks.json, bm25_index/, semantic_index/, metadata.json).
• src/indexing/create.ts — createIndexFromPath(path, options) walks files, enforces MAX_FILE_BYTES = 1_000_000, chunks each file, builds the BM25 index over tokenize(enrichForBm25(chunk)), and embeds chunks into a SelectableBasicBackend. Throws when no chunks are produced.
• src/indexing/index.ts — CspIndex class with:
  • static fromPath(path | URL, options) → resolves directory, throws if missing or not a dir
  • static fromGit(url, options) → clones via spawn('git', ['clone', '--depth', '1', ...]) with GIT_CLONE_TIMEOUT_MS (from CSP_CLONE_TIMEOUT, default 60s), manual SIGTERM on timeout, tmpdir cleanup in finally
  • static loadFromDisk(path) → reads metadata.json + chunks.json and reconstructs from disk
  • search(query, options) and findRelated(source, options) → delegated to ../search.ts, with rerank defaulting to ContentType.Code in this._content
  • save(path) → mkdir -p + writes all 4 artifacts
  • Exports DEFAULT_CONTENT, ALL_CONTENT, and GIT_CLONE_TIMEOUT_MS to match the spec
• Inline minimal stubs for sibling modules (../types.ts, ../tokens.ts, ../search.ts, ../stats.ts, ../chunking/chunk-source.ts, ./dense.ts, ./sparse.ts, ./files.ts, ./file-walker.ts) marked // TODO(integration): import from … so the unit compiles and tests pass before the dependent units land. (Stubs are not committed in this PR.)

Tests

bun test src/indexing/types.test.ts src/indexing/create.test.ts src/indexing/index.test.ts — 20/20 pass.

• PersistencePath: layout, nonExisting() empty/partial/full cases
• createIndexFromPath: chunk/bm25/semantic build, throws when empty, extension override, MAX_FILE_BYTES skip, subdir descent
• CspIndex.stats: empty + populated language distribution
• CspIndex.search('') / empty index → []
• CspIndex.findRelated: excludes the source chunk, accepts Chunk or SearchResult
• save → loadFromDisk roundtrip preserves chunks + language stats
• fromPath throws on non-existent path / file (not directory)

Code-review fixes applied during review

1. fromGit used execFile with a non-existent stdio option. child_process.execFile does not accept stdio (that's spawn-only), so the stdin-redirection request was silently dropped. Replaced with a dedicated runGitClone helper using spawn('git', args, { stdio: ['ignore', 'pipe', 'pipe'] }) and a manual setTimeout + kill('SIGTERM') so the child is reliably torn down on timeout — matching semble's subprocess.run(..., stdin=subprocess.DEVNULL, timeout=…) semantics.
2. _computeFileSizes used path.join instead of path.resolve. join('/root', '/abs/path') returns /root/abs/path, but semble's root / chunk.file_path follows pathlib rules where an absolute right-hand operand wins. Switched to resolve(root, chunk.filePath) so future absolute-path scenarios behave identically to the upstream.

Notes

• Public-API field names are camelCase (filePath, startLine, endLine) per CLAUDE.md.
• Stats file path is ~/.csp/savings.jsonl (not ~/.semble/).
• TypeScript strict + noUncheckedIndexedAccess + exactOptionalPropertyTypes honored — optional spread guards on extensions / selector.
• e2e (real model / real walker) intentionally out of scope per unit spec.

---

Summary by cubic

Ported the CspIndex indexing orchestrator to TypeScript for local and git-based indexing with hybrid (BM25 + semantic) search, plus a save/load disk format. Adds tests, enforces a 1 MB per-file cap, and improves clone reliability, filter handling, and path/size accuracy.

• New Features

  • CspIndex with fromPath, fromGit, search, findRelated, save, and loadFromDisk; exports DEFAULT_CONTENT, ALL_CONTENT, and GIT_CLONE_TIMEOUT_MS.
  • createIndexFromPath builds BM25 over tokenize(enrichForBm25(chunk)) and a semantic index from embeddings; walks subdirs, skips files > 1 MB, supports extensions and displayRoot.
  • PersistencePath standardizes on-disk layout: chunks.json, bm25_index/, semantic_index/, metadata.json.
  • fromGit clones with spawn('git', ...) and a timeout via CSP_CLONE_TIMEOUT → GIT_CLONE_TIMEOUT_MS; cleans up tmpdir.

• Bug Fixes

  • search: return [] for zero-match filters and when topK <= 0; no fallback to unfiltered results.
  • fromGit: ignore stdout to prevent pipe deadlocks; tolerate rm() failures during cleanup; keep stdin closed and timeout handling.
  • File sizes: use statSync().size for accurate byte counts and to avoid reading file contents.
  • Path handling: preserve filesystem-root trailing separators in resolveDirectory; fix absolute-path resolution via path.resolve for file size computation.

^{Written for commit fde496f. Summary will update on new commits.}

[gemini-code-assist]

Code Review

This pull request ports the indexing functionality from Python, introducing the CspIndex class for hybrid search, directory and Git repository indexing, and persistence helpers, along with comprehensive tests. The reviewer feedback highlights several critical improvements: optimizing file size calculation in _computeFileSizes by using statSync instead of reading entire files into memory, fixing a filtering bug in _getSelectorVector where empty filter results incorrectly disable filtering, avoiding potential process hangs in runGitClone by ignoring stdout instead of piping it without consumption, and cleaning up unused imports and helper functions.

[cubic-dev-ai]

4 issues found across 6 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[amondnet]

Applied all 9 bot-review findings (5 from gemini-code-assist, 4 from cubic). Pushed in fde496f.

Summary:

• _computeFileSizes: statSync().size instead of readFileSync().length — no in-memory file content, correct UTF-8 byte counts.
• _getSelectorVector: distinguishes 'no filter' (returns null) from 'filter matches nothing' (returns []). search() early-returns [] for the empty-selector case so an unmatched filter never silently falls back to an unfiltered search. Intentional divergence from semble's Python, which has the same latent bug.
• resolveDirectory: trailing-separator stripping now skips filesystem-root paths (/ on POSIX, C:\\ on Windows) so we don't mutate C:\\ → C:.
• runGitClone: stdio: ['ignore', 'ignore', 'pipe'] — stdout no longer piped, preventing OS pipe-buffer deadlock when verbose git hooks emit output. Kept spawn + manual timeout as per the original code-review decision (NOT reverted to execFile).
• fromGit: rm errors during finally are swallowed so they never mask the inner error.
• Cleanup: removed unused readFileSync import and readFileSyncSafe helper.
• Tests: added regressions for filters-with-zero-matches → [] and topK<=0 → [].

Applied 9, deferred 0.