feat(chunker): WASM/wazero tree-sitter backend — replaces gotreesitter - #81
Merged
Conversation
Alternative to feat/chunker-cgo-treesitter: the official tree-sitter C runtime + TypeScript grammar compiled to a standalone wasm32-wasi reactor module (build.sh, via zig cc) and driven from Go through wazero — no cgo, no JS, no third-party parser. Only the wazero host (wasmts.go) is bespoke; the parser is unmodified upstream C. wasm_store.c is gated by TREE_SITTER_FEATURE_WASM (we don't define it), so the stock amalgamation compiles to wasi with no stubs. Measured on the same 852-file vscode TypeScript corpus (full-tree walk): backend wall files/s ERROR trees editorOptions.ts gotreesitter (pure-Go) 13.83s 62 13 8.77s -> ERROR WASM (wazero, pure-Go) ~2.5s ~330 0 49ms cgo (native) 1.26s 675 0 17ms - WASM ~2x slower than cgo, ~5x faster than gotreesitter, correct (0 errors). - Overhead is the per-node host<->guest call boundary (~3 calls/node x 2.68M nodes), not memory — slot-pooling barely moved it. A batched "serialize subtree" export would close most of the gap (future work). - Stability: tree-sitter is robust on adversarial input under both backends; WASM additionally CONTAINS faults (resource/guest trap -> recoverable Go error, host alive) where cgo would SIGSEGV the whole process. Insurance vs unknown C bugs, not a fix for an observed crash. Trade-off vs cgo: ~2x parse cost (largely invisible end-to-end since embeddings dominate) in exchange for CGO_ENABLED=0 builds, crash-isolation, and a likely smaller binary; cost is the engineering effort to build/bundle all 31 grammars and flesh out the node API. README.md has the full comparison. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d skip, doc-comment attachment Replace gotreesitter with the official tree-sitter C runtime + 31 grammars compiled to one wasm32-wasi module (ts-core.wasm.br, brotli ~3MB) driven via wazero. No cgo: traps are contained (parse falls back to sliding window, the process survives), and the binary stays CGO_ENABLED=0. Memory design (measured on the prod-shaped churn workload): - linear memory is mmap-backed (experimental.WithMemoryAllocator) instead of wazero's default Go-heap append-grow: no realloc-copy garbage on growth and munmap-on-close returns recycled instances' memory to the OS immediately. Churn heapSys 1135→391MB, peak RSS 1070→535MB; full-repo chunking peak RSS 1516→787MB. - engine pool: hard concurrency cap (dashboard-tunable), 256MiB per-instance linear-memory ceiling (2× headroom over the worst measured instance at the indexer's 512KiB file cap), high-water-mark recycling, 1 idle instance. Chunker quality fixes: - minified/bundled js/ts/css (.min., .bundle.js, >2KiB lines) skip the parser straight to sliding window — the pathological input class that ballooned instances for near-zero semantic value. - a declaration's doc comment now attaches to its chunk (language-agnostic via tree-sitter's extra flag + same-row wrapper climb; verified for Go, TS, C, Python, Rust, Java). Generated files stop spraying comment-only micro chunks: openapi.gen.go 893→517 chunks, median 114→256B, symbols/refs byte-identical. Memory-stress harnesses are committed but gated behind CIX_MEMSTRESS=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…am (OOM fix) Two new runtime-config fields, end to end (DB migrations 16/17 → runtimecfg → admin API → openapi → dashboard): - chunk_max_concurrent — the wasm chunker's instance-concurrency cap, decoupled from embedding concurrency; resizes the live limiter without a restart. Env: CIX_CHUNK_MAX_CONCURRENT; per-instance memory knobs stay env-only (CIX_CHUNK_MEM_LIMIT_PAGES, CIX_CHUNK_RECYCLE_GROWTH_MB, CIX_CHUNK_MAX_IDLE). - llama_cache_ram_mib — llama-server's HOST prompt cache cap (--cache-ram). Upstream defaults this to 8 GiB (ggml-org/llama.cpp#16391), which is pure waste for an embeddings-only sidecar: prompts are never reused, but the cache fills anyway. Observed on prod: llama-server RSS 365MB→11.3GB within minutes of indexing vscode@main, then cgroup OOM kill — twice at the 10G limit, again at 16G. With --cache-ram 0 (our default; -1 = unlimited) it plateaus at ~900MB under the same load. Env: CIX_LLAMA_CACHE_RAM; shown in the dashboard's Runtime parameters card, applied via Save & Restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vation A full-reindex wipe ran as ONE transaction: DELETE of all refs/symbols/ file_hashes plus the trigram-FTS rows. On a vscode-sized project (~445k refs, tens of thousands of FTS rows — each FTS delete re-tokenizes its content) that held SQLite's single writer for minutes, starving every concurrent writer past busy_timeout. Prod symptom: the jobs worker logged `claim failed: SQLITE_BUSY` on every 5s poll tick for the whole wipe. - BeginIndexing full wipe: file_hashes first (its own statement — once gone, every file looks dirty, so a crash mid-wipe just resumes on the next run), then symbols/refs in 20k-row batches, then chunks_fts/chunks_meta via the batched chunksfts.DeleteByProject (500 rows per tx — FTS deletes are the expensive ones). The writer is released between batches. - projects.Delete: same batched FTS wipe, project row deleted last so a failed wipe is resumable. - jobs worker: SQLITE_BUSY on claim is expected contention, not a fault — log the streak start as WARN with a once-a-minute heartbeat instead of an ERROR per tick, and log when it clears. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dvcdsys
changed the title
dvcdsys
marked this pull request as ready for review
June 22, 2026 10:03
… after chunker swap The WASM tree-sitter chunker emits different chunks/symbols/signatures (hence different embedding vectors) than gotreesitter, so an index built by the old chunker is format-stale. Nothing in the reindex logic notices a *chunker* change (incremental is content-hash gated; full-wipe only triggers on an embedding-model change), so existing projects would silently serve a stale mix. Add a persistent, self-clearing flag so the dashboard can surface this and an admin can act on it: - db: migration 18 adds projects.full_sync_required + full_sync_reason (idempotent via columnExists) and backfills every existing project to require a full resync. Fresh installs default to 0 (in sync). - indexer: BeginIndexing records session.full; FinishIndexing clears the flag when the completed run was full — on success only, so a full run that crashes mid-way stays flagged. Incremental/reconcile runs leave it set. Both the git/clone and local CLI paths converge here, so the clear is universal. - The flag is INFORMATIONAL: it drives the badge but triggers nothing. The admin starts the resync (dashboard force-full for git projects, `cix reindex` for local). - projects/httpapi: carry the two fields through the loader + project payload. - openapi: spec fields + regenerated Go types (dashboard TS types regenerate at build via gen:api). - dashboard: "Out of sync" badge (ProjectCard) + "full resync required" alert (ProjectDetailPage), mirroring the existing stale-model badge. - tests: migration backfill, full-clears/incremental-keeps, loader round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t page
A failed reindex/sync was invisible: POST /projects/{hash}/reindex returns 200
("enqueued", success toast), but the clone/index job then fails asynchronously
in a background worker (e.g. "clone: fetch: authentication required" for a
private repo with a bad/expired GitHub token). recordFailure already persists
the reason to git_repos.last_error and sets projects.status='error', but the
dashboard only showed a grey "error" badge — no reason, no notification.
- ProjectDetailPage: prominent destructive Alert when status='error', showing
git_repos.last_error (external) verbatim, or a generic hint + `cix reindex`
for local projects. Mirrors the existing drift / full_sync alerts.
- Toast on the live not-error → error transition (a watched job just failed);
prevStatus starts undefined so landing on an already-errored project shows
only the inline alert, no spurious toast.
- hooks: useProject now polls while 'indexing' OR 'error' (was indexing-only)
so a retry's outcome appears live and the reason stays current; bounded to
the open/focused page by react-query. useProjectGitRepo gains an opt-in
`poll` arg, driven while a job is in flight / errored, so last_error refreshes
without a manual reload.
Frontend-only — the error was already persisted and exposed via
GET /projects/{hash}/git-repo; this just routes it to the operator. Enqueue
semantics unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replace the pure-Go
gotreesitterchunker backend with the official tree-sitterC runtime + 31 grammars, compiled to a single
wasm32-wasimodule(
ts-core.wasm.br, brotli ~3 MB) and driven from Go via wazero. No cgo — thebinary stays
CGO_ENABLED=0and the CPU image staysdistroless/static.Also bundles the supporting server/indexer fixes made on this branch:
llama --cache-ram(host prompt-cache OOM fix)SQLITE_BUSYwriter-starvation stormWhy
The bugs that drove this migration are still open upstream (re-verified 2026-06-22,
odvcencio/gotreesitter), so reverting to gotreesitter is not viable:Parser.Parsefatal, unrecoverable stack overflow on large table-driven Go files;
recover()can't catch it, it takes down the whole process.Go file: 330 s parse, 7.7 GB heap,
SetTimeoutMicrosbypassed in post-parsenormalization (the 100 GB-OOM-on-big-repos class). The chore(deps): bump actions/upload-artifact from 4 to 7 #113 iterative-normalize
patch only downgrades chore(deps): bump aquasecurity/trivy-action from 0.35.0 to 0.36.0 #110's crash into this stall — it does not fix the OOM.
{a}b=c→ root
ERROR, breaks most minified bundles.Latest upstream release is v0.20.2 (2026-06-06); post-release commits are only
swift/csharp/go recovery fixes — none touch the three issues above.
WASM-via-wazero gives us: correctness (0 ERROR trees on the corpora that broke
gotreesitter), crash-isolation (a guest trap → recoverable Go error, host
stays alive, where cgo would SIGSEGV the whole server — unacceptable with ~85
projects on one box),
CGO_ENABLED=0builds, and a bounded memory profile.This supersedes the cgo backend (PR #80), which is abandoned.
How
tswasm/build.sh(official tree-sitteramalgamation,
wasm32-wasi, noTREE_SITTER_FEATURE_WASMso no stubs). Driventhrough wazero with a batched
ts_dump_treeexport (flatNodeRec[], oneMemory.Read) — no per-node host↔guest boundary calls.(
experimental.WithMemoryAllocator) → no realloc-copy garbage on growth,munmap-on-close returns memory to the OS immediately (churn heapSys1135→391 MB, peak RSS 1070→535 MB; full-repo peak RSS 1516→787 MB). Engine pool
with a hard, dashboard-tunable concurrency cap, 256 MiB per-instance ceiling,
high-water-mark recycling, 1 idle instance.
.min.,.bundle.js,CIX_MEMSTRESS=1.go build ./...clean. Per-language fixture tests + the official tree-sitter CLIoracle are the gate (the cgo branch is not differentiated against).
Type of change
Checklist
go build ./...passesdeveloppost-mergeCommits:
7384cb1poc(chunker): WASM/wazero tree-sitter backend — speed + stability vs cgoddd16e9feat(chunker): wasm/wazero tree-sitter backend — mmap memory, minified skip, doc-comment attachment285713dfeat(server): dashboard-tunable chunker concurrency + llama --cache-ram (OOM fix)675ddc7fix(indexer): batch big-project wipes to stop SQLITE_BUSY writer starvationFollow-on in this PR:
full_sync_requiredflag (migration 18)Because the new chunker emits different chunks/symbols/signatures (hence different
vectors) than gotreesitter, an existing index is format-stale but nothing in
the reindex logic notices a chunker change (incremental is content-hash gated;
full-wipe only triggers on an embedding-model change). So existing projects would
silently serve a stale mix after deploy.
Added a persistent, self-clearing, reason-carrying flag:
projects.full_sync_required+full_sync_reason(idempotent via
columnExists) and backfills every existing project torequire a full resync. Fresh installs default to
0(in sync).resync required" badge (mirrors the stale-model badge); the admin starts the
resync (dashboard force-full for git projects,
cix reindex <path>for local).BeginIndexingrecordssession.full;FinishIndexingclears the flag when the completed run was full — on success only (a full
run that crashes mid-way stays flagged). Incremental/reconcile runs leave it
set. Both the git/clone and local CLI paths converge on
FinishIndexing, sothe clear is universal regardless of where the resync was triggered.
build). Tests: migration backfill, full-clears/incremental-keeps, loader
round-trip.
🤖 Generated with Claude Code