Conversation
Alternative to feat/chunker-cgo-treesitter: the official tree-sitter C runtime + TypeScript grammar compiled to a standalone wasm32-wasi reactor module (build.sh, via zig cc) and driven from Go through wazero — no cgo, no JS, no third-party parser. Only the wazero host (wasmts.go) is bespoke; the parser is unmodified upstream C. wasm_store.c is gated by TREE_SITTER_FEATURE_WASM (we don't define it), so the stock amalgamation compiles to wasi with no stubs. Measured on the same 852-file vscode TypeScript corpus (full-tree walk): backend wall files/s ERROR trees editorOptions.ts gotreesitter (pure-Go) 13.83s 62 13 8.77s -> ERROR WASM (wazero, pure-Go) ~2.5s ~330 0 49ms cgo (native) 1.26s 675 0 17ms - WASM ~2x slower than cgo, ~5x faster than gotreesitter, correct (0 errors). - Overhead is the per-node host<->guest call boundary (~3 calls/node x 2.68M nodes), not memory — slot-pooling barely moved it. A batched "serialize subtree" export would close most of the gap (future work). - Stability: tree-sitter is robust on adversarial input under both backends; WASM additionally CONTAINS faults (resource/guest trap -> recoverable Go error, host alive) where cgo would SIGSEGV the whole process. Insurance vs unknown C bugs, not a fix for an observed crash. Trade-off vs cgo: ~2x parse cost (largely invisible end-to-end since embeddings dominate) in exchange for CGO_ENABLED=0 builds, crash-isolation, and a likely smaller binary; cost is the engineering effort to build/bundle all 31 grammars and flesh out the node API. README.md has the full comparison. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…d skip, doc-comment attachment Replace gotreesitter with the official tree-sitter C runtime + 31 grammars compiled to one wasm32-wasi module (ts-core.wasm.br, brotli ~3MB) driven via wazero. No cgo: traps are contained (parse falls back to sliding window, the process survives), and the binary stays CGO_ENABLED=0. Memory design (measured on the prod-shaped churn workload): - linear memory is mmap-backed (experimental.WithMemoryAllocator) instead of wazero's default Go-heap append-grow: no realloc-copy garbage on growth and munmap-on-close returns recycled instances' memory to the OS immediately. Churn heapSys 1135→391MB, peak RSS 1070→535MB; full-repo chunking peak RSS 1516→787MB. - engine pool: hard concurrency cap (dashboard-tunable), 256MiB per-instance linear-memory ceiling (2× headroom over the worst measured instance at the indexer's 512KiB file cap), high-water-mark recycling, 1 idle instance. Chunker quality fixes: - minified/bundled js/ts/css (.min., .bundle.js, >2KiB lines) skip the parser straight to sliding window — the pathological input class that ballooned instances for near-zero semantic value. - a declaration's doc comment now attaches to its chunk (language-agnostic via tree-sitter's extra flag + same-row wrapper climb; verified for Go, TS, C, Python, Rust, Java). Generated files stop spraying comment-only micro chunks: openapi.gen.go 893→517 chunks, median 114→256B, symbols/refs byte-identical. Memory-stress harnesses are committed but gated behind CIX_MEMSTRESS=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…am (OOM fix) Two new runtime-config fields, end to end (DB migrations 16/17 → runtimecfg → admin API → openapi → dashboard): - chunk_max_concurrent — the wasm chunker's instance-concurrency cap, decoupled from embedding concurrency; resizes the live limiter without a restart. Env: CIX_CHUNK_MAX_CONCURRENT; per-instance memory knobs stay env-only (CIX_CHUNK_MEM_LIMIT_PAGES, CIX_CHUNK_RECYCLE_GROWTH_MB, CIX_CHUNK_MAX_IDLE). - llama_cache_ram_mib — llama-server's HOST prompt cache cap (--cache-ram). Upstream defaults this to 8 GiB (ggml-org/llama.cpp#16391), which is pure waste for an embeddings-only sidecar: prompts are never reused, but the cache fills anyway. Observed on prod: llama-server RSS 365MB→11.3GB within minutes of indexing vscode@main, then cgroup OOM kill — twice at the 10G limit, again at 16G. With --cache-ram 0 (our default; -1 = unlimited) it plateaus at ~900MB under the same load. Env: CIX_LLAMA_CACHE_RAM; shown in the dashboard's Runtime parameters card, applied via Save & Restart. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vation A full-reindex wipe ran as ONE transaction: DELETE of all refs/symbols/ file_hashes plus the trigram-FTS rows. On a vscode-sized project (~445k refs, tens of thousands of FTS rows — each FTS delete re-tokenizes its content) that held SQLite's single writer for minutes, starving every concurrent writer past busy_timeout. Prod symptom: the jobs worker logged `claim failed: SQLITE_BUSY` on every 5s poll tick for the whole wipe. - BeginIndexing full wipe: file_hashes first (its own statement — once gone, every file looks dirty, so a crash mid-wipe just resumes on the next run), then symbols/refs in 20k-row batches, then chunks_fts/chunks_meta via the batched chunksfts.DeleteByProject (500 rows per tx — FTS deletes are the expensive ones). The writer is released between batches. - projects.Delete: same batched FTS wipe, project row deleted last so a failed wipe is resumable. - jobs worker: SQLITE_BUSY on claim is expected contention, not a fault — log the streak start as WARN with a once-a-minute heartbeat instead of an ERROR per tick, and log when it clears. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… after chunker swap The WASM tree-sitter chunker emits different chunks/symbols/signatures (hence different embedding vectors) than gotreesitter, so an index built by the old chunker is format-stale. Nothing in the reindex logic notices a *chunker* change (incremental is content-hash gated; full-wipe only triggers on an embedding-model change), so existing projects would silently serve a stale mix. Add a persistent, self-clearing flag so the dashboard can surface this and an admin can act on it: - db: migration 18 adds projects.full_sync_required + full_sync_reason (idempotent via columnExists) and backfills every existing project to require a full resync. Fresh installs default to 0 (in sync). - indexer: BeginIndexing records session.full; FinishIndexing clears the flag when the completed run was full — on success only, so a full run that crashes mid-way stays flagged. Incremental/reconcile runs leave it set. Both the git/clone and local CLI paths converge here, so the clear is universal. - The flag is INFORMATIONAL: it drives the badge but triggers nothing. The admin starts the resync (dashboard force-full for git projects, `cix reindex` for local). - projects/httpapi: carry the two fields through the loader + project payload. - openapi: spec fields + regenerated Go types (dashboard TS types regenerate at build via gen:api). - dashboard: "Out of sync" badge (ProjectCard) + "full resync required" alert (ProjectDetailPage), mirroring the existing stale-model badge. - tests: migration backfill, full-clears/incremental-keeps, loader round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…t page
A failed reindex/sync was invisible: POST /projects/{hash}/reindex returns 200
("enqueued", success toast), but the clone/index job then fails asynchronously
in a background worker (e.g. "clone: fetch: authentication required" for a
private repo with a bad/expired GitHub token). recordFailure already persists
the reason to git_repos.last_error and sets projects.status='error', but the
dashboard only showed a grey "error" badge — no reason, no notification.
- ProjectDetailPage: prominent destructive Alert when status='error', showing
git_repos.last_error (external) verbatim, or a generic hint + `cix reindex`
for local projects. Mirrors the existing drift / full_sync alerts.
- Toast on the live not-error → error transition (a watched job just failed);
prevStatus starts undefined so landing on an already-errored project shows
only the inline alert, no spurious toast.
- hooks: useProject now polls while 'indexing' OR 'error' (was indexing-only)
so a retry's outcome appears live and the reason stays current; bounded to
the open/focused page by react-query. useProjectGitRepo gains an opt-in
`poll` arg, driven while a job is in flight / errored, so last_error refreshes
without a manual reload.
Frontend-only — the error was already persisted and exposed via
GET /projects/{hash}/git-repo; this just routes it to the operator. Enqueue
semantics unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat(chunker): WASM/wazero tree-sitter backend — replaces gotreesitter
…rnal projects at a different token
An expired GitHub PAT silently broke every external project bound to it:
tokens were immutable (delete + recreate minted a new id), and a project's
token_id was set once at creation with no way to change it.
Two admin-only capabilities, both flowing through the OpenAPI spec:
1. Rotate a key in place — PUT /api/v1/github-tokens/{id} replaces the secret,
re-validates against GitHub (GET /user) like create, and refreshes the
stored scopes. id + name are unchanged, so linked projects keep working
with no re-binding. UI: rotate dialog per token row in TokensTab.
2. Re-point an existing external project — PUT /api/v1/projects/{hash}/git-repo/token
changes git_repos.token_id (null detaches → public). Validates the token id
exists (422 otherwise); webhook left intact. UI: token dropdown in the
project's SyncSettingsCard.
Adds githubtokens.Update() + gitrepos.SetTokenID() service methods, regenerated
openapi.gen.go, and tests: service round-trip, HTTP rotate (refresh scopes /
invalid / not-found), project token attach-detach-422-404, and admin-gating 403.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PUT /api/v1/github-tokens/{id} and PUT /api/v1/projects/{hash}/git-repo/token
gate on mustBeAdmin, which returns 403 for an authenticated non-admin — but the
spec only listed 401. Add the shared Forbidden (403) response to both, between
401 and 404. openapi.gen.go regenerated via make openapi-gen (only the embedded
spec blob changes; no types/interface/route changes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…oject-token-select feat: rotate GitHub tokens in place + re-point external projects at a different token
Add a tiles/table view toggle to the Projects page, plus name search and external/local type filters that apply to both views, and sortable Name / Last indexed / Added columns in the table view. All client-side over the existing /api/v1/projects response — no backend changes. - New lib/projectList.ts: shared isExternal / label / filter / sort helpers (never-indexed rows always sort last; query-cache array is copied, never mutated in place) - New lib/viewPreference.ts: persist the view choice in localStorage, mirroring the editorPreference.ts pattern (default 'grid' = no change) - New ProjectsTable: shadcn Table with chevron-sortable headers, type + status/drift badges, and a per-row Sync button for external repos - ProjectCard now reuses the shared basename/STATUS_VARIANT/isExternal helpers so cards and table can't drift apart (no visual change) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- ProjectsTable: make rows keyboard-accessible — the name is now a real
<Link> (focusable, Cmd/Ctrl-click → new tab); the whole-row onClick
bails on defaultPrevented and modified/non-left clicks so it neither
double-navigates nor hijacks open-in-new-tab.
- sortProjects: name column sorts case-insensitively (localeCompare
sensitivity:'base'); extract compareTimes() so created and
last_indexed sink missing timestamps to the bottom identically.
- isExternal: widen to a structural { host_path } param and route the
two remaining inline host_path.startsWith('github.com/') copies
(ProjectDetailPage, WorkspaceProjectRow) through it — now the single
source of truth for the whole dashboard.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat(dashboard): grid/table views, sorting & filtering for Projects
…lters, Reindex action, adaptive width Round out the Projects list/table view ahead of release: - Sortable columns: Name, Type, Files, Symbols, Last indexed, Added (Status/Languages are filtered, not sorted, since each project carries a set of values there rather than one). - Toolbar filters (client-side, both views): Type, Languages, and an inclusive multi-select Status dropdown built over the actual badge set shown in the column (lifecycle status + "Stale model" + "Out of sync"). Options are derived from the current data, and a project matches only when its status set contains every ticked label. - Reindex button alongside Sync in each row and card (external-only, behind the existing confirm dialog). - Content width scales with the viewport (max-w-5xl → 6xl → screen-2xl) so the wide table is no longer boxed into the middle on large monitors. - Centralize the model-drift predicate (isDrifted) so the column badges and the status filter can't disagree. - Add ui/dropdown-menu.tsx (Radix, already a dependency). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL flagged the int (strconv.Atoi, 64-bit on amd64) → uint32 cast at main.go without an upper-bound check: an absurd env value would silently truncate to a wrong page limit. Guard with `v <= math.MaxUint32`. Sibling knobs are unaffected (RecycleGrowthBytes targets uint64, MaxIdleInstances is int). Clears the only PR-introduced code-scanning alert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release PR rolling
develop→main. 14 commits sinceserver/v0.10.0.Highlights
Chunker: WASM/wazero tree-sitter backend
full_sync_requiredflag surfaces format-stale projects after the chunker swap; the dashboard shows an "Out of sync" badge and the admin triggers the resync.Stability / memory
--cache-ram(fixes the in-container OOM kills; host prompt cache defaulted to 8 GiB for zero-reuse embeddings).GitHub tokens
Dashboard — Projects
Commits
$(git log --pretty='- %s' origin/main..develop)
Release steps (T9, after merge)
cd server && make scout-cuda→ verify 0 HIGH/CRITICALserver/cmd/cix-server/version.go→ next is 0.11.0 (last tagserver/v0.10.0)git tag server/v0.11.0 && git push origin server/v0.11.0release-server.ymlbuilds + pushes:0.11.0,:latest,:0.11.0-cu128,:cu128Verification
Built from
developand deployed to the dev server (RTX 3090) via Portainer; container healthy, dashboard Projects views smoke-tested.🤖 Generated with Claude Code