Release: WASM tree-sitter chunker, OOM/SQLITE fixes, GitHub token rotation, Projects dashboard views

README.md

@@ -78,7 +78,7 @@ Grep and fuzzy file search work fine for small projects. At scale they break dow
 │  └── embedded Swagger UI                                        │
 │                                                                 │
 │  Indexing pipeline                                              │
-│  ├── gotreesitter (AST chunking, 200+ languages)                │
+│  ├── tree-sitter/wasm (AST chunking, 31 langs)  (wazero)        │
 │  ├── llama-server sidecar (Unix socket → CodeRankEmbed Q8 GGUF) │
 │  ├── chromem-go (cosine similarity vector store)                │
 │  ├── SQLite FTS5 chunk mirror (BM25 — powers hybrid workspace)  │

doc/openapi.yaml

@@ -2164,6 +2164,52 @@ paths:
         "503":
           $ref: "#/components/responses/WorkspacesDisabled"
 
+  /api/v1/projects/{hash}/git-repo/token:
+    parameters:
+      - name: hash
+        in: path
+        required: true
+        schema:
+          type: string
+    put:
+      operationId: updateProjectGitRepoToken
+      tags: [projects]
+      summary: Re-point an external project at a different stored GitHub PAT
+      description: |
+        Changes which stored token (`git_repos.token_id`) an external project
+        uses to clone/fetch and to register webhooks, without deleting and
+        re-creating the project. Pass `token_id: null` (or omit) to detach the
+        token and clone the repo as public. 422 when the supplied token id does
+        not exist; 404 when the project is local (no git_repos row).
+
+        The webhook is left as-is: a hook already registered on GitHub keeps
+        delivering regardless of which PAT the server holds. The new token is
+        used by the next clone/fetch and by any later webhook re-register or
+        delete (re-save the sync method to re-register under the new token).
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: "#/components/schemas/UpdateGitRepoTokenRequest"
+      responses:
+        "200":
+          description: Updated git_repos row
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/GitRepo"
+        "401":
+          $ref: "#/components/responses/Unauthorized"
+        "403":
+          $ref: "#/components/responses/Forbidden"
+        "404":
+          $ref: "#/components/responses/NotFound"
+        "422":
+          $ref: "#/components/responses/Unprocessable"
+        "503":
+          $ref: "#/components/responses/WorkspacesDisabled"
+
   /api/v1/projects/{hash}/webhook-info:
     parameters:
       - name: hash
@@ -2471,6 +2517,46 @@ paths:
         required: true
         schema:
           type: string
+    put:
+      operationId: updateGithubToken
+      tags: [github-tokens]
+      summary: Rotate the secret of a stored GitHub PAT
+      description: |
+        Replaces the encrypted secret value of an existing token in place,
+        keeping the same id and name. The new plaintext is validated against
+        GitHub (GET /user) exactly like creation, and the stored scopes are
+        refreshed from the X-OAuth-Scopes response header. Because the id is
+        unchanged, every external project bound to this token keeps working —
+        no re-binding required. Use this when a PAT expires or is regenerated.
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: "#/components/schemas/UpdateGithubTokenRequest"
+      responses:
+        "200":
+          description: Token rotated (metadata only — plaintext is never echoed)
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/GithubToken"
+        "401":
+          $ref: "#/components/responses/Unauthorized"
+        "403":
+          $ref: "#/components/responses/Forbidden"
+        "404":
+          $ref: "#/components/responses/NotFound"
+        "422":
+          $ref: "#/components/responses/Unprocessable"
+        "502":
+          description: Could not reach GitHub to validate the token
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/Error"
+        "503":
+          $ref: "#/components/responses/WorkspacesDisabled"
     delete:
       operationId: deleteGithubToken
       tags: [github-tokens]
@@ -3245,6 +3331,8 @@ components:
         - max_embedding_concurrency
         - llama_batch_size
         - index_embed_batch_chunks
+        - chunk_max_concurrent
+        - llama_cache_ram_mib
         - source
       properties:
         embedding_model:
@@ -3270,6 +3358,14 @@ components:
           type: integer
           minimum: 0
           description: Cross-file embed-batch size for repo indexing (chunks per embed call). 0 = one call per file.
+        chunk_max_concurrent:
+          type: integer
+          minimum: 0
+          description: Chunker (tree-sitter wasm) instance-concurrency cap, decoupled from embedding concurrency. Each instance holds ~69 MiB. 0 = recommended (3).
+        llama_cache_ram_mib:
+          type: integer
+          minimum: -1
+          description: llama-server host prompt-cache cap in MiB (--cache-ram). 0 = disabled (recommended for embeddings — prompts are never reused, and llama's upstream 8 GiB default grows RSS until the container OOMs), -1 = unlimited.
         source:
           type: object
           additionalProperties:
@@ -3301,6 +3397,8 @@ components:
         - max_embedding_concurrency
         - llama_batch_size
         - index_embed_batch_chunks
+        - chunk_max_concurrent
+        - llama_cache_ram_mib
       properties:
         embedding_model: { type: string }
         llama_ctx_size: { type: integer }
@@ -3309,6 +3407,8 @@ components:
         max_embedding_concurrency: { type: integer }
         llama_batch_size: { type: integer }
         index_embed_batch_chunks: { type: integer }
+        chunk_max_concurrent: { type: integer }
+        llama_cache_ram_mib: { type: integer }
 
     RuntimeConfigUpdate:
       type: object
@@ -3339,6 +3439,13 @@ components:
         index_embed_batch_chunks:
           type: integer
           nullable: true
+        chunk_max_concurrent:
+          type: integer
+          nullable: true
+        llama_cache_ram_mib:
+          type: integer
+          nullable: true
+          description: MiB; 0 clears the override (falls back to env / recommended = disabled), -1 = unlimited.
 
     SidecarStatus:
       type: object
@@ -3809,6 +3916,20 @@ components:
           type: string
           nullable: true
           description: os.Hostname() of the indexing machine — display only.
+        full_sync_required:
+          type: boolean
+          description: |
+            True when this project's index is format-stale (e.g. the
+            chunker/embedding format changed under it) and needs a complete
+            rebuild. Informational only — it drives the dashboard "out of sync"
+            badge; the admin triggers the resync, which clears the flag on the
+            next successful full run. Absent/false means in sync.
+        full_sync_reason:
+          type: string
+          nullable: true
+          description: |
+            Human-readable explanation shown with the out-of-sync badge.
+            NULL when full_sync_required is false.
         sqlite_path:
           type: string
           nullable: true
@@ -4618,6 +4739,18 @@ components:
             consulted. Kept for backwards compatibility with older
             clients that still send it.
 
+    UpdateGithubTokenRequest:
+      type: object
+      required: [token]
+      properties:
+        token:
+          type: string
+          minLength: 1
+          description: |
+            The new plaintext PAT. Validated against GitHub and encrypted at
+            rest exactly like creation; the stored scopes are refreshed from
+            the X-OAuth-Scopes header. The token's id and name are unchanged.
+
     GitRepo:
       type: object
       required:
@@ -4744,6 +4877,17 @@ components:
             Human-readable note about the outcome — e.g. that webhook
             auto-registration failed and the server fell back to polling.
 
+    UpdateGitRepoTokenRequest:
+      type: object
+      properties:
+        token_id:
+          type: string
+          nullable: true
+          description: |
+            Id of the stored GitHub PAT this project should use. Null or
+            omitted detaches the token (token_id → NULL) and clones the repo
+            as public. 422 when the id does not match a stored token.
+
     GithubRepo:
       type: object
       required: [full_name, default_branch, private, html_url]

poc/wasm-treesitter/.gitignore

@@ -0,0 +1,3 @@
+# build artifact — rebuilt by build.sh; the committed module lives as
+# server/internal/chunker/tswasm/ts-core.wasm.br (brotli)
+ts-core.wasm

poc/wasm-treesitter/README.md

@@ -0,0 +1,70 @@
+# PoC: tree-sitter via WASM/wazero (pure-Go, no cgo)
+
+An alternative to the cgo backend on `feat/chunker-cgo-treesitter`. The **official**
+tree-sitter C runtime + the official TypeScript grammar are compiled to a single
+standalone `wasm32-wasi` reactor module (`build.sh`, via `zig cc`) and driven
+from Go through [wazero](https://github.com/tetratelabs/wazero) — **no cgo, no
+JavaScript, no third-party parser**. Only `wasmts.go` (our wazero host) is
+bespoke; the parser itself is the unmodified upstream C.
+
+Goal: give us real **speed + stability** numbers to choose between cgo and wasm.
+
+## Results — same 852-file vscode TypeScript corpus, full-tree walk
+
+| backend | wall | files/s | MB/s | ERROR trees | `editorOptions.ts` |
+|---|---|---|---|---|---|
+| gotreesitter (pure-Go GLR) | 13.83 s | 62 | 0.8 | **13** | 8.77 s → ERROR |
+| **WASM (wazero, pure-Go host)** | **~2.5 s** | **~330** | **~4.1** | **0** | **49 ms** |
+| cgo (native tree-sitter) | 1.26 s | 675 | 11.5 | 0 | 17 ms |
+
+- **WASM is ~2× slower than cgo, ~5× faster than gotreesitter, and correct** (0 ERROR trees vs gotreesitter's 13).
+- The WASM overhead is the **host↔guest call boundary**, not memory: each of the
+  2.68 M nodes costs ~3 wazero calls (`ts_node_type`, `ts_node_child_count`,
+  `ts_node_child`). Reusing node slots instead of `malloc`/`free` per node moved
+  the number only 328→357 files/s — so it's the calls. A single batched
+  "serialize subtree" export would close most of the remaining gap vs cgo
+  (future work; not done here).
+
+## Stability (`cmd/stability`)
+
+- tree-sitter is **robust**: 6 adversarial inputs (100–200 k-deep nesting, 5 MB
+  single token, invalid UTF-8, unbalanced templates) all parsed without crashing
+  — this is true of cgo too, so it is **not** a bug WASM uniquely fixes.
+- What WASM **adds** is containment: a guest-side fault (resource limit, and in
+  principle any C bug — stack overflow, OOB) surfaces as a **recoverable Go
+  error**; the host process stays alive. The memory-capped run demonstrates this.
+- Under cgo the equivalent fault is a native **SIGSEGV/abort that kills the whole
+  cix-server**. So crash-isolation is **insurance against unknown C bugs in
+  grammars/scanners**, not a fix for an observed crash.
+
+## Trade-off summary
+
+| | cgo (current) | WASM/wazero (this PoC) |
+|---|---|---|
+| Parse speed | 🟢 fastest | 🟡 ~2× slower (≈invisible end-to-end: embeddings dominate) |
+| Correctness | 🟢 official | 🟢 official (identical parser) |
+| Build | 🟡 needs C toolchain (musl-static solved it) | 🟢 `CGO_ENABLED=0`, trivial cross-compile; `zig` only at wasm-build time (one-off, artifact committed) |
+| Crash isolation | 🔴 C fault kills process | 🟢 contained → Go error |
+| Binary size | 🔴 ~78 MB (grammar tables linked natively) | 🟢 likely smaller: pure-Go host (~41 MB) + embedded `.wasm` (1.4 MB / grammar, brotli-compressible) |
+| Maturity / effort | 🟢 drop-in (official binding + 31 grammar modules) | 🔴 bespoke host; must build/bundle 31 grammar `.wasm` + flesh out node API + batched walk |
+
+## Honest read
+
+It's close. cgo is done and fastest. WASM costs ~2× on **parsing**, but since
+**embeddings dominate end-to-end indexing time**, that 2× is largely invisible in
+production — while WASM's upsides (no cgo, crash-isolation, smaller binary,
+toolchain-free server builds) are real. The price of WASM is **engineering
+effort** to productionize: build all 31 grammars into the module, write the full
+node-walk API the chunker needs (with a batched-walk export to recover speed),
+and wire it behind the same `tsgrammars`-style registry.
+
+## Build & run
+
+```bash
+brew install zig          # provides clang + wasi-libc cross-compile
+./build.sh                 # → ts-ts.wasm (official tree-sitter v0.25.10 + tree-sitter-typescript v0.23.2)
+go run ./cmd/bench  /path/to/vscode/src/vs/editor
+go run ./cmd/stability
+```
+
+`ts-ts.wasm` is committed so the benchmarks run without zig.