fix: threading crash, duplicate symbols, logging, and embedding insert (4 bugs)

[jimdawdy-hub]

Summary

Four bugs found while indexing openclaw/openclaw — a large TypeScript/Swift/Kotlin monorepo with 17,212 source files, 945 doc files, and 60+ extensions. All four bugs surface only at scale and were invisible in small test cases. A fifth bug (schema mismatch in get_index_stats) was found via MCP client validation.

---

Bug 1 — Cross-thread SQLite access crashes ~30% of file parses

Symptom: sqlite3.InterfaceError: bad parameter or other API misuse on roughly 30% of files during the parallel parse phase. Appeared as NoneType: None in logs due to Bug 4 below.

Root cause: _parse_file_for_indexing runs inside ThreadPoolExecutor workers but calls db.execute() on the shared main-thread connection. Python's sqlite3 binding is not safe for concurrent multi-thread access even with check_same_thread=False.

Fix: Pre-fetch all existing file records into a dict[path → mtime] in the main thread before launching the pool. Workers receive the dict and do a dict.get() lookup instead of a DB query. No DB access occurs in any worker thread.

---

Bug 2 — Duplicate symbols from tree-sitter AST crash DB write

Symptom: sqlite3.IntegrityError: UNIQUE constraint failed: symbols.file_id, symbols.name, symbols.kind, symbols.line_start in Phase 3 (DB write), after all parsing and GPU embedding had already completed.

Root cause: tree-sitter can produce multiple symbols with identical (name, kind, line_start) for a single file. The plain INSERT INTO symbols raised on the second occurrence and killed the entire run.

Fix: INSERT OR IGNORE INTO symbols. Use cursor.rowcount == 1 to detect a real insert. Important: cursor.lastrowid is not reliable here — after a no-op INSERT OR IGNORE it retains the rowid from the previous successful insert, not 0.

---

Bug 3 — Embedding insert crashes on sqlite-vec virtual table

Symptom: sqlite3.OperationalError: UNIQUE constraint failed on symbol_embeddings primary key immediately after the Bug 2 fix was applied.

Root cause: symbol_embeddings is a sqlite-vec virtual table. It rejects INSERT OR IGNORE at the SQL level (OperationalError, not IntegrityError). When Bug 2's fallback SELECT returned an existing symbol_id, the code still queued an embedding for it — and the existing row already had one.

Fix: Guard embedding_pairs.append() with if is_new — only queue embeddings for freshly inserted symbols. Existing symbols already have an embedding in the virtual table.

---

Bug 4 — logger.exception() reports all errors as NoneType: None

Symptom: Every parse failure logged as NoneType: None with no traceback, making Bug 1 completely invisible.

Root cause: Exceptions from worker threads are stored as return values. logger.exception("Failed to index %s", fpath) reads sys.exc_info() — the current thread's exception context — which is (None, None, None) since the exception happened in a different thread.

Fix: logger.error("Failed to index %s", fpath, exc_info=error) passes the stored exception explicitly.

---

Bug 5 — get_index_stats freshness fields fail MCP schema validation

Symptom: MCP clients that validate tool output against api_types.py receive a Pydantic ValidationError on every get_index_stats call. The tool appears to fail even though the server responds successfully.

Root cause: db.py builds the freshness dict with key last_file_indexed (a raw float Unix timestamp from MAX(last_modified)), but api_types.IndexFreshness declares last_code_indexed: str | None. Two mismatches: wrong field name and wrong type.

Fix: Rename key to last_code_indexed and convert the float via datetime.fromtimestamp(...).isoformat().

---

Test results on openclaw/openclaw

	Before	After
Files successfully parsed	~70% (~12k/17k)	100% (17,212/17,212)
DB write crash	Yes (every run)	No
Symbols indexed	—	111,000
Final DB size	—	750 MB
Total runtime (RTX 5060 Ti)	Never completed	~59 min
get_index_stats MCP validation	Fails (schema mismatch)	Passes

Verification

# extensions/telegram (337 files) — fast integration test
rm -f extensions/telegram/code_memory.db*
python -m code_memory.parser  # or via MCP index_codebase
# Before patch: 330/337 files, 7 skipped with NoneType: None
# After patch:  337/337 files, 0 errors

🤖 Generated with Claude Code