fix(recall): resolve 7 knowledge retrieval bugs (#79-#85)

[m0Nst3r873]

Summary

Fixes 7 recall/knowledge-graph bugs reported in #79-#85, plus reviewer cleanup.

#79 — --depth parameter now effective

depth	Budget	Behavior
route	1500 tokens	Title + path only
context	5000 (default)	~300-char keyword snippet
lookup	20000	Full content per result

Snippet field added to formatResults output — no truncation applied (length controlled by depth budget in queryCodeKnowledge).

#80 — codebase score normalization

Replace min-max (top result always = 10) with log-dampening:
Math.min(10, Math.log2(score + 1) * 2) — preserves relative ordering without pinning.

#81 — hasWiki path

Use resolved wikiRoot (from team-repo config) instead of hardcoded cwd/teamwiki.

#82 — buildConfidence semantics

Change from arithmetic mean (more evidence → lower score) to max(weights) — strongest evidence determines confidence. This is a deliberate design change: multiple supporting factors should not dilute confidence.

#83 — search-index shrink protection

Relax from "skip if new < 50% of old" to "skip only if new is empty". Allows legitimate file deletions to be reflected in the index.

#84 — codebase graph retrieval

• 84-1: loadWikiPages now recurses into subdirectories via collectMdFiles
• 84-2/4: Unified tokenizer (src/utils/tokenizer.ts) shared by search-index and code-knowledge-recall. Adds camelCase splitting to search-index. Truncates input to 100K chars to prevent Intl.Segmenter OOM.
• OOM fix: PageDoc.tokens replaced with uniqueTokens: Set<string> built via the shared tokenize function (same tokenizer for index-time and query-time)

#85 — auto-recall upvote scope

Use autoDetectInit() instead of requireInit() so votes are written to the correct scope (project or user).

Additional cleanup

• SEARCH_INDEX_VERSION bumped 4→5 (triggers rebuild for camelCase tokenizer change)
• README --dir entry added (EN + CN)
• callClaude JSDoc timeout corrected to 600000
• Redundant dynamic import of autoPushTeamRepo removed

Review feedback addressed

Issue	Fix
snippet truncated to 500 chars contradicts lookup's 20K budget	Removed formatResults truncation — snippet length now fully controlled by depth budget
index-time (fastTokens regex) vs query-time (tokenize Segmenter) produce different tokens → skewed BM25	Deleted fastTokens; index-time now uses the same shared tokenize as query-time
rawTokenCount counts full doc while token Set was capped at 5000 → inconsistent dl	rawTokenCount now truncates to 100K chars (same limit as tokenize)

E2E verification

Test	Result
recall (default heap, no --max-old-space-size)	✅ No OOM
recall --depth route	✅ Title + path only
recall --depth context	✅ Keyword snippet
recall --depth lookup	✅ Full content
import --from-org (11 repos, --skip-enrich)	✅ 10s
graph integrity (1179 nodes, 794 edges)	✅
codebase --lint	✅

Test plan

• npx tsc --noEmit — zero errors
• npx vitest run — 1676 passed (130 files)
• E2E: recall no OOM under default heap
• E2E: --depth route/context/lookup produce different Snippet output
• E2E: import + graph aggregation unaffected
• Index-time and query-time use identical tokenizer (no df/idf skew)

Closes #79, #80, #81, #82, #83, #84, #85

[jeff-r2026]

Automated review pass (medium effort). Three items — one functional contradiction with #79, plus two ranking-consistency issues in the new codebase tokenizer path. Left inline; none are blocking.

[jeff-r2026]

--depth lookup output is silently truncated here. This 500-char cap applies to every snippet, including codebase results. But queryCodeKnowledge gives lookup a 20000-token budget and slices up to ~16k chars per page (code-knowledge-recall.ts:300), so the #79 behavior ("full content per result") never reaches the user — all depths render ≤500 chars. Either make the cap depth-aware (raise/skip it for lookup), or drop the per-page budget in queryCodeKnowledge if 500 chars is the intended ceiling; right now the two disagree.

Separately, the maxLen juggling is redundant and can be simplified:

Suggested change

	if (entry.snippet) {
	const maxLen = entry.snippet.length > 500 ? 500 : entry.snippet.length;
	const snippet = entry.snippet.slice(0, maxLen);
	lines.push(`Snippet: ${snippet}${entry.snippet.length > maxLen ? '...' : ''}`);
	}
	if (entry.snippet) {
	const snippet = entry.snippet.slice(0, 500);
	lines.push(`Snippet: ${snippet}${entry.snippet.length > 500 ? '...' : ''}`);
	}

[jeff-r2026]

Docs and queries are tokenized by two different tokenizers. Pages are indexed with fastTokens (regex split) here, which builds df, but the query is tokenized with the shared tokenize (Intl.Segmenter + camelCase) at line 267. A query term like getuserid (Segmenter keeps it whole) is absent from df — docs only stored get/user/id — so df.get(token)=0 yields max idf, while countOccurrences still matches it in raw content (tf>0). Net effect: whole-word identifiers and CJK whole-words get inflated idf and skewed BM25 ranking.

This also cuts against the PR's stated goal of a unified tokenizer — for this module it introduces a second, divergent one. If fastTokens exists only to dodge the Segmenter OOM, prefer having the shared tokenize produce the same token surface (it already truncates to 100K chars) so index-time and query-time terms line up.

[jeff-r2026]

fastTokens caps at 5000 words, but rawTokenCount (used as BM25 dl at line 97) counts the whole document. For an evidence/code page over 5000 words, df/uniqueTokens only reflect the first 5000 words while length-normalization uses the full dl — query terms in the tail are missing from df (idf inflated, same failure mode as the tokenizer mismatch), and dl vs the token set are computed inconsistently. If the cap is just an OOM safeguard, apply the same truncation to rawTokenCount (or drop the cap) so the two stay in sync.