fix: sweep cleanup — deps, docs, tool count, brain_expand, AIDEV-NOTEs

CHANGELOG.md

@@ -1,5 +1,24 @@
 # Changelog
 
+## [Unreleased]
+
+### Added
+- `brain_expand` MCP tool — expand a chunk_id with N surrounding chunks for full context retrieval
+- Groq enrichment backend support (`GROQ_API_KEY`, `BRAINLAYER_GROQ_URL`, `BRAINLAYER_GROQ_MODEL`)
+- `[style]` optional extra: ChromaDB vector store as alternative backend
+- `faiss-cpu` added to `[brain]` optional extra for fast ANN search
+
+### Fixed
+- `brain_graph.py`: replaced correlated subquery (`SELECT source FROM chunks c2 ...`) with literal `'claude_code'` — eliminates N+1 subquery on large DBs, significant performance improvement on 240K+ chunk databases (PR #62)
+- Removed `tree-sitter>=0.21.0` from core dependencies — only `tree-sitter-languages` (in `[ast]`) is used; tree-sitter was pulled in transitively but never directly imported
+- `brainlayer serve` docs: removed non-existent `--http` flag (serve is stdio-only)
+
+### Changed
+- MCP tool count updated from 7 to 8 (brain_expand added)
+- Brain graph clustering documented as Leiden + UMAP (was incorrectly listed as HDBSCAN + UMAP)
+- Test count updated to 715 (was 698)
+- `[kg]` optional extra documented in README and Optional Extras section
+
 ## [1.0.0] - 2026-02-19
 
 ### Added

CLAUDE.md

@@ -20,7 +20,7 @@
 python3 -m venv .venv && source .venv/bin/activate
 pip install -e ".[dev]"
 brainlayer index
-brainlayer serve --http 8787
+brainlayer serve
 brainlayer search "how did I implement authentication"
 brainlayer enrich
 ```
@@ -47,7 +47,7 @@ brainlayer enrich
 - Daemon API (core): `/health`, `/stats`, `/search`, `/context/{chunk_id}`, `/session/{session_id}`
 - Brain graph API: `/brain/graph`, `/brain/node/{node_id}`
 - Backlog API: `/backlog/items` (GET/POST/PATCH/DELETE)
-- MCP tools: `brain_search`, `brain_store`, `brain_recall` (legacy `brainlayer_*` aliases)
+- MCP tools (8): `brain_search`, `brain_store`, `brain_recall`, `brain_entity`, `brain_expand`, `brain_update`, `brain_digest`, `brain_get_person` (legacy `brainlayer_*` aliases still work)
 - MCP server entrypoint: `brainlayer-mcp`
 
 ## Exports

README.md

@@ -1,18 +1,18 @@
 # BrainLayer
 
-> Persistent memory and knowledge graph for AI agents — 7 MCP tools to search, store, recall, digest, and explore entities across every conversation.
+> Persistent memory and knowledge graph for AI agents — 8 MCP tools to search, store, recall, digest, expand, and explore entities across every conversation.
 
 [![PyPI](https://img.shields.io/pypi/v/brainlayer.svg)](https://pypi.org/project/brainlayer/)
 [![CI](https://github.com/EtanHey/brainlayer/actions/workflows/ci.yml/badge.svg)](https://github.com/EtanHey/brainlayer/actions/workflows/ci.yml)
 [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
 [![Python](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
-[![MCP](https://img.shields.io/badge/MCP-7%20tools-green.svg)](https://modelcontextprotocol.io)
-[![Tests](https://img.shields.io/badge/tests-698%20passing-brightgreen.svg)](#testing)
+[![MCP](https://img.shields.io/badge/MCP-8%20tools-green.svg)](https://modelcontextprotocol.io)
+[![Tests](https://img.shields.io/badge/tests-715%20passing-brightgreen.svg)](#testing)
 [![Docs](https://img.shields.io/badge/docs-etanhey.github.io%2Fbrainlayer-blue.svg)](https://etanhey.github.io/brainlayer)
 
 ---
 
-**268,000+ chunks indexed** · **698 tests** · **Hybrid RRF search** · **7 MCP tools** · **Zero cloud dependencies**
+**268,000+ chunks indexed** · **715 tests** · **Hybrid RRF search** · **8 MCP tools** · **Zero cloud dependencies**
 
 **Your AI agent forgets everything between sessions.** Every architecture decision, every debugging session, every preference you've expressed — gone. You repeat yourself constantly.
 
@@ -93,7 +93,7 @@ That's it. Your agent now has persistent memory across every conversation.
 
 ```mermaid
 graph LR
-    A["Claude Code / Cursor / Zed"] -->|MCP| B["BrainLayer MCP Server<br/>7 tools"]
+    A["Claude Code / Cursor / Zed"] -->|MCP| B["BrainLayer MCP Server<br/>8 tools"]
     B --> C["Hybrid Search<br/>semantic + keyword (RRF)"]
     C --> D["SQLite + sqlite-vec<br/>single .db file"]
     B --> KG["Knowledge Graph<br/>entities + relations"]
@@ -113,9 +113,9 @@ graph LR
 | Search | Hybrid: vector similarity + FTS5 keyword, merged with Reciprocal Rank Fusion |
 | Enrichment | Local LLM via Ollama or MLX — 10-field metadata per chunk |
 | MCP Server | stdio-based, MCP SDK v1.26+, compatible with any MCP client |
-| Clustering | HDBSCAN + UMAP for brain graph visualization (optional) |
+| Clustering | Leiden + UMAP for brain graph visualization (optional) |
 
-## MCP Tools (7)
+## MCP Tools (8)
 
 ### Core (3)
 
@@ -125,12 +125,13 @@ graph LR
 | `brain_store` | Persist memories — ideas, decisions, learnings, mistakes. Auto-type/auto-importance. |
 | `brain_recall` | Proactive retrieval — current context, sessions, session summaries. |
 
-### Knowledge Graph (4)
+### Knowledge Graph (5)
 
 | Tool | Description |
 |------|-------------|
 | `brain_digest` | Ingest raw content — entity extraction, relations, sentiment, action items. |
 | `brain_entity` | Look up entities in the knowledge graph — type, relations, evidence. |
+| `brain_expand` | Expand a chunk_id with N surrounding chunks for full context. |
 | `brain_update` | Update, archive, or merge existing memories. |
 | `brain_get_person` | Person lookup — entity details, interactions, preferences (~200-500ms). |
 
@@ -171,7 +172,7 @@ BRAINLAYER_ENRICH_BACKEND=mlx brainlayer enrich --batch-size=100
 
 | | BrainLayer | Mem0 | Zep/Graphiti | Letta | LangChain Memory |
 |---|:---:|:---:|:---:|:---:|:---:|
-| **MCP native** | 7 tools | 1 server | 1 server | No | No |
+| **MCP native** | 8 tools | 1 server | 1 server | No | No |
 | **Think / Recall** | Yes | No | No | No | No |
 | **Local-first** | SQLite | Cloud-first | Cloud-only | Docker+PG | Framework |
 | **Zero infra** | `pip install` | API key | API key | Docker | Multiple deps |
@@ -183,7 +184,7 @@ BRAINLAYER_ENRICH_BACKEND=mlx brainlayer enrich --batch-size=100
 BrainLayer is the only memory layer that:
 1. **Thinks before answering** — categorizes past knowledge by intent (decisions, bugs, patterns) instead of raw search results
 2. **Runs on a single file** — no database servers, no Docker, no cloud accounts
-3. **Works with every MCP client** — 7 tools, instant integration, zero SDK
+3. **Works with every MCP client** — 8 tools, instant integration, zero SDK
 4. **Knowledge graph** — entities, relations, and person lookup across all indexed data
 
 ## CLI Reference
@@ -216,14 +217,19 @@ All configuration is via environment variables:
 | `BRAINLAYER_HEARTBEAT_INTERVAL` | `25` | Log progress every N chunks during enrichment |
 | `BRAINLAYER_SANITIZE_EXTRA_NAMES` | (empty) | Comma-separated names to redact from indexed content |
 | `BRAINLAYER_SANITIZE_USE_SPACY` | `true` | Use spaCy NER for PII detection |
+| `GROQ_API_KEY` | (unset) | Groq API key for cloud enrichment backend |
+| `BRAINLAYER_GROQ_URL` | `https://api.groq.com/openai/v1/chat/completions` | Groq API endpoint |
+| `BRAINLAYER_GROQ_MODEL` | `llama-3.3-70b-versatile` | Groq model for enrichment |
 
 ## Optional Extras
 
 ```bash
-pip install "brainlayer[brain]"     # Brain graph visualization (HDBSCAN + UMAP)
+pip install "brainlayer[brain]"     # Brain graph visualization (Leiden + UMAP) + FAISS
 pip install "brainlayer[cloud]"     # Cloud backfill (Gemini Batch API)
 pip install "brainlayer[youtube]"   # YouTube transcript indexing
 pip install "brainlayer[ast]"       # AST-aware code chunking (tree-sitter)
+pip install "brainlayer[kg]"        # GliNER entity extraction (209M params, EN+HE)
+pip install "brainlayer[style]"     # ChromaDB vector store (alternative backend)
 pip install "brainlayer[dev]"       # Development: pytest, ruff
 ```
 
@@ -244,7 +250,7 @@ BrainLayer can index conversations from multiple sources:
 
 ```bash
 pip install -e ".[dev]"
-pytest tests/                           # Full suite (698 tests)
+pytest tests/                           # Full suite (715 tests)
 pytest tests/ -m "not integration"      # Unit tests only (fast)
 ruff check src/                         # Linting
 ```

docs/mcp-tools.md

@@ -1,6 +1,6 @@
 # MCP Tools Reference
 
-BrainLayer exposes **7 MCP tools** — 3 core (search/store/recall) + 4 knowledge graph (digest/entity/update/get_person).
+BrainLayer exposes **8 MCP tools** — 3 core (search/store/recall) + 5 knowledge graph (digest/entity/expand/update/get_person).
 
 ## brain_search
 
@@ -104,6 +104,24 @@ Look up a known entity in the knowledge graph. Returns entity type, relations, a
 
 ---
 
+## brain_expand
+
+Expand a chunk_id with N surrounding chunks for full context. Useful when a search result is truncated and you need the surrounding conversation or code.
+
+| Parameter | Type | Required | Description |
+|-----------|------|----------|-------------|
+| `chunk_id` | string | Yes | Chunk ID to expand around |
+| `radius` | integer | No | Number of chunks to include on each side (default: 3) |
+| `project` | string | No | Scope to a project |
+
+**Returns:** The target chunk plus N preceding and N following chunks in order.
+
+**Annotations:** `readOnlyHint: true`
+
+**Use when:** A search result is a fragment and you need the full surrounding context — e.g., the full function body, the full conversation turn, or the full stack trace.
+
+---
+
 ## brain_update
 
 Update, archive, or merge existing memories.

greptile.json

@@ -0,0 +1,33 @@
+{
+  "customContext": {
+    "rules": [
+      {
+        "content": "Python project using SQLite + sqlite-vec for vector search. Uses pytest for testing."
+      },
+      {
+        "content": "MCP server with 8 tools: brain_search, brain_store, brain_recall, brain_entity, brain_update, brain_digest, brain_expand, brain_get_person."
+      },
+      {
+        "content": "Entity resolution uses cascading pipeline: normalize → exact match → vector similarity → LLM fallback. Thresholds: 0.92 auto-match, 0.75 fuzzy, <0.75 create new."
+      },
+      {
+        "content": "brain_search defaults to compact format (150-char snippets). Use detail='full' for verbose output."
+      },
+      {
+        "content": "Never hardcode secrets. Use environment variables.",
+        "scope": ["**/*.py"]
+      },
+      {
+        "content": "Flag surprises — if something in the codebase is unexpected, mention it in the review."
+      }
+    ]
+  },
+  "ignorePatterns": [
+    ".venv",
+    "__pycache__",
+    "*.egg-info",
+    ".git",
+    "*.pyc",
+    "claude.scratchpad.md"
+  ]
+}

pyproject.toml

@@ -20,9 +20,6 @@ dependencies = [
     "uvicorn>=0.20.0",
     "httpx>=0.24.0",
 
-    # AST parsing for code chunking (optional - has fallback)
-    "tree-sitter>=0.21.0",
-
     # MCP server
     "mcp>=1.0.0",
 
@@ -58,6 +55,7 @@ brain = [
     "igraph>=0.11.0",  # Graph operations + Leiden via leidenalg
     "leidenalg>=0.10.0",  # Leiden community detection (replaces graspologic)
     "umap-learn>=0.5.0",  # UMAP 3D layout
+    "faiss-cpu>=1.7.0",  # Fast approximate nearest-neighbor search
 ]
 youtube = [
     "yt-dlp>=2024.1.0",  # YouTube video metadata + subtitle download
@@ -70,6 +68,9 @@ ast = [
 kg = [
     "gliner>=0.2.20",  # GLiNER multi-lingual NER (209M params, EN+HE)
 ]
+style = [
+    "chromadb>=0.4.0",  # ChromaDB vector store (alternative to sqlite-vec)
+]
 dev = [
     "pytest>=7.0.0",
     "pytest-asyncio>=0.21.0",

src/brainlayer/kg/__init__.py

@@ -1,4 +1,9 @@
-"""BrainLayer Knowledge Graph — standardized KG spec (matches Convex kgSpec.ts)."""
+"""BrainLayer Knowledge Graph — standardized KG spec (matches Convex kgSpec.ts).
+
+AIDEV-NOTE: Shared KG spec types. Used by 6PM Convex backend. Kept here as the canonical
+reference for the BrainLayer KG standard. If you change ENTITY_TYPES, RELATION_TYPES, or
+DECAY_CONSTANTS, mirror the change in 6pm-mini/convex/lib/kgSpec.ts to keep both in sync.
+"""
 
 import math
 from typing import Optional

src/brainlayer/pipeline/brain_graph.py

@@ -69,14 +69,16 @@ def load_sessions(db_path: str, project: Optional[str] = None) -> list[dict]:
         }
 
     # Get all source_files with their chunk counts and metadata
+    # AIDEV-NOTE: dominant_source is hardcoded to 'claude_code' (not a subquery) — the original
+    # correlated subquery (SELECT source FROM chunks c2 WHERE c2.source_file = chunks.source_file ...)
+    # caused an N+1 performance problem on large DBs (240K+ chunks). The literal is correct for all
+    # current sources; if multi-source support is needed, pre-aggregate in a CTE instead.
     query = """
         SELECT source_file, project, COUNT(*) as chunk_count,
                GROUP_CONCAT(DISTINCT content_type) as types,
                GROUP_CONCAT(DISTINCT intent) as intents,
                AVG(CASE WHEN importance IS NOT NULL THEN importance END) as avg_importance,
-               (SELECT source FROM chunks c2
-                WHERE c2.source_file = chunks.source_file AND c2.source IS NOT NULL
-                GROUP BY source ORDER BY COUNT(*) DESC LIMIT 1) as dominant_source
+               'claude_code' as dominant_source
         FROM chunks
     """
     params = []