Knowledge Graph: Citation/Authority Graph + Interactive Visualization

[Number531]

Summary

Add a Knowledge Graph feature that extracts citations, legal authorities, sections, agents, and their relationships from completed pipeline sessions, stores them in PostgreSQL, and renders an interactive force-directed graph in a new center-panel tab alongside Chat.

Spec: docs/pending-updates/knowledge-graph.md

Motivation

The Citation Chat answers "what does the memo say about X?" but cannot answer "how are these citations connected?" or "which authorities support the CFIUS analysis and where did they come from?" A knowledge graph fills this gap by making relationships, provenance, and cross-section dependencies visible and queryable.

Architecture Decisions (validated via Exa research, March 2026)

Decision	Choice	Rationale
Graph storage	PostgreSQL adjacency tables	Already running pgvector:pg16. LightRAG (29K stars) validates this pattern. Apache AGE as future upgrade path.
Extraction	Hybrid: rule-based 80% + LLM 20%	SAP Practical GraphRAG: dependency parsing achieves 94% of LLM quality. Our pipeline already produces structured citation-map.json.
Visualization	force-graph v1.51 (Canvas)	140K weekly npm downloads, handles 75K elements, CDN-loadable, MIT.
Retrieval	Reciprocal Rank Fusion	Proven 62% → 84% precision lift when merging graph + vector results.
Tab placement	Center panel (like Chat)	Same session-scoped, post-pipeline lifecycle.

Roadmap

Phase 1 — Schema & Extraction (Backend)

• Add ensureKnowledgeGraphSchema() to postgres.js — 5 tables: kg_nodes, kg_edges, kg_evolution, kg_provenance, kg_messages
• Add KNOWLEDGE_GRAPH feature flag to featureFlags.js
• Create knowledgeGraphExtractor.js — post-session extraction:
  • Rule-based: section nodes, agent nodes, source_doc nodes from DB
  • Citation parsing: parse consolidated-footnotes.md → citation nodes + verification tags
  • Citation map: parse citation-map.json → CITES edges (section → citation)
  • LLM classification: single Sonnet call to classify authorities (case/statute/regulation)
  • Similarity edges: cosine > 0.85 from existing report_embeddings
• Wire hookDBBridge.js SessionEnd → setImmediate(buildSessionKnowledgeGraph) (fire-and-forget)
• Add schema init + router mount to claude-sdk-server.js

Phase 2 — API Endpoints

• GET /api/db/sessions/:key/kg/graph — full graph in {nodes, links} format
• GET /api/db/sessions/:key/kg/neighbors/:nodeId — 1-hop neighbors
• GET /api/db/sessions/:key/kg/evolution — graph construction timeline
• GET /api/db/sessions/:key/kg/provenance/:nodeId — extraction audit trail

Phase 3 — Frontend Visualization

• Add force-graph CDN to index.html
• Add "Graph" tab button (locked until session available, same pattern as Chat)
• Add graph tab content: toolbar (layout mode, confidence slider), canvas container, detail panel, composer
• Add styles.css: toolbar, canvas, detail panel, node type colors (reuse design tokens)
• Add app.js: state vars, enableGraph()/disableGraph(), renderForceGraph(), node click → neighbors + provenance, layout switching, confidence filtering

Phase 4 — Graph-Aware Q&A (RRF Hybrid Retrieval)

• Create knowledgeGraphRouter.js — SSE streaming endpoint
  • Keyword match on node labels + embedding similarity on query
  • 2-hop graph traversal via recursive CTE
  • Vector similarity via existing searchSimilar()
  • Reciprocal Rank Fusion merge (k=60)
  • Messages API streaming with Anthropic Citations
• Wire frontend: openKgStream(), handleKgEvent(), highlight relevant nodes on query

Phase 5 — Tests & Polish

• Unit tests: schema creation, citation parsing, edge creation, RRF fusion, API endpoints
• Integration test: live pipeline → verify graph populated → verify force-graph renders
• Evolution timeline playback (scrubber to replay graph growth across phases)
• Cross-session authority dedup via canonical_key (same case across memos)

Files

New (3): knowledgeGraphExtractor.js, knowledgeGraphRouter.js, knowledge-graph.test.js

Modified (8): postgres.js, featureFlags.js, claude-sdk-server.js, hookDBBridge.js, dbFrontendRouter.js, index.html, styles.css, app.js

Docker: Zero changes — CDN-loaded lib, auto-created schema, no new npm deps.

Dependencies

• Requires HOOK_DB_PERSISTENCE=true + EMBEDDING_PERSISTENCE=true (both already in production)
• Related: Embedding Persistence — Gemini pgvector semantic search (v4.9.0) #46 (Embedding Persistence), feat: Database-Frontend Integration (Session History) #19 (DB-Frontend Integration), JSON to PostgreSQL Database Migration — Cross-Session Analytics & ACID Storage #9 (JSON→PostgreSQL Migration)

---

🤖 Generated with Claude Code

[Number531]

Update: Pre-Deployment Validation Gate Added

The plan file has been updated with comprehensive testing requirements that must pass before Docker deployment (commit 79f2c01).

What was added

§12 Testing Plan now includes:

Test Category	Checks	Purpose
Docker Smoke	6	Build, startup, health flag, schema existence, empty endpoint, CDN load
Degraded-Mode	4	Gemini unavailable, no citations, unhandled errors, DB connection loss
Circuit Breaker Isolation	2	KG failures must NOT trip the shared persistence breaker
Feature Flag Rollback	3	Off by default, toggle on/off, idempotent schema
Frontend Resilience	6	CDN fallback, empty state, 500+ nodes perf, API failure, SSE abort, cross-browser
Live Pipeline Smoke	10	Full end-to-end against 393-footnote session
Performance Benchmarks	7	Extraction <30s, render <3s, endpoints <200ms, 30fps canvas

Critical finding: Circuit Breaker Isolation

The KG extractor must use a dedicated circuit breaker instance, not the shared one in hookDBBridge.js. If KG writes fail and trip the shared breaker, it would suspend ALL persistence (reports, audit log, agent states) — the highest-risk integration bug.

Complexity: Moderate-High (~1,150 new lines)

Highest-risk items requiring most review:

1. knowledgeGraphExtractor.js — citation parsing edge cases, LLM call failure handling
2. Circuit breaker isolation — separate instance mandatory
3. hookDBBridge.js trigger — must not block/break SessionEnd flow

Frontend hardening

• CDN onerror handler → shows "Graph visualization unavailable" instead of blank tab
• Empty state messaging for sessions with no citation data
• renderForceGraph() guards against missing ForceGraph global

[Number531]

Merged in PR #62 (96 commits, +9,161 lines). Live on GCE staging v5.2.0. 10-phase extraction pipeline, provenance chains, force-graph visualization, RRF hybrid Q&A.