diff --git a/docs/memory-benchmark-roadmap.md b/docs/memory-benchmark-roadmap.md new file mode 100644 index 00000000..07d3de18 --- /dev/null +++ b/docs/memory-benchmark-roadmap.md @@ -0,0 +1,230 @@ +# Memory Benchmark Roadmap + +This document lays out how Inkwell should evaluate its memory system as we move from simple retrieval toward richer long-term memory, reflection, and context-eviction behavior. + +## Benchmark philosophy + +We should start by measuring ourselves against existing public memory benchmarks before inventing new ones. That gives us an honest baseline, makes external comparison possible, and forces discipline around evaluation before we optimize for our own runtime. + +There are two axes we ultimately care about: + +1. **Retrieval quality** — does the right memory surface? +2. **Continuity efficiency** — how much context/token budget does it take to preserve continuity over time? + +Phase 1 is about the first axis. Phase 2 introduces the second. + +## Lessons from other systems + +### MemPal + +MemPal's most important lesson is not any single top-line score. It is benchmark discipline. + +- Keep a strong simple baseline. +- Distinguish clean vs contaminated results. +- Store per-case outputs, not just aggregate metrics. +- Separate retrieval quality from end-to-end answer quality. +- Be explicit when improvements come from heuristics, reranking, or architecture changes. + +Architecturally, MemPal also suggests that **verbatim memory is a stronger baseline than many systems assume**. Compression and extraction can destroy signal. We should treat raw memory plus good retrieval as a real baseline, not something to immediately outgrow. + +### Hermes + +Hermes draws a strong line between: + +- small curated memory that is always in-context +- larger searchable history that is only pulled when needed + +That distinction matters for Inkwell too. Our future benchmarks should separate: + +- long-term retrieval quality +- bootstrap relevance +- live context budget behavior + +### Honcho + +Honcho's lesson is that memory can be more than retrieval. It can become a **stateful user/project model** with asynchronous background derivation and lightweight representations for prompt hydration. + +That should influence our later dream-phase work, but it should not distract us from proving the base retrieval layer first. + +## Benchmark tracks + +### Track 1 — Standard public retrieval benchmarks + +This is the immediate priority. + +We should support and compare on: + +- **LongMemEval** — long-horizon conversational memory retrieval +- **LoCoMo** — multi-hop conversational QA / temporal retrieval pressure +- **ConvoMem** — large-scale conversational memory evaluation +- **MemBench / BEAM-style suites** — broader long-context and noisy-memory stress tests + +For each benchmark, we should be able to evaluate: + +- text retrieval +- semantic retrieval +- hybrid retrieval +- chunked semantic retrieval +- optional rerank as a separate tier + +Metrics: + +- Recall@1 / @3 / @5 / @10 +- MRR +- NDCG +- latency +- optional rerank cost + +### Track 2 — Bootstrap relevance + +We already have the beginning of this. + +Question: + +- given a thread/focus/session context, do we inject the right memories into bootstrap? + +This measures relevance of **memory selection for live work**, not just abstract retrieval. + +### Track 3 — Ink-native context eviction + +This is where Ink can become genuinely differentiated. + +Question: + +- when an SB can manage and evict its own context, how well does continuity survive? + +This belongs after public benchmark parity, not before. + +## Parallel second brain implications + +The most relevant and unique lesson from our parallel second brain direction is that memory should not be a single retrieval layer. We should preserve **raw memories** while also letting a slower background system derive additional memory views. + +The likely long-term layers are: + +- raw chunked memory for faithful recall +- durable fact extraction for concise stable claims +- entity/person/project memories for who-or-what centric lookup +- summary memories for coarse routing across large histories +- override / contradiction links so newer policy or state can explicitly supersede older memory + +That means our future benchmark tiers should not just compare text vs semantic vs hybrid. They should eventually compare: + +- raw only +- raw + durable facts +- raw + entity/fact indexes +- raw + override-aware scoring + +This is especially important for domains like policy and healthcare, where semantic similarity alone is not enough. If one memory overrides another, retrieval quality depends on chronology, provenance, and explicit linkage as much as embedding distance. + +For now, this affects the benchmark roadmap in one specific way: we should keep our baseline honest, but design the harness so we can later add multiple retrieval views and score how much each derived layer helps or hurts. + +## Evaluation rules + +We should adopt explicit benchmark hygiene rules: + +- Keep a **cheap baseline** that uses no LLM extraction/rerank. +- Introduce a fixed **dev / held-out split** for any internally tuned benchmark set. +- Label runs as: + - `clean` + - `tuned_on_dev` + - `contaminated` +- Persist **per-case failures** and top retrieved candidates. +- Treat retrieval and answer-generation as separate measurements. +- Never publish a score without saying whether reranking / LLM extraction was involved. + +## Inkwell benchmark roadmap + +### Phase 1 — Public benchmark parity + +Goal: run Inkwell against standard external benchmark families and produce honest baseline numbers. + +Deliverables: + +- dataset loaders/adapters for standard public benchmarks +- benchmark run metadata that records family, split, and mode +- per-case result persistence +- clean baseline vs rerank-assisted tiers + +### Phase 2 — Benchmark hygiene upgrade + +Goal: make our results publishable and comparable over time. + +Deliverables: + +- dev/held-out split support for internal sets +- contamination labeling +- regression tracking by architecture version +- benchmark comparison tables over time + +Current implementation direction: + +- phase-2 retrieval should query **multiple chunk views** (`summary`, `fact`, `topic`, `entity`, `content`) +- retrieval should be able to filter chunk types at the RPC layer +- `memory_embedding_chunks(user_id, chunk_type)` should be indexed so view-specific retrieval stays cheap +- derived-view matches and raw-content matches should merge before hybrid scoring + +Current ablation flag: + +```bash +MEMORY_BENCHMARK_VARIANT= yarn benchmark:memory-recall +``` + +Supported variants: + +- `default` +- `content-only` +- `derived-only` +- `multiview-no-boost` +- `multiview-no-chrono` + +### Phase 3 — Dream-phase memory + +Goal: test the value of durable fact extraction and higher-order summaries. + +Deliverables: + +- benchmark modes for: + - raw memory only + - raw + durable facts + - raw + dream-phase summaries + - raw + dream-phase + rerank +- explicit comparison between extraction-enhanced memory and verbatim baselines + +Early implementation slices: + +- chronology-aware reranking as the first optional second pass +- durable fact candidates linked back to source memories +- duplicate-candidate detection for same-topic memories +- supersession / contradiction candidate detection for chronological review + +### Phase 4 — Ink-native context eviction benchmark + +Goal: measure continuity under context pressure. + +Possible metrics: + +- task success after eviction +- recovery rate for evicted-but-needed context +- false reinjection rate +- turns-to-recovery +- tokens freed vs continuity preserved + +## Near-term implementation order + +1. Add benchmark family descriptors and public benchmark scaffolding. +2. Wire the first external benchmark family into the existing benchmark scripts. +3. Add dev/held-out split support for internal sets. +4. Add run labeling for clean vs contaminated evaluation. +5. Only then design the first Ink-native eviction benchmark. + +## What success looks like + +Short term: + +- Inkwell can run against the same public memory benchmarks other systems cite. +- We can report quality and cost honestly. +- We can compare raw, hybrid, chunked, and reranked modes clearly. + +Long term: + +- Ink can show not just that it retrieves well, but that it preserves continuity under self-managed context pressure. diff --git a/package.json b/package.json index ebf0696f..a13fc68e 100644 --- a/package.json +++ b/package.json @@ -43,8 +43,8 @@ "supabase:local:setup": "bash ./scripts/setup-local-supabase.sh", "test:integration:db:local": "bash ./scripts/test-integration-db-local.sh", "test:integration:runtime": "yarn workspace @inklabs/api test:integration:runtime", - "benchmark:memory-recall": "yarn workspace @inklabs/api benchmark:memory-recall", - "benchmark:bootstrap-relevance": "yarn workspace @inklabs/api benchmark:bootstrap-relevance", + "benchmark:memory-recall": "yarn workspace @inklabs/benchmarks benchmark:memory-recall", + "benchmark:bootstrap-relevance": "yarn workspace @inklabs/benchmarks benchmark:bootstrap-relevance", "backfill:memory-embeddings": "yarn workspace @inklabs/api backfill:memory-embeddings", "lint": "yarn workspaces foreach -A run lint", "type-check": "yarn workspaces foreach -A -t run type-check", diff --git a/packages/api/package.json b/packages/api/package.json index 23c2b266..c82f3eea 100644 --- a/packages/api/package.json +++ b/packages/api/package.json @@ -6,6 +6,10 @@ "publishConfig": { "access": "public" }, + "exports": { + "./benchmarks": "./src/benchmarks.ts", + "./benchmark-constants": "./src/services/memory-benchmark-constants.ts" + }, "description": "Inkwell API server with MCP, Telegram bot, and REST API", "main": "dist/index.js", "types": "dist/index.d.ts", @@ -25,10 +29,12 @@ "test:coverage": "vitest run --coverage", "test:connection": "tsx src/test-connection.ts", "test:channels": "tsx src/test-channels.ts", - "benchmark:memory-recall": "tsx src/scripts/benchmark-memory-recall.ts", - "benchmark:bootstrap-relevance": "tsx src/scripts/benchmark-bootstrap-relevance.ts", + "benchmark:memory-recall": "yarn workspace @inklabs/benchmarks benchmark:memory-recall", + "benchmark:bootstrap-relevance": "yarn workspace @inklabs/benchmarks benchmark:bootstrap-relevance", "backfill:memory-embeddings": "tsx src/scripts/backfill-memory-embeddings.ts", "backfill:artifact-embeddings": "tsx src/scripts/backfill-artifact-embeddings.ts", + "audit:memory-llm-extractions": "tsx src/scripts/audit-memory-llm-extractions.ts", + "extract:memory-llm-views": "tsx src/scripts/extract-memory-llm-views.ts", "lint": "eslint src --ext .ts", "type-check": "yarn workspace @inklabs/shared build && tsc --noEmit", "clean": "rm -rf dist" diff --git a/packages/api/src/agent/tools/pi-coding-tools.ts b/packages/api/src/agent/tools/pi-coding-tools.ts index d0870652..b45646ac 100644 --- a/packages/api/src/agent/tools/pi-coding-tools.ts +++ b/packages/api/src/agent/tools/pi-coding-tools.ts @@ -8,6 +8,7 @@ */ import path from 'path'; +import { readdir, readFile, stat } from 'fs/promises'; import type Anthropic from '@anthropic-ai/sdk'; import { logger } from '../../utils/logger'; import { guardBashCommand } from './bash-guard'; @@ -124,6 +125,107 @@ function formatToolResult(result: unknown): string { } const DEFAULT_BASH_TIMEOUT_SECONDS = 120; +const MAX_PORTABLE_SEARCH_FILES = 2000; + +function relativeToolPath(cwd: string, filePath: string): string { + const relative = path.relative(cwd, filePath); + return relative || '.'; +} + +async function collectFiles(root: string, files: string[] = []): Promise { + if (files.length >= MAX_PORTABLE_SEARCH_FILES) return files; + + const entries = await readdir(root, { withFileTypes: true }); + for (const entry of entries) { + if (files.length >= MAX_PORTABLE_SEARCH_FILES) break; + if (entry.name === 'node_modules' || entry.name === '.git') continue; + + const entryPath = path.join(root, entry.name); + if (entry.isDirectory()) { + await collectFiles(entryPath, files); + } else if (entry.isFile()) { + files.push(entryPath); + } + } + + return files; +} + +function globToRegExp(pattern: string): RegExp { + const escaped = pattern + .replace(/[.+^${}()|[\]\\]/g, '\\$&') + .replace(/\*/g, '.*') + .replace(/\?/g, '.'); + return new RegExp(`^${escaped}$`); +} + +async function portableGrep(params: Record, cwd: string): Promise { + const pattern = typeof params.pattern === 'string' ? params.pattern : ''; + if (!pattern) return 'Error: pattern is required'; + + const searchPath = typeof params.path === 'string' && params.path ? params.path : '.'; + const root = path.resolve(cwd, searchPath); + const rootStat = await stat(root); + const files = rootStat.isDirectory() ? await collectFiles(root) : [root]; + const matches: string[] = []; + + for (const file of files) { + let content: string; + try { + content = await readFile(file, 'utf-8'); + } catch { + continue; + } + + const lines = content.split(/\r?\n/); + for (const [index, line] of lines.entries()) { + if (line.includes(pattern)) { + matches.push(`${relativeToolPath(cwd, file)}:${index + 1}:${line}`); + } + } + } + + return matches.length ? matches.join('\n') : '(no matches)'; +} + +async function portableFind(params: Record, cwd: string): Promise { + const pattern = typeof params.pattern === 'string' && params.pattern ? params.pattern : '*'; + const searchPath = typeof params.path === 'string' && params.path ? params.path : '.'; + const root = path.resolve(cwd, searchPath); + const matcher = globToRegExp(pattern); + const rootStat = await stat(root); + const files = rootStat.isDirectory() ? await collectFiles(root) : [root]; + const matches = files + .map((file) => relativeToolPath(cwd, file)) + .filter((file) => matcher.test(path.basename(file))); + + return matches.length ? matches.join('\n') : '(no matches)'; +} + +async function maybePortableSearchFallback( + toolName: string, + formattedResult: string, + params: Record, + cwd: string +): Promise { + if ( + toolName === 'grep' && + formattedResult.includes('ripgrep (rg) is not available and could not be downloaded') + ) { + logger.warn('Pi grep unavailable; using portable grep fallback'); + return portableGrep(params, cwd); + } + + if ( + toolName === 'find' && + formattedResult.includes('fd is not available and could not be downloaded') + ) { + logger.warn('Pi find unavailable; using portable find fallback'); + return portableFind(params, cwd); + } + + return null; +} /** * Create Pi coding tools adapted for the Ink backend. @@ -202,9 +304,19 @@ export async function createInkCodingTools( const callId = `ink-${tool.name}-${Date.now()}`; try { const result = await tool.execute(callId, params, signal); - return formatToolResult(result); + const formatted = formatToolResult(result); + return ( + (await maybePortableSearchFallback(tool.name, formatted, params, config.cwd)) ?? formatted + ); } catch (err) { const message = err instanceof Error ? err.message : String(err); + const fallback = await maybePortableSearchFallback( + tool.name, + `Error: ${message}`, + params, + config.cwd + ); + if (fallback !== null) return fallback; logger.error(`Pi tool ${tool.name} failed`, { error: message, params }); return `Error: ${message}`; } diff --git a/packages/api/src/benchmarks.ts b/packages/api/src/benchmarks.ts new file mode 100644 index 00000000..0854ef0f --- /dev/null +++ b/packages/api/src/benchmarks.ts @@ -0,0 +1,13 @@ +export { createSupabaseClient } from './data/supabase/client'; +export { MemoryRepository } from './data/repositories/memory-repository'; +export { MEMORY_EMBEDDING_CHUNKS_VERSION } from './services/embeddings/memory-chunks'; +export { + DEFAULT_MEMORY_LLM_MODEL, + MEMORY_EXTRACTION_VERSION, +} from './services/memory-llm-extraction'; +export type { + MemoryHybridChunkStrategy, + MemorySemanticQueryStrategy, + MemorySearchChunkType, + MemorySearchOptions, +} from './data/models/memory'; diff --git a/packages/api/src/config/env.ts b/packages/api/src/config/env.ts index f16d4e5d..87fde85d 100644 --- a/packages/api/src/config/env.ts +++ b/packages/api/src/config/env.ts @@ -177,6 +177,29 @@ const envSchema = z.object({ MEMORY_EMBEDDING_DIMENSIONS: optionalNumber, MEMORY_EMBEDDING_QUERY_THRESHOLD: optionalNumber, MEMORY_EMBEDDING_MATCH_COUNT_MULTIPLIER: optionalNumber, + MEMORY_LLM_EXTRACTION_ENABLED: z + .enum(['true', 'false']) + .default('false') + .transform((v) => v === 'true'), + MEMORY_EXTRACTION_MODE: z.enum(['heuristic', 'llm', 'merged']).default('heuristic'), + MEMORY_LLM_MODEL: optionalString, + MEMORY_LLM_MAX_INPUT_CHARS: optionalNumber, + MEMORY_LLM_ENTITY_ENABLED: z + .enum(['true', 'false']) + .default('false') + .transform((v) => v === 'true'), + MEMORY_LLM_DURABLE_FACT_ENABLED: z + .enum(['true', 'false']) + .default('false') + .transform((v) => v === 'true'), + MEMORY_LLM_SUMMARY_ENABLED: z + .enum(['true', 'false']) + .default('false') + .transform((v) => v === 'true'), + MEMORY_LLM_CURRENT_STATE_ENABLED: z + .enum(['true', 'false']) + .default('false') + .transform((v) => v === 'true'), OLLAMA_BASE_URL: optionalUrl, OPENAI_API_KEY: optionalString, OPENAI_BASE_URL: optionalUrl, @@ -200,7 +223,8 @@ const parseEnv = () => { } // Create normalized keys (prefer INK_, fall back to PCP_ for backward compat) - const hasBaseOverride = parsed.INK_PORT_BASE !== undefined || parsed.PCP_PORT_BASE !== undefined; + const hasBaseOverride = + parsed.INK_PORT_BASE !== undefined || parsed.PCP_PORT_BASE !== undefined; // Base is MCP-first: MCP=base, WEB=base+1, MYRA=base+2 const portBase = parsed.INK_PORT_BASE ?? parsed.PCP_PORT_BASE ?? 3001; @@ -221,6 +245,7 @@ const parseEnv = () => { const memoryEmbeddingDimensions = parsed.MEMORY_EMBEDDING_DIMENSIONS ?? 1024; const memoryEmbeddingQueryThreshold = parsed.MEMORY_EMBEDDING_QUERY_THRESHOLD ?? 0.2; const memoryEmbeddingMatchCountMultiplier = parsed.MEMORY_EMBEDDING_MATCH_COUNT_MULTIPLIER ?? 5; + const memoryLlmMaxInputChars = parsed.MEMORY_LLM_MAX_INPUT_CHARS ?? 12000; return { ...parsed, @@ -234,6 +259,7 @@ const parseEnv = () => { MEMORY_EMBEDDING_DIMENSIONS: memoryEmbeddingDimensions, MEMORY_EMBEDDING_QUERY_THRESHOLD: memoryEmbeddingQueryThreshold, MEMORY_EMBEDDING_MATCH_COUNT_MULTIPLIER: memoryEmbeddingMatchCountMultiplier, + MEMORY_LLM_MAX_INPUT_CHARS: memoryLlmMaxInputChars, } as typeof parsed & { INK_PORT_BASE: number; PORT: number; @@ -245,6 +271,7 @@ const parseEnv = () => { MEMORY_EMBEDDING_DIMENSIONS: number; MEMORY_EMBEDDING_QUERY_THRESHOLD: number; MEMORY_EMBEDDING_MATCH_COUNT_MULTIPLIER: number; + MEMORY_LLM_MAX_INPUT_CHARS: number; }; } catch (error) { if (error instanceof z.ZodError) { diff --git a/packages/api/src/data/models/memory.ts b/packages/api/src/data/models/memory.ts index be9c58e8..3936d3d3 100644 --- a/packages/api/src/data/models/memory.ts +++ b/packages/api/src/data/models/memory.ts @@ -64,6 +64,16 @@ export interface MemoryCreateInput { contactId?: string; // Per-sender memory scoping } +export type MemorySearchChunkType = + | 'summary' + | 'fact' + | 'topic' + | 'entity' + | 'current_state' + | 'content'; +export type MemoryHybridChunkStrategy = 'default' | 'content-only' | 'derived-only' | 'multi-view'; +export type MemorySemanticQueryStrategy = 'single' | 'parallel-content-entity'; + export interface MemorySearchOptions { recallMode?: 'auto' | 'text' | 'semantic' | 'hybrid'; source?: MemorySource; @@ -75,6 +85,12 @@ export interface MemorySearchOptions { agentId?: string; // Filter by agent includeShared?: boolean; // Include shared memories (agentId=null) when filtering. Default true. contactId?: string; // Filter by contact for per-sender isolation + semanticChunkTypes?: MemorySearchChunkType[]; + semanticQueryStrategy?: MemorySemanticQueryStrategy; + hybridChunkStrategy?: MemoryHybridChunkStrategy; + applyChunkTypeBoosts?: boolean; + applyMultiViewBoost?: boolean; + applyChronologyBoost?: boolean; } export type SessionPhase = diff --git a/packages/api/src/data/repositories/memory-repository.test.ts b/packages/api/src/data/repositories/memory-repository.test.ts index bcd13b2e..2406aaa7 100644 --- a/packages/api/src/data/repositories/memory-repository.test.ts +++ b/packages/api/src/data/repositories/memory-repository.test.ts @@ -208,6 +208,184 @@ describe('MemoryRepository', () => { expect(mockSupabase._queryBuilder.overlaps).toHaveBeenCalledWith('topics', ['work', 'ai']); }); + + it('passes semantic chunk type filters through to chunked semantic recall', async () => { + const semanticSpy = vi + .spyOn(repo as any, 'trySemanticRecallCandidates') + .mockResolvedValue([]); + + await repo.recall('user-456', 'latest policy', { + recallMode: 'semantic', + semanticChunkTypes: ['content'], + applyChunkTypeBoosts: false, + }); + + expect(semanticSpy).toHaveBeenCalledWith( + 'user-456', + 'latest policy', + expect.objectContaining({ + recallMode: 'semantic', + semanticChunkTypes: ['content'], + applyChunkTypeBoosts: false, + }), + 20, + 0, + ['content'] + ); + }); + + it('supports parallel semantic(content) and semantic(entity) recall plans', async () => { + const contentMemory = { + id: 'mem-content', + userId: 'user-456', + content: 'daily commute takes 35 minutes by bus', + source: 'observation', + salience: 'medium', + topics: [], + metadata: {}, + version: 1, + createdAt: new Date('2026-01-26T12:00:00Z'), + }; + const entityMemory = { + id: 'mem-entity', + userId: 'user-456', + content: 'playlist named Neon Rain', + source: 'observation', + salience: 'medium', + topics: [], + metadata: {}, + version: 1, + createdAt: new Date('2026-01-25T12:00:00Z'), + }; + + const semanticSpy = vi + .spyOn(repo as any, 'trySemanticRecallCandidates') + .mockImplementation(async (_userId, _query, _options, _limit, _offset, chunkTypes) => { + if (Array.isArray(chunkTypes) && chunkTypes.length === 1 && chunkTypes[0] === 'content') { + return [ + { + memory: contentMemory, + semanticScore: 0.9, + matchedChunkType: 'content', + semanticEvidenceCount: 1, + finalScore: 0.9, + }, + ]; + } + + if (Array.isArray(chunkTypes) && chunkTypes.length === 1 && chunkTypes[0] === 'entity') { + return [ + { + memory: entityMemory, + semanticScore: 0.85, + matchedChunkType: 'entity', + semanticEvidenceCount: 1, + finalScore: 0.85, + }, + ]; + } + + return []; + }); + + const results = await repo.recall('user-456', 'what is relevant?', { + recallMode: 'semantic', + semanticChunkTypes: ['content', 'entity'], + semanticQueryStrategy: 'parallel-content-entity', + applyChunkTypeBoosts: false, + }); + + expect(results.map((memory) => memory.id)).toEqual(['mem-content', 'mem-entity']); + expect(semanticSpy).toHaveBeenCalledTimes(2); + expect(semanticSpy).toHaveBeenNthCalledWith( + 1, + 'user-456', + 'what is relevant?', + expect.objectContaining({ + semanticChunkTypes: ['content', 'entity'], + semanticQueryStrategy: 'parallel-content-entity', + applyChunkTypeBoosts: false, + }), + 20, + 0, + ['content'] + ); + expect(semanticSpy).toHaveBeenNthCalledWith( + 2, + 'user-456', + 'what is relevant?', + expect.objectContaining({ + semanticChunkTypes: ['content', 'entity'], + semanticQueryStrategy: 'parallel-content-entity', + applyChunkTypeBoosts: false, + }), + 20, + 0, + ['entity'] + ); + }); + + it('supports hybrid content-only ablations without derived semantic pass', async () => { + const textMemory = { + id: 'mem-text', + userId: 'user-456', + content: 'latest policy', + source: 'observation', + salience: 'medium', + topics: [], + metadata: {}, + version: 1, + createdAt: new Date('2026-01-26T12:00:00Z'), + }; + + const semanticSpy = vi + .spyOn(repo as any, 'trySemanticRecallCandidates') + .mockImplementation(async (_userId, _query, _options, _limit, _offset, chunkTypes) => { + if (Array.isArray(chunkTypes) && chunkTypes.includes('content')) { + return [ + { + memory: textMemory, + semanticScore: 0.8, + matchedChunkType: 'content', + finalScore: 0.8, + }, + ]; + } + + return []; + }); + vi.spyOn(repo as any, 'textRecallCandidates').mockResolvedValue([ + { + memory: textMemory, + textScore: 1, + finalScore: 1, + }, + ]); + + const results = await repo.recall('user-456', 'latest policy', { + recallMode: 'hybrid', + hybridChunkStrategy: 'content-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + + expect(results).toHaveLength(1); + expect(semanticSpy).toHaveBeenCalledTimes(1); + expect(semanticSpy).toHaveBeenCalledWith( + 'user-456', + 'latest policy', + expect.objectContaining({ + hybridChunkStrategy: 'content-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }), + expect.any(Number), + 0, + ['content'] + ); + }); }); describe('forget', () => { @@ -1056,22 +1234,22 @@ describe('MemoryRepository', () => { chunk_index: 0, chunk_text: 'Chunked summary', }), - expect.objectContaining({ - memory_id: 'mem-hm-5', - chunk_type: 'content', - chunk_index: 1, - }), ]), expect.objectContaining({ onConflict: 'memory_id,chunk_index' }) ); + expect(chunkRows.some((row) => row.chunk_type === 'content')).toBe(true); + expect(chunkRows.some((row) => row.chunk_type === 'topic')).toBe(true); expect(mockSupabase._queryBuilder.update).toHaveBeenCalledWith( expect.objectContaining({ - embedding_chunks_version: 1, + embedding_chunks_version: 2, embedding_chunk_count: chunkRows.length, metadata: expect.objectContaining({ embedding_chunks: expect.objectContaining({ chunkCount: chunkRows.length, - version: 1, + version: 2, + viewCounts: expect.objectContaining({ + summary: 1, + }), }), embedding: expect.objectContaining({ provider: 'ollama', @@ -1344,13 +1522,25 @@ describe('MemoryRepository', () => { topics: ['pr:214'], agent_id: 'lumen', embedding: '[0.1,0.2,0.3]', - metadata: {}, + metadata: { + embedding_chunks: { + version: 2, + viewCounts: { + summary: 1, + fact: 1, + topic: 1, + entity: 0, + content: 1, + }, + }, + }, version: 1, created_at: '2026-03-16T00:00:00Z', expires_at: null, sb_id: null, matched_chunk_index: 0, matched_chunk_text: 'Semantic memory', + matched_chunk_type: 'summary', similarity: 0.9, }, ], @@ -1489,6 +1679,186 @@ describe('MemoryRepository', () => { expect(rpc).not.toHaveBeenCalled(); expect(results).toEqual([]); }); + + it('hybrid recall queries derived and raw chunk views separately', async () => { + const rpc = vi + .fn() + .mockResolvedValueOnce({ + data: [ + { + id: 'mem-derived-1', + user_id: 'user-456', + content: 'Current policy override note', + summary: 'Current policy override note', + topic_key: 'policy:wound-care', + source: 'observation', + salience: 'high', + topics: ['policy:wound-care'], + agent_id: 'lumen', + embedding: '[0.1,0.2,0.3]', + metadata: {}, + version: 1, + created_at: '2026-03-16T00:00:00Z', + expires_at: null, + identity_id: null, + matched_chunk_index: 1, + matched_chunk_text: 'Current policy override note', + matched_chunk_type: 'fact', + similarity: 0.88, + }, + ], + error: null, + }) + .mockResolvedValueOnce({ + data: [ + { + id: 'mem-derived-1', + user_id: 'user-456', + content: 'Current policy override note', + summary: 'Current policy override note', + topic_key: 'policy:wound-care', + source: 'observation', + salience: 'high', + topics: ['policy:wound-care'], + agent_id: 'lumen', + embedding: '[0.1,0.2,0.3]', + metadata: {}, + version: 1, + created_at: '2026-03-16T00:00:00Z', + expires_at: null, + identity_id: null, + matched_chunk_index: 4, + matched_chunk_text: 'Current policy override note', + matched_chunk_type: 'content', + similarity: 0.81, + }, + ], + error: null, + }); + + (mockSupabase as any).rpc = rpc; + (repo as any).embeddingRouter = { + embedQuery: vi.fn().mockResolvedValue({ + vector: [0.1, 0.2, 0.3], + provider: 'openai', + model: 'text-embedding-3-small', + dimensions: 1024, + }), + getRuntimeConfig: vi.fn().mockReturnValue({ + enabled: true, + provider: 'openai', + model: 'text-embedding-3-small', + dimensions: 1024, + queryThreshold: 0.2, + matchCountMultiplier: 5, + ollamaBaseUrl: 'http://localhost:11434', + openaiBaseUrl: 'https://api.openai.com', + hasOpenAIKey: true, + }), + }; + (repo as any).textRecallCandidates = vi.fn().mockResolvedValue([]); + + const results = await repo.recall('user-456', 'current override policy', { + recallMode: 'hybrid', + hybridChunkStrategy: 'multi-view', + }); + + expect(rpc).toHaveBeenNthCalledWith( + 1, + 'match_memory_embedding_chunks', + expect.objectContaining({ + p_chunk_types: ['summary', 'fact', 'topic', 'entity', 'current_state'], + }) + ); + expect(rpc).toHaveBeenNthCalledWith( + 2, + 'match_memory_embedding_chunks', + expect.objectContaining({ + p_chunk_types: ['content'], + }) + ); + expect(results).toHaveLength(1); + expect(results[0].id).toBe('mem-derived-1'); + }); + + it('chronology-aware reranking prefers newer override memories for current-policy queries', async () => { + const olderMemory = { + id: 'mem-old', + userId: 'user-456', + content: 'Wound-care escalation policy. Notify the old triage lead on call.', + summary: 'Old escalation policy', + topicKey: 'policy:wound-care', + source: 'observation', + salience: 'high', + topics: ['policy:wound-care'], + metadata: {}, + version: 1, + createdAt: new Date('2026-03-01T00:00:00Z'), + }; + const newerMemory = { + id: 'mem-new', + userId: 'user-456', + content: + 'Current wound-care escalation policy now uses the inpatient lead and overrides the previous protocol.', + summary: 'Current escalation policy overrides previous protocol', + topicKey: 'policy:wound-care', + source: 'observation', + salience: 'high', + topics: ['policy:wound-care'], + metadata: {}, + version: 1, + createdAt: new Date('2026-03-20T00:00:00Z'), + }; + + (repo as any).trySemanticRecallCandidates = vi + .fn() + .mockResolvedValueOnce([ + { + memory: olderMemory, + semanticScore: 0.95, + matchedChunkType: 'summary', + finalScore: 0.95, + }, + { + memory: newerMemory, + semanticScore: 0.9, + matchedChunkType: 'fact', + finalScore: 0.9, + }, + ]) + .mockResolvedValueOnce([ + { + memory: olderMemory, + semanticScore: 0.9, + matchedChunkType: 'content', + finalScore: 0.9, + }, + { + memory: newerMemory, + semanticScore: 0.88, + matchedChunkType: 'content', + finalScore: 0.88, + }, + ]); + (repo as any).textRecallCandidates = vi.fn().mockResolvedValue([]); + (repo as any).embeddingRouter = { + getRuntimeConfig: vi.fn().mockReturnValue({ + matchCountMultiplier: 5, + }), + }; + + const results = await (repo as any).hybridRecall( + 'user-456', + 'what is the current wound-care policy override', + { hybridChunkStrategy: 'multi-view' }, + 5, + 0 + ); + + expect(results).toHaveLength(2); + expect(results[0].id).toBe('mem-new'); + expect(results[1].id).toBe('mem-old'); + }); }); // ─── Per-Sender Memory Isolation Tests ─── diff --git a/packages/api/src/data/repositories/memory-repository.ts b/packages/api/src/data/repositories/memory-repository.ts index 81e1cc66..9f70a3a7 100644 --- a/packages/api/src/data/repositories/memory-repository.ts +++ b/packages/api/src/data/repositories/memory-repository.ts @@ -10,15 +10,23 @@ import { buildChunkMetadataUpdate, buildChunkRows, buildMemoryEmbeddingChunks, + countChunkViews, + type EmbeddedMemoryChunk, formatVectorLiteral, + inferChunkTypeFromMetadata, + type MemoryChunkType, MEMORY_EMBEDDING_CHUNKS_VERSION, } from '../../services/embeddings/memory-chunks'; import { EmbeddingRouter } from '../../services/embeddings/router'; import { getVettedEmbeddingModel } from '../../services/embeddings/vetted-models'; +import { computeChronologyAwareBoost } from '../../services/memory-dreaming'; +import { env } from '../../config/env'; import type { Memory, MemoryCreateInput, + MemorySearchChunkType, MemoryRow, + MemorySemanticQueryStrategy, MemorySearchOptions, MemoryHistory, MemoryHistoryRow, @@ -42,6 +50,8 @@ export interface RecallCandidate { memory: Memory; semanticScore?: number; textScore?: number; + matchedChunkType?: MemoryChunkType | null; + semanticEvidenceCount?: number; finalScore: number; } @@ -76,9 +86,94 @@ type SemanticChunkMatchRow = Omit & { similarity?: number; matched_chunk_index?: number | null; matched_chunk_text?: string | null; + matched_chunk_type?: string | null; }; const DAY_MS = 24 * 60 * 60 * 1000; +const EMBEDDING_PERSIST_RETRY_ATTEMPTS = 3; +const DERIVED_CHUNK_TYPES: MemoryChunkType[] = [ + 'summary', + 'fact', + 'topic', + 'entity', + 'current_state', +]; +const CONTENT_CHUNK_TYPES: MemoryChunkType[] = ['content']; + +function toMemoryChunkTypes(chunkTypes?: MemorySearchChunkType[]): MemoryChunkType[] | undefined { + if (!chunkTypes || chunkTypes.length === 0) return undefined; + return chunkTypes as MemoryChunkType[]; +} + +function computeChunkTypeBoost(chunkType?: MemoryChunkType | null): number { + switch (chunkType) { + case 'fact': + return 0.08; + case 'topic': + return 0.05; + case 'entity': + return 0.04; + case 'summary': + return 0.03; + case 'current_state': + return 0.05; + default: + return 0; + } +} + +function mergeSemanticCandidateGroups( + candidateGroups: Array, + offset: number, + limit: number +): RecallCandidate[] { + const grouped = new Map(); + + for (const candidates of candidateGroups) { + for (const candidate of candidates || []) { + const existing = grouped.get(candidate.memory.id); + if (!existing || (candidate.finalScore ?? 0) > (existing.finalScore ?? 0)) { + grouped.set(candidate.memory.id, { + memory: candidate.memory, + semanticScore: Math.max(existing?.semanticScore ?? 0, candidate.semanticScore ?? 0), + matchedChunkType: + computeChunkTypeBoost(candidate.matchedChunkType) > + computeChunkTypeBoost(existing?.matchedChunkType) + ? candidate.matchedChunkType + : (existing?.matchedChunkType ?? candidate.matchedChunkType), + semanticEvidenceCount: (existing?.semanticEvidenceCount ?? 0) + 1, + finalScore: Math.max(existing?.finalScore ?? 0, candidate.finalScore ?? 0), + }); + continue; + } + + grouped.set(candidate.memory.id, { + ...existing, + semanticScore: Math.max(existing.semanticScore ?? 0, candidate.semanticScore ?? 0), + semanticEvidenceCount: (existing.semanticEvidenceCount ?? 0) + 1, + }); + } + } + + return Array.from(grouped.values()) + .sort( + (a, b) => + (b.finalScore ?? 0) - (a.finalScore ?? 0) || + b.memory.createdAt.getTime() - a.memory.createdAt.getTime() + ) + .slice(offset, offset + limit); +} + +function buildSemanticChunkPlans( + strategy: MemorySemanticQueryStrategy | undefined, + explicitChunkTypes: MemoryChunkType[] | undefined +): Array { + if (strategy === 'parallel-content-entity') { + return [['content'], ['entity']]; + } + + return [explicitChunkTypes]; +} function parseEmbeddingValue(value: MemoryRow['embedding'] | string | null): number[] | undefined { if (!value) return undefined; @@ -161,6 +256,50 @@ function computeFocusBoost(memory: Memory, focusText?: string): number { return 1 + Math.min(0.35, ratio * 0.35); } +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +function normalizeErrorDetails(error: unknown): Record { + if (error instanceof Error) { + return { + message: error.message, + name: error.name, + stack: error.stack, + }; + } + + if (error && typeof error === 'object') { + return error as Record; + } + + return { message: String(error) }; +} + +function summarizeChunkPersistenceContext(params: { + memory: Memory; + embeddedChunks: EmbeddedMemoryChunk[]; +}): Record { + const { memory, embeddedChunks } = params; + const primary = embeddedChunks[0]?.embedding; + + return { + memoryId: memory.id, + userId: memory.userId, + salience: memory.salience, + topicKey: memory.topicKey, + topicCount: memory.topics.length, + summaryLength: memory.summary?.length ?? 0, + contentLength: memory.content.length, + chunkCount: embeddedChunks.length, + chunkTypes: embeddedChunks.map((chunk) => chunk.chunkType), + chunkLengths: embeddedChunks.map((chunk) => chunk.text.length), + provider: primary?.provider, + model: primary?.model, + dimensions: primary?.dimensions, + }; +} + export function computeKnowledgeMemoryScore( memory: Memory, context: KnowledgeMemoryContext = {}, @@ -264,25 +403,31 @@ export class MemoryRepository { } if (recallMode === 'semantic') { - const semanticCandidates = await this.trySemanticRecallCandidates( - userId, - normalizedQuery, - options, - limit, - offset + const semanticChunkPlans = buildSemanticChunkPlans( + options.semanticQueryStrategy, + toMemoryChunkTypes(options.semanticChunkTypes) ); - return semanticCandidates?.map((c) => c.memory) || []; + const semanticGroups = await Promise.all( + semanticChunkPlans.map((chunkTypes) => + this.trySemanticRecallCandidates(userId, normalizedQuery, options, limit, 0, chunkTypes) + ) + ); + const semanticCandidates = mergeSemanticCandidateGroups(semanticGroups, offset, limit); + return semanticCandidates.map((c) => c.memory); } if (recallMode === 'auto') { - const semanticCandidates = await this.trySemanticRecallCandidates( - userId, - normalizedQuery, - options, - limit, - offset + const semanticChunkPlans = buildSemanticChunkPlans( + options.semanticQueryStrategy, + toMemoryChunkTypes(options.semanticChunkTypes) ); - if (semanticCandidates && semanticCandidates.length > 0) { + const semanticGroups = await Promise.all( + semanticChunkPlans.map((chunkTypes) => + this.trySemanticRecallCandidates(userId, normalizedQuery, options, limit, 0, chunkTypes) + ) + ); + const semanticCandidates = mergeSemanticCandidateGroups(semanticGroups, offset, limit); + if (semanticCandidates.length > 0) { return semanticCandidates.map((c) => c.memory); } return this.textRecall(userId, normalizedQuery, options, limit, offset); @@ -358,20 +503,63 @@ export class MemoryRepository { limit, (offset + limit) * Math.max(1, config.matchCountMultiplier) ); - - const [semanticCandidates, textCandidates] = await Promise.all([ - this.trySemanticRecallCandidates(userId, query, options, limit, offset), + const chunkStrategy = options.hybridChunkStrategy || 'default'; + const isMultiViewRouter = chunkStrategy === 'multi-view'; + const explicitChunkTypes = toMemoryChunkTypes(options.semanticChunkTypes); + const applyChunkTypeBoosts = isMultiViewRouter && options.applyChunkTypeBoosts !== false; + const applyMultiViewBoost = isMultiViewRouter && options.applyMultiViewBoost !== false; + const applyChronologyBoost = isMultiViewRouter && options.applyChronologyBoost !== false; + + const semanticRequests = isMultiViewRouter + ? [ + this.trySemanticRecallCandidates( + userId, + query, + options, + candidatePool, + 0, + DERIVED_CHUNK_TYPES + ), + this.trySemanticRecallCandidates( + userId, + query, + options, + candidatePool, + 0, + CONTENT_CHUNK_TYPES + ), + ] + : buildSemanticChunkPlans( + options.semanticQueryStrategy, + explicitChunkTypes || + (chunkStrategy === 'content-only' + ? CONTENT_CHUNK_TYPES + : chunkStrategy === 'derived-only' + ? DERIVED_CHUNK_TYPES + : undefined) + ).map((chunkTypes) => + this.trySemanticRecallCandidates(userId, query, options, candidatePool, 0, chunkTypes) + ); + + const [semanticCandidateGroups, textCandidates] = await Promise.all([ + Promise.all(semanticRequests), this.textRecallCandidates(userId, query, options, candidatePool, 0), ]); const byId = new Map(); - for (const candidate of semanticCandidates || []) { + for (const candidate of semanticCandidateGroups.flatMap((candidates) => candidates || [])) { const existing = byId.get(candidate.memory.id); byId.set(candidate.memory.id, { memory: candidate.memory, - semanticScore: candidate.semanticScore, + semanticScore: Math.max(existing?.semanticScore ?? 0, candidate.semanticScore ?? 0), textScore: existing?.textScore, + matchedChunkType: + computeChunkTypeBoost(candidate.matchedChunkType) > + computeChunkTypeBoost(existing?.matchedChunkType) + ? candidate.matchedChunkType + : (existing?.matchedChunkType ?? candidate.matchedChunkType), + semanticEvidenceCount: (existing?.semanticEvidenceCount ?? 0) + 1, finalScore: 0, }); } @@ -382,14 +570,41 @@ export class MemoryRepository { memory: candidate.memory, semanticScore: existing?.semanticScore, textScore: candidate.textScore, + matchedChunkType: existing?.matchedChunkType, + semanticEvidenceCount: existing?.semanticEvidenceCount, finalScore: 0, }); } - const merged = Array.from(byId.values()).map((candidate) => ({ - ...candidate, - finalScore: this.computeHybridScore(candidate.semanticScore, candidate.textScore), - })); + const chronologyWindow = this.buildChronologyWindow( + Array.from(byId.values()).map((c) => c.memory) + ); + const merged = Array.from(byId.values()).map((candidate) => { + const chronologyBoost = chronologyWindow + ? computeChronologyAwareBoost({ + query, + memory: candidate.memory, + minCreatedAt: chronologyWindow.min, + maxCreatedAt: chronologyWindow.max, + }) + : 0; + + return { + ...candidate, + finalScore: this.computeHybridScore( + candidate.semanticScore, + candidate.textScore, + candidate.matchedChunkType, + candidate.semanticEvidenceCount, + chronologyBoost, + { + applyChunkTypeBoosts, + applyMultiViewBoost, + applyChronologyBoost, + } + ), + }; + }); merged.sort( (a, b) => @@ -399,11 +614,43 @@ export class MemoryRepository { return merged.slice(offset, offset + limit); } - private computeHybridScore(semanticScore?: number, textScore?: number): number { + private computeHybridScore( + semanticScore?: number, + textScore?: number, + matchedChunkType?: MemoryChunkType | null, + semanticEvidenceCount?: number, + chronologyBoost = 0, + options: { + applyChunkTypeBoosts?: boolean; + applyMultiViewBoost?: boolean; + applyChronologyBoost?: boolean; + } = {} + ): number { const s = semanticScore ?? 0; const t = textScore ?? 0; + const multiViewBoost = + options.applyMultiViewBoost === false + ? 0 + : Math.max(0, (semanticEvidenceCount ?? 1) - 1) * 0.03; + const chunkTypeBoost = + options.applyChunkTypeBoosts === false ? 0 : computeChunkTypeBoost(matchedChunkType); + const chronologyScore = options.applyChronologyBoost === false ? 0 : chronologyBoost; // Blend with heavier semantic weighting, but allow lexical key matches to lift ranking. - return s * 0.7 + t * 0.3; + return Math.max( + 0, + Math.min(1, s * 0.7 + t * 0.3 + chunkTypeBoost + multiViewBoost + chronologyScore) + ); + } + + private buildChronologyWindow(memories: Memory[]): { min: Date; max: Date } | null { + if (memories.length === 0) return null; + let min = memories[0].createdAt; + let max = memories[0].createdAt; + for (const memory of memories) { + if (memory.createdAt.getTime() < min.getTime()) min = memory.createdAt; + if (memory.createdAt.getTime() > max.getTime()) max = memory.createdAt; + } + return { min, max }; } private buildTextScore(query: string, memory: Memory): number { @@ -551,7 +798,8 @@ export class MemoryRepository { query: string, options: MemorySearchOptions, limit: number, - offset: number + offset: number, + chunkTypes?: MemoryChunkType[] ): Promise { const queryEmbedding = await this.embeddingRouter.embedQuery(query); if (!queryEmbedding) return null; @@ -583,6 +831,7 @@ export class MemoryRepository { p_agent_id: options.agentId, p_include_shared: options.includeShared !== false, p_include_expired: options.includeExpired === true, + p_chunk_types: chunkTypes && chunkTypes.length > 0 ? chunkTypes : undefined, }; const rpcClient = this.supabase as unknown as MatchMemoriesRpcClient; @@ -613,14 +862,25 @@ export class MemoryRepository { continue; } + const matchedChunkType = + row.matched_chunk_type && + ['summary', 'fact', 'topic', 'entity', 'content'].includes(row.matched_chunk_type) + ? (row.matched_chunk_type as MemoryChunkType) + : inferChunkTypeFromMetadata(row.matched_chunk_index, memory.metadata); const semanticScore = Math.max(0, Math.min(1, row.similarity ?? 0)); + const boostedSemanticScore = Math.min( + 1, + semanticScore + + (options.applyChunkTypeBoosts === false ? 0 : computeChunkTypeBoost(matchedChunkType)) + ); const existing = grouped.get(memory.id); - if (!existing || semanticScore > (existing.semanticScore ?? 0)) { + if (!existing || boostedSemanticScore > existing.finalScore) { grouped.set(memory.id, { memory, semanticScore, - finalScore: semanticScore, + matchedChunkType, + finalScore: boostedSemanticScore, }); } } @@ -628,8 +888,7 @@ export class MemoryRepository { return Array.from(grouped.values()) .sort( (a, b) => - (b.semanticScore ?? 0) - (a.semanticScore ?? 0) || - b.memory.createdAt.getTime() - a.memory.createdAt.getTime() + b.finalScore - a.finalScore || b.memory.createdAt.getTime() - a.memory.createdAt.getTime() ) .slice(offset, offset + limit); } @@ -692,10 +951,20 @@ export class MemoryRepository { const config = this.embeddingRouter.getRuntimeConfig(); const vettedModel = getVettedEmbeddingModel(config.provider, config.model); + const llmExtractions = + memory.metadata && typeof memory.metadata === 'object' && 'llm_extractions' in memory.metadata + ? (memory.metadata.llm_extractions as Record) + : null; const chunks = buildMemoryEmbeddingChunks({ summary: input.summary, content: input.content, + topicKey: input.topicKey, + topics: input.topics, + source: input.source, + salience: input.salience, model: vettedModel, + llmExtractions, + extractionMode: env.MEMORY_EXTRACTION_MODE, }); if (chunks.length === 0) return; @@ -715,49 +984,119 @@ export class MemoryRepository { userId: memory.userId, chunks: embeddedChunks.map(({ chunk, embedding }) => ({ ...chunk, embedding })), }); + const chunkContext = summarizeChunkPersistenceContext({ + memory, + embeddedChunks: embeddedChunks.map(({ chunk, embedding }) => ({ ...chunk, embedding })), + }); - const { error: chunkError } = await this.supabase - .from('memory_embedding_chunks') - .upsert(chunkRows, { - onConflict: 'memory_id,chunk_index', - }); + let chunkErrorDetails: Record | null = null; + for (let attempt = 1; attempt <= EMBEDDING_PERSIST_RETRY_ATTEMPTS; attempt += 1) { + const { error: chunkError } = await this.supabase + .from('memory_embedding_chunks') + .upsert(chunkRows, { + onConflict: 'memory_id,chunk_index', + }); + + if (!chunkError) { + if (attempt > 1) { + logger.info('Memory embedding chunk persistence recovered after retry', { + ...chunkContext, + stage: 'chunk_upsert', + attempt, + attempts: EMBEDDING_PERSIST_RETRY_ATTEMPTS, + }); + } + chunkErrorDetails = null; + break; + } - if (chunkError) { + chunkErrorDetails = normalizeErrorDetails(chunkError); logger.warn('Failed to persist memory embedding chunks', { - memoryId: memory.id, - error: chunkError.message, + ...chunkContext, + stage: 'chunk_upsert', + attempt, + retrying: attempt < EMBEDDING_PERSIST_RETRY_ATTEMPTS, + error: chunkErrorDetails, + }); + + if (attempt < EMBEDDING_PERSIST_RETRY_ATTEMPTS) { + await sleep(200 * attempt); + } + } + + if (chunkErrorDetails) { + logger.error('Giving up on memory embedding chunk persistence', { + ...chunkContext, + stage: 'chunk_upsert', + attempts: EMBEDDING_PERSIST_RETRY_ATTEMPTS, + error: chunkErrorDetails, }); return; } - const { error } = await this.supabase - .from('memories') - .update({ - embedding: formatVectorLiteral(primaryEmbedding.vector), - embedding_chunks_version: MEMORY_EMBEDDING_CHUNKS_VERSION, - embedding_chunk_count: embeddedChunks.length, - metadata: { - ...buildChunkMetadataUpdate({ - provider: primaryEmbedding.provider, - model: primaryEmbedding.model, - chunkCount: embeddedChunks.length, - existingMetadata: memory.metadata || {}, - }), - embedding: { - provider: primaryEmbedding.provider, - model: primaryEmbedding.model, - dimensions: primaryEmbedding.dimensions, - updatedAt: new Date().toISOString(), - }, - } as Database['public']['Tables']['memories']['Update']['metadata'], - }) - .eq('id', memory.id) - .eq('user_id', memory.userId); + const memoryUpdate: Database['public']['Tables']['memories']['Update'] = { + embedding: formatVectorLiteral(primaryEmbedding.vector), + embedding_chunks_version: MEMORY_EMBEDDING_CHUNKS_VERSION, + embedding_chunk_count: embeddedChunks.length, + metadata: { + ...buildChunkMetadataUpdate({ + provider: primaryEmbedding.provider, + model: primaryEmbedding.model, + chunkCount: embeddedChunks.length, + viewCounts: countChunkViews(embeddedChunks.map(({ chunk }) => chunk)), + extractionMode: env.MEMORY_EXTRACTION_MODE, + existingMetadata: memory.metadata || {}, + }), + embedding: { + provider: primaryEmbedding.provider, + model: primaryEmbedding.model, + dimensions: primaryEmbedding.dimensions, + updatedAt: new Date().toISOString(), + }, + } as Database['public']['Tables']['memories']['Update']['metadata'], + }; - if (error) { - logger.warn('Failed to persist memory embedding', { - memoryId: memory.id, - error: error.message, + let memoryErrorDetails: Record | null = null; + for (let attempt = 1; attempt <= EMBEDDING_PERSIST_RETRY_ATTEMPTS; attempt += 1) { + const { error } = await this.supabase + .from('memories') + .update(memoryUpdate) + .eq('id', memory.id) + .eq('user_id', memory.userId); + + if (!error) { + if (attempt > 1) { + logger.info('Memory embedding metadata persistence recovered after retry', { + ...chunkContext, + stage: 'memory_update', + attempt, + attempts: EMBEDDING_PERSIST_RETRY_ATTEMPTS, + }); + } + memoryErrorDetails = null; + break; + } + + memoryErrorDetails = normalizeErrorDetails(error); + logger.warn('Failed to persist memory embedding metadata', { + ...chunkContext, + stage: 'memory_update', + attempt, + retrying: attempt < EMBEDDING_PERSIST_RETRY_ATTEMPTS, + error: memoryErrorDetails, + }); + + if (attempt < EMBEDDING_PERSIST_RETRY_ATTEMPTS) { + await sleep(200 * attempt); + } + } + + if (memoryErrorDetails) { + logger.error('Giving up on memory embedding metadata persistence', { + ...chunkContext, + stage: 'memory_update', + attempts: EMBEDDING_PERSIST_RETRY_ATTEMPTS, + error: memoryErrorDetails, }); return; } @@ -768,6 +1107,8 @@ export class MemoryRepository { provider: primaryEmbedding.provider, model: primaryEmbedding.model, chunkCount: embeddedChunks.length, + viewCounts: countChunkViews(embeddedChunks.map(({ chunk }) => chunk)), + extractionMode: env.MEMORY_EXTRACTION_MODE, existingMetadata: memory.metadata || {}, }), embedding: { diff --git a/packages/api/src/scripts/audit-memory-llm-extractions.ts b/packages/api/src/scripts/audit-memory-llm-extractions.ts new file mode 100644 index 00000000..3dce5795 --- /dev/null +++ b/packages/api/src/scripts/audit-memory-llm-extractions.ts @@ -0,0 +1,755 @@ +import { createSupabaseClient } from '../data/supabase/client'; +import type { Database } from '../data/supabase/types'; +import { + buildDurableFactEmbeddingTexts, + buildEntityEmbeddingTexts, + buildSummaryEmbeddingTexts, + normalizeMemoryExtractions, + type MemoryExtractions, +} from '../services/memory-llm-extraction'; +import { env } from '../config/env'; +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; + +type MemoryRow = Pick< + Database['public']['Tables']['memories']['Row'], + 'id' | 'content' | 'summary' | 'topic_key' | 'topics' | 'metadata' +>; + +type CaseRole = 'target' | 'distractor' | 'unknown'; + +interface SeedCase { + caseId: string; + topic: string; + targetMemoryIds: string[]; + distractorMemoryIds: string[]; +} + +interface SeedFile { + seedId: string; + seededCases: Record; +} + +interface LongMemCase { + question_id: string; + question_type: string; + question: string; + answer: string | number; + answer_session_ids: string[]; +} + +interface RoleInfo { + caseId: string; + role: CaseRole; +} + +interface AuditedMemory { + row: MemoryRow; + extraction: MemoryExtractions | null; + role: RoleInfo; +} + +interface CaseCoverage { + caseId: string; + questionType: string; + question: string; + answer: string; + targetMemoryIds: string[]; + targetContentHasAnswer: boolean; + targetExtractionInputHasAnswer: boolean; + targetOverInputLimit: boolean; + entityHasAnswer: boolean; + durableFactHasAnswer: boolean; + summaryHasAnswer: boolean; + derivedHasAnswer: boolean; + maxDerivedAnswerTokenCoverage: number; + sourceSnippet: string; + derivedSnippet: string; +} + +interface AuditSummary { + generatedAt: string; + topic: string; + seedId: string | null; + totalMemories: number; + roleCounts: Record; + completeExtractionCount: number; + missingExtractionCount: number; + normalizedPresence: { + entity: number; + durableFact: number; + summary: number; + rawEntity: number; + rawDurableFact: number; + rawSummary: number; + }; + extractionCounts: { + entityItems: number; + durableFactItems: number; + summaryKeyPoints: number; + emptyEntityMemories: number; + emptyDurableFactMemories: number; + }; + rawOverflowCounts: { + entity: number; + durableFact: number; + summaryKeyPoints: number; + }; + labelLeakCounts: { + benchmarkTerms: number; + targetDistractorTerms: number; + }; + contentLimit: { + maxInputChars: number; + overLimitMemories: number; + overLimitTargets: number; + }; + entityTypeCounts: Record; + durableFactCategoryCounts: Record; + extractionVersionCounts: Record; + extractionProviderCounts: Record; + answerCoverage: { + cases: number; + targetContentHasAnswer: number; + targetExtractionInputHasAnswer: number; + entityHasAnswer: number; + durableFactHasAnswer: number; + summaryHasAnswer: number; + derivedHasAnswer: number; + derivedMissWhenTargetContentHasAnswer: number; + derivedMissWhenTargetExtractionInputHasAnswer: number; + }; +} + +function parsePositiveInt(raw: string | undefined, defaultValue: number): number { + if (!raw) return defaultValue; + const parsed = Number(raw); + return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : defaultValue; +} + +function parseList(raw: string | undefined): string[] { + if (!raw) return []; + return raw + .split(/[,\n]/) + .map((item) => item.trim()) + .filter(Boolean); +} + +function asRecord(value: unknown): Record { + return value && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function normalizeText(text: string | number): string { + return String(text) + .toLowerCase() + .normalize('NFKD') + .replace(/[\u0300-\u036f]/g, '') + .replace(/[^a-z0-9]+/g, ' ') + .replace(/\s+/g, ' ') + .trim(); +} + +const ANSWER_STOPWORDS = new Set([ + 'a', + 'an', + 'and', + 'are', + 'as', + 'at', + 'be', + 'by', + 'for', + 'from', + 'had', + 'has', + 'have', + 'he', + 'her', + 'his', + 'i', + 'in', + 'is', + 'it', + 'my', + 'of', + 'on', + 'or', + 'our', + 'she', + 'that', + 'the', + 'their', + 'they', + 'to', + 'was', + 'were', + 'with', + 'you', +]); + +function answerTokenCoverage(text: string, answer: string | number): number { + const normalizedText = normalizeText(text); + const normalizedAnswer = normalizeText(answer); + if (!normalizedAnswer) return 0; + if (normalizedText.includes(normalizedAnswer)) return 1; + const tokens = normalizedAnswer + .split(' ') + .filter((token) => token.length >= 3 || /^\d+$/.test(token)) + .filter((token) => !ANSWER_STOPWORDS.has(token)); + if (tokens.length === 0) return 0; + const hitCount = tokens.filter((token) => normalizedText.includes(token)).length; + return hitCount / tokens.length; +} + +function hasAnswer(text: string, answer: string | number): boolean { + const coverage = answerTokenCoverage(text, answer); + if (coverage >= 0.8) return true; + const normalizedAnswer = normalizeText(answer); + return normalizedAnswer.length >= 4 && normalizeText(text).includes(normalizedAnswer); +} + +function compact(text: string, maxChars = 500): string { + const oneLine = text.replace(/\s+/g, ' ').trim(); + if (oneLine.length <= maxChars) return oneLine; + return `${oneLine.slice(0, Math.max(0, maxChars - 1)).trimEnd()}…`; +} + +function snippetAroundAnswer(text: string, answer: string | number, maxChars = 700): string { + const normalizedAnswer = normalizeText(answer); + const normalizedText = normalizeText(text); + const index = normalizedAnswer ? normalizedText.indexOf(normalizedAnswer) : -1; + if (index < 0) return compact(text, maxChars); + + // Indexes after normalization are approximate; use a proportional source slice. + const ratio = index / Math.max(normalizedText.length, 1); + const sourceIndex = Math.floor(text.length * ratio); + const start = Math.max(0, sourceIndex - Math.floor(maxChars / 2)); + return compact(text.slice(start, start + maxChars), maxChars); +} + +function extractionText( + extraction: MemoryExtractions | null, + kind?: 'entity' | 'durable' | 'summary' +) { + if (!extraction) return ''; + const parts: string[] = []; + if ((!kind || kind === 'entity') && extraction.entity) { + parts.push(buildEntityEmbeddingTexts(extraction.entity).join('\n')); + } + if ((!kind || kind === 'durable') && extraction.durable_fact) { + parts.push(buildDurableFactEmbeddingTexts(extraction.durable_fact).join('\n')); + } + if ((!kind || kind === 'summary') && extraction.summary) { + parts.push(buildSummaryEmbeddingTexts(extraction.summary).join('\n')); + } + return parts.join('\n'); +} + +function rawArrayLength(raw: unknown, key: 'entities' | 'durableFacts' | 'keyPoints'): number { + const value = asRecord(raw)[key]; + return Array.isArray(value) ? value.length : 0; +} + +async function loadJsonFile(path: string): Promise { + return JSON.parse(await readFile(path, 'utf8')) as T; +} + +async function loadAllMemories(params: { + userId: string; + topic: string; + pageSize: number; +}): Promise { + const supabase = createSupabaseClient(); + const rows: MemoryRow[] = []; + let offset = 0; + + while (true) { + const { data, error } = await supabase + .from('memories') + .select('id,content,summary,topic_key,topics,metadata') + .eq('user_id', params.userId) + .contains('topics', [params.topic]) + .order('created_at', { ascending: true }) + .range(offset, offset + params.pageSize - 1); + + if (error) throw new Error(`Failed to load memories: ${error.message}`); + const page = (data || []) as MemoryRow[]; + rows.push(...page); + if (page.length < params.pageSize) break; + offset += params.pageSize; + } + + return rows; +} + +function buildRoleMap(seed: SeedFile | null): Map { + const roleByMemoryId = new Map(); + if (!seed) return roleByMemoryId; + + for (const [caseId, seededCase] of Object.entries(seed.seededCases)) { + for (const memoryId of seededCase.targetMemoryIds || []) { + roleByMemoryId.set(memoryId, { caseId, role: 'target' }); + } + for (const memoryId of seededCase.distractorMemoryIds || []) { + roleByMemoryId.set(memoryId, { caseId, role: 'distractor' }); + } + } + + return roleByMemoryId; +} + +function increment(counter: Record, key: string) { + counter[key] = (counter[key] || 0) + 1; +} + +function auditCaseCoverage(params: { + seed: SeedFile; + dataset: LongMemCase[]; + memoryById: Map; + caseIds?: Set; +}): CaseCoverage[] { + const datasetById = new Map(params.dataset.map((item) => [item.question_id, item])); + const cases: CaseCoverage[] = []; + + for (const [caseId, seededCase] of Object.entries(params.seed.seededCases)) { + if (params.caseIds && !params.caseIds.has(caseId)) continue; + const benchmarkCase = datasetById.get(caseId); + if (!benchmarkCase) continue; + const targetMemories = seededCase.targetMemoryIds + .map((memoryId) => params.memoryById.get(memoryId)) + .filter((item): item is AuditedMemory => Boolean(item)); + const targetContent = targetMemories.map((item) => item.row.content).join('\n\n'); + const targetExtractionInput = targetMemories + .map((item) => + [item.row.summary || '', item.row.content.slice(0, env.MEMORY_LLM_MAX_INPUT_CHARS)].join( + '\n' + ) + ) + .join('\n\n'); + const targetOverInputLimit = targetMemories.some( + (item) => item.row.content.length > env.MEMORY_LLM_MAX_INPUT_CHARS + ); + const entityText = targetMemories + .map((item) => extractionText(item.extraction, 'entity')) + .join('\n'); + const durableText = targetMemories + .map((item) => extractionText(item.extraction, 'durable')) + .join('\n'); + const summaryText = targetMemories + .map((item) => extractionText(item.extraction, 'summary')) + .join('\n'); + const derivedText = [entityText, durableText, summaryText].join('\n'); + + cases.push({ + caseId, + questionType: benchmarkCase.question_type, + question: benchmarkCase.question, + answer: String(benchmarkCase.answer), + targetMemoryIds: seededCase.targetMemoryIds, + targetContentHasAnswer: hasAnswer(targetContent, benchmarkCase.answer), + targetExtractionInputHasAnswer: hasAnswer(targetExtractionInput, benchmarkCase.answer), + targetOverInputLimit, + entityHasAnswer: hasAnswer(entityText, benchmarkCase.answer), + durableFactHasAnswer: hasAnswer(durableText, benchmarkCase.answer), + summaryHasAnswer: hasAnswer(summaryText, benchmarkCase.answer), + derivedHasAnswer: hasAnswer(derivedText, benchmarkCase.answer), + maxDerivedAnswerTokenCoverage: answerTokenCoverage(derivedText, benchmarkCase.answer), + sourceSnippet: snippetAroundAnswer(targetContent, benchmarkCase.answer), + derivedSnippet: compact(derivedText, 900), + }); + } + + return cases; +} + +function summarizeAudits(params: { + topic: string; + seed: SeedFile | null; + audited: AuditedMemory[]; + caseCoverage: CaseCoverage[]; +}): AuditSummary { + const roleCounts: Record = { target: 0, distractor: 0, unknown: 0 }; + const entityTypeCounts: Record = {}; + const durableFactCategoryCounts: Record = {}; + const extractionVersionCounts: Record = {}; + const extractionProviderCounts: Record = {}; + const normalizedPresence = { + entity: 0, + durableFact: 0, + summary: 0, + rawEntity: 0, + rawDurableFact: 0, + rawSummary: 0, + }; + const extractionCounts = { + entityItems: 0, + durableFactItems: 0, + summaryKeyPoints: 0, + emptyEntityMemories: 0, + emptyDurableFactMemories: 0, + }; + const rawOverflowCounts = { entity: 0, durableFact: 0, summaryKeyPoints: 0 }; + const labelLeakCounts = { benchmarkTerms: 0, targetDistractorTerms: 0 }; + const contentLimit = { + maxInputChars: env.MEMORY_LLM_MAX_INPUT_CHARS, + overLimitMemories: 0, + overLimitTargets: 0, + }; + + let completeExtractionCount = 0; + let missingExtractionCount = 0; + + for (const item of params.audited) { + roleCounts[item.role.role] += 1; + if (item.row.content.length > env.MEMORY_LLM_MAX_INPUT_CHARS) { + contentLimit.overLimitMemories += 1; + if (item.role.role === 'target') contentLimit.overLimitTargets += 1; + } + + const extraction = item.extraction; + if (!extraction) { + missingExtractionCount += 1; + continue; + } + const hasAllKinds = Boolean(extraction.entity && extraction.durable_fact && extraction.summary); + increment(extractionVersionCounts, String(extraction.version)); + increment(extractionProviderCounts, extraction.provider); + if ( + hasAllKinds && + extraction.raw?.entity && + extraction.raw?.durable_fact && + extraction.raw?.summary + ) { + completeExtractionCount += 1; + } + + if (extraction.entity) { + normalizedPresence.entity += 1; + extractionCounts.entityItems += extraction.entity.entities.length; + if (extraction.entity.entities.length === 0) extractionCounts.emptyEntityMemories += 1; + for (const entity of extraction.entity.entities) + increment(entityTypeCounts, entity.entityType); + } + if (extraction.durable_fact) { + normalizedPresence.durableFact += 1; + extractionCounts.durableFactItems += extraction.durable_fact.durableFacts.length; + if (extraction.durable_fact.durableFacts.length === 0) { + extractionCounts.emptyDurableFactMemories += 1; + } + for (const fact of extraction.durable_fact.durableFacts) { + increment(durableFactCategoryCounts, fact.category); + } + } + if (extraction.summary) { + normalizedPresence.summary += 1; + extractionCounts.summaryKeyPoints += extraction.summary.keyPoints.length; + } + + if (extraction.raw?.entity) { + normalizedPresence.rawEntity += 1; + if ( + rawArrayLength(extraction.raw.entity, 'entities') > + (extraction.entity?.entities.length || 0) + ) { + rawOverflowCounts.entity += 1; + } + } + if (extraction.raw?.durable_fact) { + normalizedPresence.rawDurableFact += 1; + if ( + rawArrayLength(extraction.raw.durable_fact, 'durableFacts') > + (extraction.durable_fact?.durableFacts.length || 0) + ) { + rawOverflowCounts.durableFact += 1; + } + } + if (extraction.raw?.summary) { + normalizedPresence.rawSummary += 1; + if ( + rawArrayLength(extraction.raw.summary, 'keyPoints') > + (extraction.summary?.keyPoints.length || 0) + ) { + rawOverflowCounts.summaryKeyPoints += 1; + } + } + + const text = normalizeText(extractionText(extraction)); + if (text.includes('benchmark')) labelLeakCounts.benchmarkTerms += 1; + if ( + /\b(benchmark target|benchmark distractor|target memory|distractor memory|target source|distractor source|target session|distractor session|target case|distractor case)\b/.test( + text + ) + ) { + labelLeakCounts.targetDistractorTerms += 1; + } + } + + return { + generatedAt: new Date().toISOString(), + topic: params.topic, + seedId: params.seed?.seedId || null, + totalMemories: params.audited.length, + roleCounts, + completeExtractionCount, + missingExtractionCount, + normalizedPresence, + extractionCounts, + rawOverflowCounts, + labelLeakCounts, + contentLimit, + entityTypeCounts, + durableFactCategoryCounts, + extractionVersionCounts, + extractionProviderCounts, + answerCoverage: { + cases: params.caseCoverage.length, + targetContentHasAnswer: params.caseCoverage.filter((item) => item.targetContentHasAnswer) + .length, + targetExtractionInputHasAnswer: params.caseCoverage.filter( + (item) => item.targetExtractionInputHasAnswer + ).length, + entityHasAnswer: params.caseCoverage.filter((item) => item.entityHasAnswer).length, + durableFactHasAnswer: params.caseCoverage.filter((item) => item.durableFactHasAnswer).length, + summaryHasAnswer: params.caseCoverage.filter((item) => item.summaryHasAnswer).length, + derivedHasAnswer: params.caseCoverage.filter((item) => item.derivedHasAnswer).length, + derivedMissWhenTargetContentHasAnswer: params.caseCoverage.filter( + (item) => item.targetContentHasAnswer && !item.derivedHasAnswer + ).length, + derivedMissWhenTargetExtractionInputHasAnswer: params.caseCoverage.filter( + (item) => item.targetExtractionInputHasAnswer && !item.derivedHasAnswer + ).length, + }, + }; +} + +function buildMarkdownReport(params: { + summary: AuditSummary; + misses: CaseCoverage[]; + lowCoverage: CaseCoverage[]; + samples: CaseCoverage[]; +}): string { + const lines: string[] = []; + const pct = (count: number, total: number) => + total === 0 ? '0.0%' : `${((count / total) * 100).toFixed(1)}%`; + + lines.push(`# Memory LLM Extraction Audit`); + lines.push(''); + lines.push(`Generated: ${params.summary.generatedAt}`); + lines.push(`Topic: \`${params.summary.topic}\``); + if (params.summary.seedId) lines.push(`Seed: \`${params.summary.seedId}\``); + lines.push(''); + lines.push('## Aggregate integrity'); + lines.push(''); + lines.push(`- Memories: ${params.summary.totalMemories}`); + lines.push( + `- Roles: target=${params.summary.roleCounts.target}, distractor=${params.summary.roleCounts.distractor}, unknown=${params.summary.roleCounts.unknown}` + ); + lines.push( + `- Complete normalized+raw extraction rows: ${params.summary.completeExtractionCount}` + ); + lines.push(`- Missing extraction rows: ${params.summary.missingExtractionCount}`); + lines.push( + `- Content over LLM input cap (${params.summary.contentLimit.maxInputChars} chars): ${params.summary.contentLimit.overLimitMemories} memories, including ${params.summary.contentLimit.overLimitTargets} targets` + ); + lines.push( + `- Label leakage: benchmark terms in ${params.summary.labelLeakCounts.benchmarkTerms}, target/distractor terms in ${params.summary.labelLeakCounts.targetDistractorTerms}` + ); + lines.push( + `- Raw overflow preserved: entity=${params.summary.rawOverflowCounts.entity}, durable_fact=${params.summary.rawOverflowCounts.durableFact}, summary_key_points=${params.summary.rawOverflowCounts.summaryKeyPoints}` + ); + lines.push( + `- Extraction versions: ${Object.entries(params.summary.extractionVersionCounts) + .map(([version, count]) => `v${version}=${count}`) + .join(', ')}` + ); + lines.push( + `- Providers: ${Object.entries(params.summary.extractionProviderCounts) + .map(([provider, count]) => `${provider}=${count}`) + .join(', ')}` + ); + lines.push(''); + lines.push('## Target-answer coverage heuristic'); + lines.push(''); + lines.push( + `- Target source text contains answer: ${params.summary.answerCoverage.targetContentHasAnswer}/${params.summary.answerCoverage.cases} (${pct(params.summary.answerCoverage.targetContentHasAnswer, params.summary.answerCoverage.cases)})` + ); + lines.push( + `- Extraction-visible target input contains answer: ${params.summary.answerCoverage.targetExtractionInputHasAnswer}/${params.summary.answerCoverage.cases} (${pct(params.summary.answerCoverage.targetExtractionInputHasAnswer, params.summary.answerCoverage.cases)})` + ); + lines.push( + `- Entity view contains answer: ${params.summary.answerCoverage.entityHasAnswer}/${params.summary.answerCoverage.cases} (${pct(params.summary.answerCoverage.entityHasAnswer, params.summary.answerCoverage.cases)})` + ); + lines.push( + `- Durable-fact view contains answer: ${params.summary.answerCoverage.durableFactHasAnswer}/${params.summary.answerCoverage.cases} (${pct(params.summary.answerCoverage.durableFactHasAnswer, params.summary.answerCoverage.cases)})` + ); + lines.push( + `- Summary view contains answer: ${params.summary.answerCoverage.summaryHasAnswer}/${params.summary.answerCoverage.cases} (${pct(params.summary.answerCoverage.summaryHasAnswer, params.summary.answerCoverage.cases)})` + ); + lines.push( + `- Any derived view contains answer: ${params.summary.answerCoverage.derivedHasAnswer}/${params.summary.answerCoverage.cases} (${pct(params.summary.answerCoverage.derivedHasAnswer, params.summary.answerCoverage.cases)})` + ); + lines.push( + `- Derived miss when target content has answer: ${params.summary.answerCoverage.derivedMissWhenTargetContentHasAnswer}` + ); + lines.push( + `- Derived miss when extraction-visible target input has answer: ${params.summary.answerCoverage.derivedMissWhenTargetExtractionInputHasAnswer}` + ); + lines.push(''); + lines.push('## Entity types'); + lines.push(''); + for (const [type, count] of Object.entries(params.summary.entityTypeCounts).sort( + (a, b) => b[1] - a[1] + )) { + lines.push(`- ${type}: ${count}`); + } + lines.push(''); + lines.push('## Durable fact categories'); + lines.push(''); + for (const [type, count] of Object.entries(params.summary.durableFactCategoryCounts).sort( + (a, b) => b[1] - a[1] + )) { + lines.push(`- ${type}: ${count}`); + } + lines.push(''); + lines.push('## Cases where target text had answer but derived views missed it'); + lines.push(''); + for (const item of params.misses) { + lines.push(`### ${item.caseId} — ${item.questionType}`); + lines.push(`- Q: ${item.question}`); + lines.push(`- A: ${item.answer}`); + lines.push(`- Extraction-visible input has answer: ${item.targetExtractionInputHasAnswer}`); + lines.push(`- Target over input cap: ${item.targetOverInputLimit}`); + lines.push(`- Derived answer token coverage: ${item.maxDerivedAnswerTokenCoverage.toFixed(2)}`); + lines.push(`- Source: ${item.sourceSnippet}`); + lines.push(`- Derived: ${item.derivedSnippet}`); + lines.push(''); + } + lines.push('## Low derived answer-coverage samples'); + lines.push(''); + for (const item of params.lowCoverage) { + lines.push(`### ${item.caseId} — ${item.questionType}`); + lines.push(`- Q: ${item.question}`); + lines.push(`- A: ${item.answer}`); + lines.push(`- Source has answer: ${item.targetContentHasAnswer}`); + lines.push(`- Extraction-visible input has answer: ${item.targetExtractionInputHasAnswer}`); + lines.push(`- Target over input cap: ${item.targetOverInputLimit}`); + lines.push( + `- Entity/fact/summary hit: ${item.entityHasAnswer}/${item.durableFactHasAnswer}/${item.summaryHasAnswer}` + ); + lines.push(`- Derived answer token coverage: ${item.maxDerivedAnswerTokenCoverage.toFixed(2)}`); + lines.push(`- Source: ${item.sourceSnippet}`); + lines.push(`- Derived: ${item.derivedSnippet}`); + lines.push(''); + } + lines.push('## Representative target samples'); + lines.push(''); + for (const item of params.samples) { + lines.push(`### ${item.caseId} — ${item.questionType}`); + lines.push(`- Q: ${item.question}`); + lines.push(`- A: ${item.answer}`); + lines.push(`- Target source has answer: ${item.targetContentHasAnswer}`); + lines.push(`- Extraction-visible input has answer: ${item.targetExtractionInputHasAnswer}`); + lines.push(`- Target over input cap: ${item.targetOverInputLimit}`); + lines.push( + `- Entity/fact/summary hit: ${item.entityHasAnswer}/${item.durableFactHasAnswer}/${item.summaryHasAnswer}` + ); + lines.push(`- Source: ${item.sourceSnippet}`); + lines.push(`- Derived: ${item.derivedSnippet}`); + lines.push(''); + } + + return `${lines.join('\n')}\n`; +} + +async function main() { + const userId = process.env.MEMORY_LLM_AUDIT_USER_ID || process.env.BENCHMARK_USER_ID; + if (!userId) throw new Error('MEMORY_LLM_AUDIT_USER_ID or BENCHMARK_USER_ID is required'); + + const topic = process.env.MEMORY_LLM_AUDIT_TOPIC; + if (!topic) throw new Error('MEMORY_LLM_AUDIT_TOPIC is required'); + + const pageSize = Math.min(parsePositiveInt(process.env.MEMORY_LLM_AUDIT_PAGE_SIZE, 1000), 1000); + const sampleLimit = parsePositiveInt(process.env.MEMORY_LLM_AUDIT_SAMPLE_LIMIT, 12); + const outputPath = + process.env.MEMORY_LLM_AUDIT_OUTPUT_PATH || + resolve( + process.cwd(), + 'output', + 'memory-extraction-audits', + `memory-llm-audit-${Date.now()}.json` + ); + const markdownPath = + process.env.MEMORY_LLM_AUDIT_MARKDOWN_PATH || outputPath.replace(/\.json$/i, '.md'); + const seedPath = process.env.MEMORY_LLM_AUDIT_SEED_PATH; + const datasetPath = process.env.MEMORY_LLM_AUDIT_DATASET_PATH; + const caseIds = new Set(parseList(process.env.MEMORY_LLM_AUDIT_CASE_IDS)); + + const [seed, dataset, rows] = await Promise.all([ + seedPath ? loadJsonFile(seedPath) : Promise.resolve(null), + datasetPath ? loadJsonFile(datasetPath) : Promise.resolve([]), + loadAllMemories({ userId, topic, pageSize }), + ]); + + const roleMap = buildRoleMap(seed); + const audited = rows.map((row): AuditedMemory => { + const metadata = asRecord(row.metadata); + const extraction = normalizeMemoryExtractions(metadata.llm_extractions); + return { + row, + extraction, + role: roleMap.get(row.id) || { caseId: 'unknown', role: 'unknown' }, + }; + }); + const filteredAudited = + caseIds.size > 0 + ? audited.filter((item) => item.role.caseId !== 'unknown' && caseIds.has(item.role.caseId)) + : audited; + const memoryById = new Map(audited.map((item) => [item.row.id, item])); + const caseCoverage = seed + ? auditCaseCoverage({ + seed, + dataset, + memoryById, + caseIds: caseIds.size > 0 ? caseIds : undefined, + }) + : []; + const summary = summarizeAudits({ topic, seed, audited: filteredAudited, caseCoverage }); + const misses = caseCoverage + .filter((item) => item.targetContentHasAnswer && !item.derivedHasAnswer) + .sort((a, b) => a.maxDerivedAnswerTokenCoverage - b.maxDerivedAnswerTokenCoverage) + .slice(0, sampleLimit); + const lowCoverage = caseCoverage + .filter((item) => !item.derivedHasAnswer) + .sort((a, b) => a.maxDerivedAnswerTokenCoverage - b.maxDerivedAnswerTokenCoverage) + .slice(0, sampleLimit); + const samples = caseCoverage.filter((item) => item.derivedHasAnswer).slice(0, sampleLimit); + + await mkdir(dirname(outputPath), { recursive: true }); + await writeFile( + outputPath, + `${JSON.stringify( + { + summary, + caseIds: [...caseIds], + misses, + lowCoverage, + samples, + caseCoverage, + }, + null, + 2 + )}\n` + ); + await writeFile(markdownPath, buildMarkdownReport({ summary, misses, lowCoverage, samples })); + + console.log(`[memory-llm-audit] output=${outputPath}`); + console.log(`[memory-llm-audit] markdown=${markdownPath}`); + console.log(JSON.stringify(summary, null, 2)); +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/packages/api/src/scripts/backfill-memory-embeddings.ts b/packages/api/src/scripts/backfill-memory-embeddings.ts index feb4a785..33ccd39d 100644 --- a/packages/api/src/scripts/backfill-memory-embeddings.ts +++ b/packages/api/src/scripts/backfill-memory-embeddings.ts @@ -4,15 +4,19 @@ import { buildChunkMetadataUpdate, buildChunkRows, buildMemoryEmbeddingChunks, + countChunkViews, formatVectorLiteral, MEMORY_EMBEDDING_CHUNKS_VERSION, } from '../services/embeddings/memory-chunks'; import { EmbeddingRouter } from '../services/embeddings/router'; import { getVettedEmbeddingModel } from '../services/embeddings/vetted-models'; +import { env } from '../config/env'; type MemoryRow = Database['public']['Tables']['memories']['Row']; const DEFAULT_BATCH_SIZE = 100; +const DEFAULT_PROGRESS_EVERY = 100; +const DEFAULT_ROW_ATTEMPTS = 3; function parseBoolean(raw: string | undefined, defaultValue: boolean): boolean { if (raw === undefined) return defaultValue; @@ -25,20 +29,43 @@ function parsePositiveInt(raw: string | undefined, defaultValue: number): number return Number.isFinite(parsed) && parsed > 0 ? parsed : defaultValue; } +function parseNonNegativeInt(raw: string | undefined, defaultValue: number): number { + if (!raw) return defaultValue; + const parsed = Number.parseInt(raw, 10); + return Number.isFinite(parsed) && parsed >= 0 ? parsed : defaultValue; +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + async function main() { - const userId = process.env.BACKFILL_MEMORY_USER_ID; + const userId = process.env.BACKFILL_MEMORY_USER_ID || process.env.BENCHMARK_USER_ID; if (!userId) { throw new Error( - 'BACKFILL_MEMORY_USER_ID is required. Example: BACKFILL_MEMORY_USER_ID= yarn backfill:memory-embeddings' + 'BACKFILL_MEMORY_USER_ID or BENCHMARK_USER_ID is required. Example: BACKFILL_MEMORY_USER_ID= yarn backfill:memory-embeddings' ); } const agentId = process.env.BACKFILL_MEMORY_AGENT_ID; + const topic = process.env.BACKFILL_MEMORY_TOPIC; + const memoryId = process.env.BACKFILL_MEMORY_ID; const batchSize = parsePositiveInt(process.env.BACKFILL_MEMORY_BATCH_SIZE, DEFAULT_BATCH_SIZE); + const startOffset = parseNonNegativeInt(process.env.BACKFILL_MEMORY_OFFSET, 0); const limit = process.env.BACKFILL_MEMORY_LIMIT ? parsePositiveInt(process.env.BACKFILL_MEMORY_LIMIT, batchSize) : null; + const progressEvery = parsePositiveInt( + process.env.BACKFILL_MEMORY_PROGRESS_EVERY, + DEFAULT_PROGRESS_EVERY + ); + const maxRowAttempts = parsePositiveInt( + process.env.BACKFILL_MEMORY_ROW_ATTEMPTS, + DEFAULT_ROW_ATTEMPTS + ); + const continueOnError = parseBoolean(process.env.BACKFILL_MEMORY_CONTINUE_ON_ERROR, false); const dryRun = parseBoolean(process.env.BACKFILL_MEMORY_DRY_RUN, false); + const force = parseBoolean(process.env.MEMORY_EMBEDDINGS_FORCE, false); const router = new EmbeddingRouter(); if (!router.isEnabled()) { @@ -54,7 +81,17 @@ async function main() { let processed = 0; let updated = 0; let skipped = 0; + let failed = 0; let scanned = 0; + let cursor = startOffset; + const failures: Array<{ memoryId: string; message: string }> = []; + + console.log( + `[memory-embedding-backfill] user=${userId} agent=${agentId || '*'} memory=${memoryId || '*'} topic=${topic || '*'} ` + + `offset=${startOffset} limit=${limit ?? 'all'} batchSize=${batchSize} force=${force} dryRun=${dryRun} ` + + `continueOnError=${continueOnError} rowAttempts=${maxRowAttempts} ` + + `mode=${env.MEMORY_EXTRACTION_MODE} chunkVersion=${MEMORY_EMBEDDING_CHUNKS_VERSION}` + ); while (limit === null || scanned < limit) { const remaining = limit === null ? batchSize : Math.min(batchSize, limit - scanned); @@ -63,16 +100,24 @@ async function main() { let query = supabase .from('memories') .select( - 'id,user_id,agent_id,content,summary,metadata,embedding,embedding_chunks_version,embedding_chunk_count' + 'id,user_id,agent_id,content,summary,topic_key,topics,source,salience,metadata,embedding,embedding_chunks_version,embedding_chunk_count' ) .eq('user_id', userId) .order('created_at', { ascending: true }) - .range(scanned, scanned + remaining - 1); + .range(cursor, cursor + remaining - 1); if (agentId) { query = query.eq('agent_id', agentId); } + if (topic?.trim()) { + query = query.contains('topics', [topic.trim()]); + } + + if (memoryId?.trim()) { + query = query.eq('id', memoryId.trim()); + } + const { data, error } = await query; if (error) { throw new Error(`Failed to fetch memories for backfill: ${error.message}`); @@ -85,6 +130,10 @@ async function main() { | 'agent_id' | 'content' | 'summary' + | 'topic_key' + | 'topics' + | 'source' + | 'salience' | 'metadata' | 'embedding' | 'embedding_chunks_version' @@ -93,116 +142,176 @@ async function main() { if (rows.length === 0) break; scanned += rows.length; + cursor += rows.length; for (const row of rows) { processed += 1; - const hasCurrentChunks = - row.embedding_chunks_version === MEMORY_EMBEDDING_CHUNKS_VERSION && - (row.embedding_chunk_count || 0) > 0; + let lastError: unknown = null; - if (hasCurrentChunks) { - skipped += 1; - continue; - } + for (let attempt = 1; attempt <= maxRowAttempts; attempt += 1) { + try { + const hasCurrentChunks = + row.embedding_chunks_version === MEMORY_EMBEDDING_CHUNKS_VERSION && + (row.embedding_chunk_count || 0) > 0; - const chunks = buildMemoryEmbeddingChunks({ - summary: row.summary, - content: row.content, - model: vettedModel, - }); - if (chunks.length === 0) { - skipped += 1; - continue; - } + if (hasCurrentChunks && !force) { + skipped += 1; + lastError = null; + break; + } - const embeddedChunks = []; - for (const chunk of chunks) { - const embedding = await router.embedDocument(chunk.text); - if (!embedding) continue; - embeddedChunks.push({ chunk, embedding }); - } + const chunks = buildMemoryEmbeddingChunks({ + summary: row.summary, + content: row.content, + topicKey: row.topic_key, + topics: row.topics, + source: row.source, + salience: row.salience, + model: vettedModel, + extractionMode: env.MEMORY_EXTRACTION_MODE, + llmExtractions: + row.metadata && typeof row.metadata === 'object' && 'llm_extractions' in row.metadata + ? (row.metadata.llm_extractions as Record) + : null, + }); + if (chunks.length === 0) { + skipped += 1; + lastError = null; + break; + } - if (embeddedChunks.length === 0) { - skipped += 1; - continue; - } + const embeddedChunks = []; + for (const chunk of chunks) { + const embedding = await router.embedDocument(chunk.text); + if (!embedding) continue; + embeddedChunks.push({ chunk, embedding }); + } - const primaryEmbedding = embeddedChunks[0].embedding; - const chunkRows = buildChunkRows({ - memoryId: row.id, - userId: row.user_id, - chunks: embeddedChunks.map(({ chunk, embedding }) => ({ ...chunk, embedding })), - }); + if (embeddedChunks.length === 0) { + skipped += 1; + lastError = null; + break; + } - if (dryRun) { - console.log( - `DRY RUN would backfill memory ${row.id} (${row.agent_id || 'shared'}) with ${embeddedChunks.length} chunk(s) via ${primaryEmbedding.provider}:${primaryEmbedding.model}` - ); - updated += 1; - continue; - } + const primaryEmbedding = embeddedChunks[0].embedding; + const chunkRows = buildChunkRows({ + memoryId: row.id, + userId: row.user_id, + chunks: embeddedChunks.map(({ chunk, embedding }) => ({ ...chunk, embedding })), + }); - const { error: chunkDeleteError } = await supabase - .from('memory_embedding_chunks') - .delete() - .eq('memory_id', row.id); + if (dryRun) { + console.log( + `DRY RUN would backfill memory ${row.id} (${row.agent_id || 'shared'}) with ${embeddedChunks.length} chunk(s) via ${primaryEmbedding.provider}:${primaryEmbedding.model}` + ); + updated += 1; + lastError = null; + break; + } - if (chunkDeleteError) { - throw new Error(`Failed to clear chunks for memory ${row.id}: ${chunkDeleteError.message}`); - } + const { error: chunkDeleteError } = await supabase + .from('memory_embedding_chunks') + .delete() + .eq('memory_id', row.id); - const { error: chunkUpsertError } = await supabase - .from('memory_embedding_chunks') - .upsert(chunkRows, { onConflict: 'memory_id,chunk_index' }); + if (chunkDeleteError) { + throw new Error( + `Failed to clear chunks for memory ${row.id}: ${chunkDeleteError.message}` + ); + } - if (chunkUpsertError) { - throw new Error( - `Failed to upsert chunks for memory ${row.id}: ${chunkUpsertError.message}` - ); + const { error: chunkUpsertError } = await supabase + .from('memory_embedding_chunks') + .upsert(chunkRows, { onConflict: 'memory_id,chunk_index' }); + + if (chunkUpsertError) { + throw new Error( + `Failed to upsert chunks for memory ${row.id}: ${chunkUpsertError.message}` + ); + } + + const { error: updateError } = await supabase + .from('memories') + .update({ + embedding: formatVectorLiteral(primaryEmbedding.vector), + embedding_chunks_version: MEMORY_EMBEDDING_CHUNKS_VERSION, + embedding_chunk_count: embeddedChunks.length, + metadata: { + ...buildChunkMetadataUpdate({ + provider: primaryEmbedding.provider, + model: primaryEmbedding.model, + chunkCount: embeddedChunks.length, + viewCounts: countChunkViews(embeddedChunks.map(({ chunk }) => chunk)), + extractionMode: env.MEMORY_EXTRACTION_MODE, + existingMetadata: ((row.metadata as Record | null) || + {}) as Record | null, + }), + embedding: { + provider: primaryEmbedding.provider, + model: primaryEmbedding.model, + dimensions: primaryEmbedding.dimensions, + updatedAt: new Date().toISOString(), + backfilled: true, + }, + } as Database['public']['Tables']['memories']['Update']['metadata'], + }) + .eq('id', row.id) + .eq('user_id', row.user_id); + + if (updateError) { + throw new Error(`Failed to update memory ${row.id}: ${updateError.message}`); + } + + updated += 1; + console.log( + `Backfilled memory ${row.id} (${row.agent_id || 'shared'}) with ${embeddedChunks.length} chunk(s) via ${primaryEmbedding.provider}:${primaryEmbedding.model}` + ); + lastError = null; + break; + } catch (error) { + lastError = error; + const message = error instanceof Error ? error.message : String(error); + console.warn( + `[memory-embedding-backfill] memory=${row.id} failed attempt=${attempt}/${maxRowAttempts} retrying=${attempt < maxRowAttempts}: ${message}` + ); + if (attempt < maxRowAttempts) { + await sleep(500 * attempt); + } + } } - const { error: updateError } = await supabase - .from('memories') - .update({ - embedding: formatVectorLiteral(primaryEmbedding.vector), - embedding_chunks_version: MEMORY_EMBEDDING_CHUNKS_VERSION, - embedding_chunk_count: embeddedChunks.length, - metadata: { - ...buildChunkMetadataUpdate({ - provider: primaryEmbedding.provider, - model: primaryEmbedding.model, - chunkCount: embeddedChunks.length, - existingMetadata: ((row.metadata as Record | null) || {}) as Record< - string, - unknown - > | null, - }), - embedding: { - provider: primaryEmbedding.provider, - model: primaryEmbedding.model, - dimensions: primaryEmbedding.dimensions, - updatedAt: new Date().toISOString(), - backfilled: true, - }, - } as Database['public']['Tables']['memories']['Update']['metadata'], - }) - .eq('id', row.id) - .eq('user_id', row.user_id); - - if (updateError) { - throw new Error(`Failed to update memory ${row.id}: ${updateError.message}`); + if (lastError) { + const message = lastError instanceof Error ? lastError.message : String(lastError); + failed += 1; + failures.push({ memoryId: row.id, message }); + if (!continueOnError) { + throw new Error(message); + } } - updated += 1; - console.log( - `Backfilled memory ${row.id} (${row.agent_id || 'shared'}) with ${embeddedChunks.length} chunk(s) via ${primaryEmbedding.provider}:${primaryEmbedding.model}` - ); + if (processed % progressEvery === 0) { + console.log( + `[memory-embedding-backfill] progress offset=${startOffset} cursor=${cursor} scanned=${scanned} processed=${processed} updated=${updated} skipped=${skipped} failed=${failed}` + ); + } } } console.log( - `Backfill complete. scanned=${scanned} processed=${processed} updated=${updated} skipped=${skipped} dryRun=${dryRun}` + `[memory-embedding-backfill] complete offset=${startOffset} scanned=${scanned} processed=${processed} updated=${updated} skipped=${skipped} failed=${failed} dryRun=${dryRun}` ); + if (failures.length > 0) { + console.log( + JSON.stringify( + { + failures: failures.slice(0, 20), + failureCount: failures.length, + }, + null, + 2 + ) + ); + } } main().catch((error) => { diff --git a/packages/api/src/scripts/benchmark-memory-recall.ts b/packages/api/src/scripts/benchmark-memory-recall.ts deleted file mode 100644 index 9ac363b3..00000000 --- a/packages/api/src/scripts/benchmark-memory-recall.ts +++ /dev/null @@ -1,295 +0,0 @@ -import { mkdir, writeFile } from 'node:fs/promises'; -import { dirname, resolve } from 'node:path'; -import { randomUUID } from 'node:crypto'; -import { createSupabaseClient } from '../data/supabase/client'; -import { MemoryRepository } from '../data/repositories/memory-repository'; -import { getBenchmarkDataset } from './benchmark-data/datasets'; -import { loadHfBenchmarkDataset } from './benchmark-data/hf-loader'; - -type RecallMode = 'text' | 'semantic' | 'hybrid' | 'auto'; - -interface CaseRun { - caseId: string; - query: string; - mode: RecallMode; - rank: number | null; - topSummaries: string[]; -} - -interface SummaryMetric { - mode: RecallMode; - cases: number; - recallAt1: number; - recallAt3: number; - recallAt5: number; - mrr: number; -} - -const TOP_K = 5; -const BENCHMARK_TOPIC = 'benchmark:memory-recall'; -const BENCHMARK_AGENT_ID = 'lumen'; -const DEFAULT_DATASET = 'internal-gold-v1'; -const MAX_CONTENT_CHARS = 1200; - -function parseModes(raw?: string): RecallMode[] { - if (!raw) return ['text', 'semantic', 'hybrid']; - const parsed = raw - .split(',') - .map((m) => m.trim()) - .filter(Boolean) as RecallMode[]; - return parsed.length > 0 ? parsed : ['text', 'semantic', 'hybrid']; -} - -function mean(values: number[]): number { - if (values.length === 0) return 0; - return values.reduce((acc, v) => acc + v, 0) / values.length; -} - -function round(value: number): number { - return Number(value.toFixed(4)); -} - -function parseBoolean(raw: string | undefined, defaultValue: boolean): boolean { - if (raw === undefined) return defaultValue; - return ['1', 'true', 'yes', 'on'].includes(raw.toLowerCase()); -} - -function clampContent(text: string): string { - if (text.length <= MAX_CONTENT_CHARS) return text; - return `${text.slice(0, MAX_CONTENT_CHARS)}...`; -} - -function buildSummaryMetrics(modes: RecallMode[], runs: CaseRun[]): SummaryMetric[] { - return modes.map((mode) => { - const modeRuns = runs.filter((r) => r.mode === mode); - const reciprocalRanks = modeRuns.map((r) => (r.rank ? 1 / r.rank : 0)); - const hitsAt1 = modeRuns.filter((r) => r.rank === 1).length / modeRuns.length; - const hitsAt3 = modeRuns.filter((r) => r.rank !== null && r.rank <= 3).length / modeRuns.length; - const hitsAt5 = modeRuns.filter((r) => r.rank !== null && r.rank <= 5).length / modeRuns.length; - - return { - mode, - cases: modeRuns.length, - recallAt1: round(hitsAt1), - recallAt3: round(hitsAt3), - recallAt5: round(hitsAt5), - mrr: round(mean(reciprocalRanks)), - }; - }); -} - -async function persistRun( - supabase: any, - params: { - runId: string; - userId: string; - dataset: string; - topK: number; - caseCount: number; - modes: RecallMode[]; - summary: SummaryMetric[]; - runs: CaseRun[]; - } -): Promise { - const { runId, userId, dataset, topK, caseCount, modes, summary, runs } = params; - - const modeRows = summary.map((metric) => ({ - run_id: runId, - mode: metric.mode, - cases: metric.cases, - recall_at_1: metric.recallAt1, - recall_at_3: metric.recallAt3, - recall_at_5: metric.recallAt5, - mrr: metric.mrr, - })); - - const caseRows = runs.map((run) => ({ - run_id: runId, - case_id: run.caseId, - mode: run.mode, - query: run.query, - rank: run.rank, - top_summaries: run.topSummaries, - })); - - const runRow = { - run_id: runId, - user_id: userId, - dataset, - provider: process.env.MEMORY_EMBEDDING_PROVIDER || 'default', - model: process.env.MEMORY_EMBEDDING_MODEL || 'default', - embeddings_enabled: parseBoolean(process.env.MEMORY_EMBEDDINGS_ENABLED, false), - top_k: topK, - case_count: caseCount, - modes, - summary, - metadata: { - benchmarkTopic: BENCHMARK_TOPIC, - benchmarkAgentId: BENCHMARK_AGENT_ID, - }, - }; - - const { error: runError } = await supabase.from('memory_recall_benchmark_runs').insert(runRow); - if (runError) throw new Error(`Failed to persist benchmark run: ${runError.message}`); - - const { error: metricsError } = await supabase - .from('memory_recall_benchmark_metrics') - .insert(modeRows); - if (metricsError) throw new Error(`Failed to persist benchmark metrics: ${metricsError.message}`); - - const { error: caseError } = await supabase - .from('memory_recall_benchmark_case_results') - .insert(caseRows); - if (caseError) throw new Error(`Failed to persist benchmark case results: ${caseError.message}`); -} - -async function writeJsonOutput(outputPath: string, payload: unknown): Promise { - await mkdir(dirname(outputPath), { recursive: true }); - await writeFile(outputPath, JSON.stringify(payload, null, 2), 'utf-8'); -} - -async function loadBenchmarkCases(dataset: string) { - if (dataset === 'hf') { - const hf = await loadHfBenchmarkDataset(); - return { cases: hf.cases, source: hf.source }; - } - - return { cases: getBenchmarkDataset(dataset), source: `builtin:${dataset}` }; -} - -async function main() { - const userId = process.env.BENCHMARK_USER_ID; - if (!userId) { - throw new Error( - 'BENCHMARK_USER_ID is required. Example: BENCHMARK_USER_ID= yarn benchmark:memory-recall' - ); - } - - const dataset = process.env.MEMORY_BENCHMARK_DATASET || DEFAULT_DATASET; - const { cases: benchmarkCases, source: datasetSource } = await loadBenchmarkCases(dataset); - const modes = parseModes(process.env.MEMORY_BENCHMARK_MODES); - const persistResults = parseBoolean(process.env.MEMORY_BENCHMARK_PERSIST, true); - const writeOutputFile = parseBoolean(process.env.MEMORY_BENCHMARK_WRITE_FILE, true); - - const runId = `membench-${Date.now()}-${randomUUID().slice(0, 8)}`; - const outputPath = - process.env.MEMORY_BENCHMARK_OUTPUT_PATH || - resolve(process.cwd(), 'output', 'memory-benchmarks', `${runId}.json`); - - const supabase = createSupabaseClient(); - const repo = new MemoryRepository(supabase); - const createdMemoryIds: string[] = []; - - const caseTargets: Record = {}; - const caseTopics: Record = {}; - - try { - for (const benchCase of benchmarkCases) { - const caseTopic = `${BENCHMARK_TOPIC}:${runId}:${benchCase.id}`; - caseTopics[benchCase.id] = [caseTopic]; - - const target = await repo.remember({ - userId, - agentId: BENCHMARK_AGENT_ID, - content: clampContent(benchCase.targetContent), - summary: `benchmark target ${benchCase.id}`, - source: 'observation', - salience: 'low', - topicKey: BENCHMARK_TOPIC, - topics: [BENCHMARK_TOPIC, caseTopic], - }); - createdMemoryIds.push(target.id); - caseTargets[benchCase.id] = target.id; - - for (let i = 0; i < benchCase.distractors.length; i += 1) { - const distractor = await repo.remember({ - userId, - agentId: BENCHMARK_AGENT_ID, - content: clampContent(benchCase.distractors[i]), - summary: `benchmark distractor ${benchCase.id} #${i + 1}`, - source: 'observation', - salience: 'low', - topicKey: BENCHMARK_TOPIC, - topics: [BENCHMARK_TOPIC, caseTopic], - }); - createdMemoryIds.push(distractor.id); - } - } - - const runs: CaseRun[] = []; - - for (const mode of modes) { - for (const benchCase of benchmarkCases) { - const results = await repo.recall(userId, benchCase.query, { - recallMode: mode, - limit: TOP_K, - agentId: BENCHMARK_AGENT_ID, - includeShared: true, - topics: caseTopics[benchCase.id], - }); - - const expectedId = caseTargets[benchCase.id]; - const rank = results.findIndex((m) => m.id === expectedId); - - runs.push({ - caseId: benchCase.id, - query: benchCase.query, - mode, - rank: rank >= 0 ? rank + 1 : null, - topSummaries: results.map((m) => m.summary || m.content.slice(0, 80)), - }); - } - } - - const summary = buildSummaryMetrics(modes, runs); - - if (persistResults) { - await persistRun(supabase as any, { - runId, - userId, - dataset, - topK: TOP_K, - caseCount: benchmarkCases.length, - modes, - summary, - runs, - }); - } - - const payload = { - runId, - settings: { - dataset, - model: process.env.MEMORY_EMBEDDING_MODEL || 'default', - provider: process.env.MEMORY_EMBEDDING_PROVIDER || 'default', - embeddingsEnabled: process.env.MEMORY_EMBEDDINGS_ENABLED || 'default', - topK: TOP_K, - benchmarkCases: benchmarkCases.length, - persistResults, - datasetSource, - }, - summary, - runs, - outputPath: writeOutputFile ? outputPath : null, - }; - - if (writeOutputFile) { - await writeJsonOutput(outputPath, payload); - } - - console.log(JSON.stringify(payload, null, 2)); - } finally { - for (const memoryId of createdMemoryIds) { - try { - await repo.forget(memoryId, userId); - } catch { - // best-effort cleanup - } - } - } -} - -main().catch((error) => { - console.error('[memory-benchmark] failed:', error); - process.exit(1); -}); diff --git a/packages/api/src/scripts/extract-memory-llm-views.ts b/packages/api/src/scripts/extract-memory-llm-views.ts new file mode 100644 index 00000000..34f02a65 --- /dev/null +++ b/packages/api/src/scripts/extract-memory-llm-views.ts @@ -0,0 +1,1191 @@ +import { createSupabaseClient } from '../data/supabase/client'; +import type { Database } from '../data/supabase/types'; +import { appendFile, mkdir, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import { setTimeout as sleep } from 'node:timers/promises'; +import { + buildBatchExtractionPrompt, + buildExtractionPrompt, + buildCurrentStateEmbeddingTexts, + buildDurableFactEmbeddingTexts, + buildEntityEmbeddingTexts, + buildSummaryEmbeddingTexts, + MemoryLlmExtractor, + normalizeMemoryExtractions, + MEMORY_EXTRACTION_VERSION, + batchMemoryExtractionResponseSchema, + coerceExtractionPayload, + type ExtractionKind, + type BatchMemoryExtractionSource, + type MemoryExtractionSource, + type MemoryExtractions, +} from '../services/memory-llm-extraction'; +import { ClaudeRunner, CodexRunner } from '../services/sessions'; +import type { ClaudeRunnerConfig, IRunner } from '../services/sessions/types'; +import { env } from '../config/env'; + +type MemoryRow = Database['public']['Tables']['memories']['Row']; +type ExtractableMemoryRow = Pick< + MemoryRow, + | 'id' + | 'user_id' + | 'content' + | 'summary' + | 'topic_key' + | 'topics' + | 'source' + | 'salience' + | 'metadata' +>; + +function parseBoolean(raw: string | undefined, defaultValue: boolean): boolean { + if (raw === undefined) return defaultValue; + return ['1', 'true', 'yes', 'on'].includes(raw.toLowerCase()); +} + +function parsePositiveInt(raw: string | undefined, defaultValue: number): number { + if (!raw) return defaultValue; + const parsed = Number(raw); + return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : defaultValue; +} + +function parseNonNegativeInt(raw: string | undefined, defaultValue: number): number { + if (!raw) return defaultValue; + const parsed = Number(raw); + return Number.isFinite(parsed) && parsed >= 0 ? Math.floor(parsed) : defaultValue; +} + +function buildExtractionEmbeddingTexts( + llmExtractions: MemoryExtractions +): Record { + return { + entity: llmExtractions.entity ? buildEntityEmbeddingTexts(llmExtractions.entity) : [], + durable_fact: llmExtractions.durable_fact + ? buildDurableFactEmbeddingTexts(llmExtractions.durable_fact) + : [], + summary: llmExtractions.summary ? buildSummaryEmbeddingTexts(llmExtractions.summary) : [], + current_state: llmExtractions.current_state + ? buildCurrentStateEmbeddingTexts(llmExtractions.current_state) + : [], + }; +} + +function hasAllEnabledKinds( + existing: MemoryExtractions | null, + enabledKinds: ExtractionKind[], + options: { requireRaw?: boolean; requireVersion?: number } = {} +): boolean { + if (!existing) return false; + if (options.requireVersion && existing.version !== options.requireVersion) return false; + return enabledKinds.every( + (kind) => Boolean(existing[kind]) && (!options.requireRaw || existing.raw?.[kind] !== undefined) + ); +} + +function mergeMemoryExtractions( + existing: MemoryExtractions | null, + next: MemoryExtractions, + options: { replaceKinds?: ExtractionKind[] } = {} +): MemoryExtractions { + const replaceKinds = new Set(options.replaceKinds || []); + const raw = + existing?.raw || next.raw + ? { + ...(existing?.raw || {}), + ...(next.raw || {}), + } + : undefined; + if (raw) { + for (const kind of replaceKinds) { + if (!next.raw || next.raw[kind] === undefined) { + delete raw[kind]; + } + } + } + return normalizeMemoryExtractions({ + ...(existing || {}), + ...next, + entity: replaceKinds.has('entity') ? next.entity : (next.entity ?? existing?.entity), + durable_fact: replaceKinds.has('durable_fact') + ? next.durable_fact + : (next.durable_fact ?? existing?.durable_fact), + summary: replaceKinds.has('summary') ? next.summary : (next.summary ?? existing?.summary), + current_state: replaceKinds.has('current_state') + ? next.current_state + : (next.current_state ?? existing?.current_state), + version: next.version, + provider: next.provider, + model: next.model, + extractedAt: next.extractedAt, + raw, + }) as MemoryExtractions; +} + +function normalizeExtractionHistory(value: unknown): MemoryExtractions[] { + if (!Array.isArray(value)) return []; + return value + .map((item) => normalizeMemoryExtractions(item)) + .filter((item): item is MemoryExtractions => Boolean(item)); +} + +function sameExtractionIdentity(left: MemoryExtractions, right: MemoryExtractions): boolean { + return ( + left.version === right.version && + left.provider === right.provider && + left.model === right.model && + left.extractedAt === right.extractedAt + ); +} + +function buildMetadataWithExtraction(params: { + metadata: Record; + existingExtractions: MemoryExtractions | null; + mergedExtractions: MemoryExtractions; + keepHistory: boolean; + historyLimit: number; +}): Record { + const { metadata, existingExtractions, mergedExtractions, keepHistory, historyLimit } = params; + if ( + !keepHistory || + !existingExtractions || + sameExtractionIdentity(existingExtractions, mergedExtractions) + ) { + return { + ...metadata, + llm_extractions: mergedExtractions, + }; + } + + const history = normalizeExtractionHistory(metadata.llm_extraction_versions); + const alreadyStored = history.some((item) => sameExtractionIdentity(item, existingExtractions)); + const nextHistory = alreadyStored ? history : [...history, existingExtractions]; + return { + ...metadata, + llm_extractions: mergedExtractions, + llm_extraction_versions: nextHistory.slice(-historyLimit), + }; +} + +function buildMemoryQuery( + supabase: ReturnType, + params: { + userId: string; + topic?: string; + memoryId?: string; + } +) { + let query = supabase + .from('memories') + .select('id,user_id,content,summary,topic_key,topics,source,salience,metadata') + .eq('user_id', params.userId) + .order('created_at', { ascending: true }); + + if (params.topic?.trim()) { + query = query.contains('topics', [params.topic.trim()]); + } + + if (params.memoryId?.trim()) { + query = query.eq('id', params.memoryId.trim()); + } + + return query; +} + +async function countMatchingMemories( + supabase: ReturnType, + params: { + userId: string; + topic?: string; + memoryId?: string; + } +): Promise { + let query = supabase + .from('memories') + .select('id', { count: 'exact', head: true }) + .eq('user_id', params.userId); + + if (params.topic?.trim()) { + query = query.contains('topics', [params.topic.trim()]); + } + + if (params.memoryId?.trim()) { + query = query.eq('id', params.memoryId.trim()); + } + + const { count, error } = await query; + if (error) { + console.warn(`[memory-llm-extract] failed to count matching memories: ${error.message}`); + return null; + } + + return count ?? 0; +} + +async function loadMemoryPage( + supabase: ReturnType, + params: { + userId: string; + topic?: string; + memoryId?: string; + offset: number; + limit: number; + } +): Promise { + const { data, error } = await buildMemoryQuery(supabase, params).range( + params.offset, + params.offset + params.limit - 1 + ); + if (error) throw new Error(`Failed to load memories: ${error.message}`); + return (data || []) as ExtractableMemoryRow[]; +} + +type ExtractionStatus = 'extracted' | 'skip-existing' | 'skip-no-output'; + +interface BatchItem { + row: ExtractableMemoryRow; + index: number; + metadata: Record; + existingExtractions: MemoryExtractions | null; +} + +interface BatchResultStatus { + index: number; + rowId: string; + status: ExtractionStatus; +} + +function rowToExtractionSource(row: ExtractableMemoryRow): MemoryExtractionSource { + return { + summary: sanitizeSyntheticBenchmarkSummary(row.summary), + content: row.content, + topicKey: row.topic_key, + topics: row.topics, + source: row.source, + salience: row.salience, + }; +} + +function sanitizeSyntheticBenchmarkSummary(summary: string | null): string | null { + if (!summary) return summary; + return /^benchmark\s+(target|distractor)\b/i.test(summary.trim()) ? null : summary; +} + +function estimateBatchChars(row: ExtractableMemoryRow): number { + return ( + Math.min(row.content.length, env.MEMORY_LLM_MAX_INPUT_CHARS) + + Math.min(row.summary?.length || 0, 2000) + + JSON.stringify(row.topics || []).length + + (row.topic_key?.length || 0) + + (row.source?.length || 0) + + 400 + ); +} + +function isTransientPersistenceError(error: { message?: string; code?: string } | null): boolean { + const message = error?.message?.toLowerCase() || ''; + const code = error?.code?.toLowerCase() || ''; + return ( + code === '57014' || + message.includes('upstream') || + message.includes('timeout') || + message.includes('temporarily') || + message.includes('connection') || + message.includes('econnreset') || + message.includes('fetch failed') + ); +} + +async function persistMemoryMetadataWithRetry(params: { + supabase: ReturnType; + row: ExtractableMemoryRow; + metadata: Record; + maxAttempts?: number; +}) { + const { supabase, row, metadata, maxAttempts = 4 } = params; + let lastError: { message: string; code?: string } | null = null; + + for (let attempt = 1; attempt <= maxAttempts; attempt += 1) { + const { error } = await supabase + .from('memories') + .update({ + metadata: metadata as Database['public']['Tables']['memories']['Update']['metadata'], + }) + .eq('id', row.id) + .eq('user_id', row.user_id); + + if (!error) { + if (attempt > 1) { + console.log( + `[memory-llm-extract] memory metadata update succeeded after retry memory=${row.id} attempt=${attempt}` + ); + } + return; + } + + lastError = { message: error.message, code: error.code }; + if (attempt >= maxAttempts || !isTransientPersistenceError(lastError)) { + break; + } + + const delayMs = 500 * 2 ** (attempt - 1); + console.warn( + `[memory-llm-extract] transient memory metadata update failed memory=${row.id} attempt=${attempt}/${maxAttempts} retryInMs=${delayMs}: ${error.message}` + ); + await sleep(delayMs); + } + + throw new Error( + `Failed to update memory ${row.id}: ${lastError?.message || 'unknown persistence error'}` + ); +} + +async function writeExtractionResult(params: { + row: ExtractableMemoryRow; + metadata: Record; + existingExtractions: MemoryExtractions | null; + llmExtractions: MemoryExtractions; + dryRun: boolean; + outputPath: string; + supabase: ReturnType; + extractedKinds: ExtractionKind[]; + replaceExistingKinds: boolean; + keepHistory: boolean; + historyLimit: number; +}) { + const { + row, + metadata, + existingExtractions, + llmExtractions, + dryRun, + outputPath, + supabase, + extractedKinds, + replaceExistingKinds, + keepHistory, + historyLimit, + } = params; + const mergedExtractions = mergeMemoryExtractions(existingExtractions, llmExtractions, { + replaceKinds: replaceExistingKinds ? extractedKinds : undefined, + }); + + if (!dryRun) { + await persistMemoryMetadataWithRetry({ + supabase, + row, + metadata: buildMetadataWithExtraction({ + metadata, + existingExtractions, + mergedExtractions, + keepHistory, + historyLimit, + }), + }); + } + + await appendFile( + outputPath, + `${JSON.stringify({ + type: 'extraction', + memoryId: row.id, + topicKey: row.topic_key, + topics: row.topics, + source: row.source, + salience: row.salience, + summary: row.summary, + contentLength: row.content.length, + extractedKinds, + llmExtractions: mergedExtractions, + embeddingTexts: buildExtractionEmbeddingTexts(mergedExtractions), + dryRun, + extractedAt: new Date().toISOString(), + })}\n` + ); +} + +async function processMemoryRow(params: { + row: ExtractableMemoryRow; + index: number; + total: number; + backend: string; + dryRun: boolean; + force: boolean; + requireRaw: boolean; + requireVersion: number; + extractor: RunnerBackedMemoryExtractor | MemoryLlmExtractor; + outputPath: string; + supabase: ReturnType; + keepHistory: boolean; + historyLimit: number; +}): Promise<'extracted' | 'skip-existing' | 'skip-no-output'> { + const { + row, + index, + total, + backend, + dryRun, + force, + requireRaw, + requireVersion, + extractor, + outputPath, + supabase, + keepHistory, + historyLimit, + } = params; + const metadata = (row.metadata as Record | null) || {}; + const existingExtractions = normalizeMemoryExtractions(metadata.llm_extractions); + if ( + !force && + hasAllEnabledKinds(existingExtractions, extractor.getEnabledKinds(), { + requireRaw, + requireVersion, + }) + ) { + console.log( + `[memory-llm-extract] progress processed=${index}/${total} backend=${backend} memory=${row.id} status=skip-existing` + ); + return 'skip-existing'; + } + + const llmExtractions = await extractor.extract({ + ...rowToExtractionSource(row), + }); + + if (!llmExtractions) { + console.log( + `[memory-llm-extract] progress processed=${index}/${total} backend=${backend} memory=${row.id} status=skip-no-output` + ); + return 'skip-no-output'; + } + + await writeExtractionResult({ + row, + metadata, + existingExtractions, + llmExtractions, + dryRun, + outputPath, + supabase, + extractedKinds: extractor.getEnabledKinds(), + replaceExistingKinds: force, + keepHistory, + historyLimit, + }); + console.log( + `[memory-llm-extract] progress processed=${index}/${total} backend=${backend} memory=${row.id} status=${dryRun ? 'dry-run-extract' : 'extracted'} kinds=${extractor.getEnabledKinds().join(',')}` + ); + return 'extracted'; +} + +async function processMemoryBatch(params: { + items: BatchItem[]; + total: number; + backend: string; + dryRun: boolean; + extractor: RunnerBackedMemoryExtractor; + outputPath: string; + supabase: ReturnType; + force: boolean; + keepHistory: boolean; + historyLimit: number; +}): Promise { + const { + items, + total, + backend, + dryRun, + extractor, + outputPath, + supabase, + force, + keepHistory, + historyLimit, + } = params; + if (items.length === 0) return []; + const batchChars = items.reduce((sum, item) => sum + estimateBatchChars(item.row), 0); + console.log( + `[memory-llm-extract] batch start size=${items.length} indexRange=${items[0]?.index}-${items[items.length - 1]?.index} approxChars=${batchChars} backend=${backend} kinds=${extractor.getEnabledKinds().join(',')}` + ); + + const extractionByMemoryId = await extractor.extractBatch( + items.map((item) => ({ + memoryId: item.row.id, + source: rowToExtractionSource(item.row), + })) + ); + const statuses: BatchResultStatus[] = []; + + for (const item of items) { + const llmExtractions = extractionByMemoryId.get(item.row.id); + if (!llmExtractions) { + console.log( + `[memory-llm-extract] progress processed=${item.index}/${total} backend=${backend} memory=${item.row.id} status=skip-no-output batch=true` + ); + statuses.push({ index: item.index, rowId: item.row.id, status: 'skip-no-output' }); + continue; + } + + await writeExtractionResult({ + row: item.row, + metadata: item.metadata, + existingExtractions: item.existingExtractions, + llmExtractions, + dryRun, + outputPath, + supabase, + extractedKinds: extractor.getEnabledKinds(), + replaceExistingKinds: force, + keepHistory, + historyLimit, + }); + console.log( + `[memory-llm-extract] progress processed=${item.index}/${total} backend=${backend} memory=${item.row.id} status=${dryRun ? 'dry-run-extract' : 'extracted'} kinds=${extractor.getEnabledKinds().join(',')} batch=true` + ); + statuses.push({ index: item.index, rowId: item.row.id, status: 'extracted' }); + } + + console.log( + `[memory-llm-extract] batch complete size=${items.length} extracted=${statuses.filter((status) => status.status === 'extracted').length} skipped=${statuses.filter((status) => status.status !== 'extracted').length} backend=${backend}` + ); + return statuses; +} + +function getEnabledKinds(options: { batchAllKinds?: boolean } = {}): ExtractionKind[] { + if (options.batchAllKinds) return ['entity', 'durable_fact', 'summary']; + const enabledKinds: ExtractionKind[] = []; + if (env.MEMORY_LLM_ENTITY_ENABLED) enabledKinds.push('entity'); + if (env.MEMORY_LLM_DURABLE_FACT_ENABLED) enabledKinds.push('durable_fact'); + if (env.MEMORY_LLM_SUMMARY_ENABLED) enabledKinds.push('summary'); + if (env.MEMORY_LLM_CURRENT_STATE_ENABLED) enabledKinds.push('current_state'); + return enabledKinds; +} + +function extractJsonObject(text: string): string { + const trimmed = text.trim(); + if (trimmed.startsWith('{') && trimmed.endsWith('}')) return trimmed; + const fencedJson = trimmed.match(/```(?:json)?\s*([\s\S]*?)```/i); + if (fencedJson?.[1]?.trim()) return extractJsonObject(fencedJson[1]); + const firstBrace = trimmed.indexOf('{'); + const lastBrace = trimmed.lastIndexOf('}'); + if (firstBrace === -1 || lastBrace === -1 || lastBrace <= firstBrace) { + throw new Error('runner extraction response did not contain a JSON object'); + } + return trimmed.slice(firstBrace, lastBrace + 1); +} + +function compactLogSnippet(text: string, maxChars = 500): string { + const compacted = text.replace(/\s+/g, ' ').trim(); + if (compacted.length <= maxChars) return compacted; + return `${compacted.slice(0, Math.max(0, maxChars - 1)).trimEnd()}…`; +} + +function parseKindPayload(kind: ExtractionKind, raw: unknown): MemoryExtractions[ExtractionKind] { + return coerceExtractionPayload(kind, raw); +} + +function assignExtractionPayload( + payload: Partial, + kind: ExtractionKind, + result: MemoryExtractions[ExtractionKind] +) { + switch (kind) { + case 'entity': + payload.entity = result as MemoryExtractions['entity']; + break; + case 'durable_fact': + payload.durable_fact = result as MemoryExtractions['durable_fact']; + break; + case 'summary': + payload.summary = result as MemoryExtractions['summary']; + break; + case 'current_state': + payload.current_state = result as MemoryExtractions['current_state']; + break; + } +} + +function assignRawExtractionPayload( + payload: Partial, + kind: ExtractionKind, + raw: unknown +) { + payload.raw = { + ...(payload.raw || {}), + [kind]: raw, + }; +} + +interface RunnerExtractionResult { + normalized: MemoryExtractions[ExtractionKind]; + raw: unknown; +} + +function clampText(text: string, maxChars: number): string { + if (text.length <= maxChars) return text; + return `${text.slice(0, Math.max(0, maxChars - 1)).trimEnd()}…`; +} + +function sanitizeSourceForRunner(source: MemoryExtractionSource): MemoryExtractionSource { + return { + ...source, + summary: source.summary + ? clampText(source.summary, Math.min(env.MEMORY_LLM_MAX_INPUT_CHARS, 2000)) + : source.summary, + content: clampText(source.content, env.MEMORY_LLM_MAX_INPUT_CHARS), + }; +} + +class RunnerBackedMemoryExtractor { + private readonly enabledKinds: ExtractionKind[]; + private readonly runner: IRunner; + private readonly backend: 'claude' | 'codex'; + private readonly config: ClaudeRunnerConfig; + + constructor(backend: 'claude' | 'codex', enabledKinds: ExtractionKind[] = getEnabledKinds()) { + this.backend = backend; + this.enabledKinds = enabledKinds; + this.runner = backend === 'claude' ? new ClaudeRunner() : new CodexRunner(); + this.config = { + workingDirectory: process.env.MEMORY_LLM_EXTRACT_WORKING_DIRECTORY || process.cwd(), + mcpConfigPath: process.env.MEMORY_LLM_EXTRACT_MCP_CONFIG_PATH || '', + ...(env.MEMORY_LLM_MODEL ? { model: env.MEMORY_LLM_MODEL } : {}), + systemPrompt: + 'You are a deterministic memory extraction worker. Do not use tools. Return only strict JSON matching the requested schema.', + sandboxBypass: false, + }; + } + + isEnabled(): boolean { + return env.MEMORY_LLM_EXTRACTION_ENABLED && this.enabledKinds.length > 0; + } + + getEnabledKinds(): ExtractionKind[] { + return [...this.enabledKinds]; + } + + async extract(source: MemoryExtractionSource): Promise { + if (!this.isEnabled()) return null; + const sanitizedSource = sanitizeSourceForRunner(source); + const extractedAt = new Date().toISOString(); + const payload: Partial = { + version: MEMORY_EXTRACTION_VERSION, + provider: `runner:${this.backend}`, + model: env.MEMORY_LLM_MODEL || this.backend, + extractedAt, + raw: { + provider: `runner:${this.backend}`, + model: env.MEMORY_LLM_MODEL || this.backend, + extractedAt, + }, + }; + + for (const kind of this.enabledKinds) { + const result = await this.extractKind(kind, sanitizedSource); + if (!result) continue; + assignExtractionPayload(payload, kind, result.normalized); + assignRawExtractionPayload(payload, kind, result.raw); + } + + const normalized = normalizeMemoryExtractions(payload); + return normalized && + Object.keys(normalized).some((key) => + ['entity', 'durable_fact', 'summary', 'current_state'].includes(key) + ) + ? normalized + : null; + } + + async extractBatch( + items: BatchMemoryExtractionSource[] + ): Promise> { + if (!this.isEnabled() || items.length === 0) return new Map(); + if (items.length === 1) { + const extraction = await this.extract(items[0].source); + return extraction ? new Map([[items[0].memoryId, extraction]]) : new Map(); + } + + const sanitizedItems = items.map((item) => ({ + memoryId: item.memoryId, + source: sanitizeSourceForRunner(item.source), + })); + const parsed = await this.extractBatchOnce(sanitizedItems); + if (!parsed) { + const midpoint = Math.ceil(items.length / 2); + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch parse failed; splitting size=${items.length} into ${midpoint}+${items.length - midpoint}` + ); + const [left, right] = await Promise.all([ + this.extractBatch(items.slice(0, midpoint)), + this.extractBatch(items.slice(midpoint)), + ]); + return new Map([...left, ...right]); + } + + const output = new Map(parsed.extractions); + const retryIds = new Set([...parsed.invalidMemoryIds, ...parsed.missingMemoryIds]); + if (retryIds.size > 0) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch partial failure; retrying individually count=${retryIds.size}` + ); + } + + for (const item of items) { + if (!retryIds.has(item.memoryId)) continue; + const extraction = await this.extract(item.source); + if (extraction) output.set(item.memoryId, extraction); + } + + return output; + } + + private async extractKind( + kind: ExtractionKind, + source: MemoryExtractionSource + ): Promise { + const prompt = buildExtractionPrompt(source, kind); + const message = [ + prompt.systemPrompt, + prompt.schemaDescription, + '', + 'Return only the JSON object. Do not wrap it in Markdown. Do not call tools.', + '', + prompt.userPrompt, + ].join('\n'); + const result = await this.runner.run(message, { config: this.config }); + if (!result.success) { + const retryAtMatch = result.error?.match(/try again at ([^.]+)\./i); + console.warn( + `[memory-llm-extract] runner backend=${this.backend} kind=${kind} failed: ${result.error || 'unknown error'}` + ); + if ( + result.error?.includes("You've hit your usage limit") || + result.error?.toLowerCase().includes('usage limit') + ) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} appears rate-limited${ + retryAtMatch?.[1] ? ` until ${retryAtMatch[1]}` : '' + }` + ); + } + return null; + } + + const content = + result.finalTextResponse || + result.responses + .map((response) => response.content) + .filter(Boolean) + .join('\n'); + if (!content.trim()) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} kind=${kind} returned no text` + ); + return null; + } + + try { + const parsedJson = JSON.parse(extractJsonObject(content)); + return { + normalized: parseKindPayload(kind, parsedJson), + raw: parsedJson, + }; + } catch (error) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} kind=${kind} returned invalid JSON: ${ + error instanceof Error ? error.message : String(error) + }; contentSnippet=${JSON.stringify(compactLogSnippet(content))}` + ); + return null; + } + } + + private async extractBatchOnce(items: BatchMemoryExtractionSource[]): Promise<{ + extractions: Map; + invalidMemoryIds: Set; + missingMemoryIds: Set; + } | null> { + const prompt = buildBatchExtractionPrompt(items, this.enabledKinds); + const message = [ + prompt.systemPrompt, + prompt.schemaDescription, + '', + 'Return only the JSON object. Do not wrap it in Markdown. Do not call tools.', + '', + prompt.userPrompt, + ].join('\n'); + const result = await this.runner.run(message, { config: this.config }); + if (!result.success) { + const retryAtMatch = result.error?.match(/try again at ([^.]+)\./i); + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch failed: ${result.error || 'unknown error'}` + ); + if ( + result.error?.includes("You've hit your usage limit") || + result.error?.toLowerCase().includes('usage limit') + ) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} appears rate-limited${ + retryAtMatch?.[1] ? ` until ${retryAtMatch[1]}` : '' + }` + ); + } + return null; + } + + const content = + result.finalTextResponse || + result.responses + .map((response) => response.content) + .filter(Boolean) + .join('\n'); + if (!content.trim()) { + console.warn(`[memory-llm-extract] runner backend=${this.backend} batch returned no text`); + return null; + } + + try { + const parsed = batchMemoryExtractionResponseSchema.parse( + JSON.parse(extractJsonObject(content)) + ); + const requestedMemoryIds = new Set(items.map((item) => item.memoryId)); + const seenMemoryIds = new Set(); + const invalidMemoryIds = new Set(); + const extractions = new Map(); + + for (const resultItem of parsed.results) { + if (!requestedMemoryIds.has(resultItem.memoryId)) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch returned unknown memoryId=${resultItem.memoryId}` + ); + continue; + } + seenMemoryIds.add(resultItem.memoryId); + + const extractedAt = new Date().toISOString(); + const payload: Partial = { + version: MEMORY_EXTRACTION_VERSION, + provider: `runner:${this.backend}:batch`, + model: env.MEMORY_LLM_MODEL || this.backend, + extractedAt, + raw: { + provider: `runner:${this.backend}:batch`, + model: env.MEMORY_LLM_MODEL || this.backend, + extractedAt, + }, + }; + let invalid = false; + for (const kind of this.enabledKinds) { + const rawKindPayload = resultItem[kind]; + if (rawKindPayload === undefined) { + invalid = true; + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch memory=${resultItem.memoryId} missing kind=${kind}` + ); + break; + } + try { + assignExtractionPayload(payload, kind, parseKindPayload(kind, rawKindPayload)); + assignRawExtractionPayload(payload, kind, rawKindPayload); + } catch (error) { + invalid = true; + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch memory=${resultItem.memoryId} invalid kind=${kind}: ${ + error instanceof Error ? error.message : String(error) + }` + ); + break; + } + } + + const normalized = normalizeMemoryExtractions(payload); + if ( + invalid || + !normalized || + !Object.keys(normalized).some((key) => + ['entity', 'durable_fact', 'summary', 'current_state'].includes(key) + ) + ) { + invalidMemoryIds.add(resultItem.memoryId); + continue; + } + extractions.set(resultItem.memoryId, normalized); + } + + const missingMemoryIds = new Set( + items.map((item) => item.memoryId).filter((memoryId) => !seenMemoryIds.has(memoryId)) + ); + return { extractions, invalidMemoryIds, missingMemoryIds }; + } catch (error) { + console.warn( + `[memory-llm-extract] runner backend=${this.backend} batch returned invalid JSON: ${ + error instanceof Error ? error.message : String(error) + }; contentSnippet=${JSON.stringify(compactLogSnippet(content))}` + ); + return null; + } + } +} + +async function main() { + const userId = process.env.MEMORY_LLM_EXTRACT_USER_ID || process.env.BENCHMARK_USER_ID; + if (!userId) { + throw new Error('MEMORY_LLM_EXTRACT_USER_ID or BENCHMARK_USER_ID is required'); + } + + const limit = parsePositiveInt(process.env.MEMORY_LLM_EXTRACT_LIMIT, 100); + const offset = parseNonNegativeInt(process.env.MEMORY_LLM_EXTRACT_OFFSET, 0); + const pageSize = Math.min(parsePositiveInt(process.env.MEMORY_LLM_EXTRACT_PAGE_SIZE, 1000), 1000); + const batchAllKinds = parseBoolean(process.env.MEMORY_LLM_EXTRACT_BATCH_ALL_KINDS, false); + const batchSize = Math.min(parsePositiveInt(process.env.MEMORY_LLM_EXTRACT_BATCH_SIZE, 1), 50); + const batchMaxChars = parsePositiveInt(process.env.MEMORY_LLM_EXTRACT_BATCH_MAX_CHARS, 60_000); + const maxConsecutiveFailures = parsePositiveInt( + process.env.MEMORY_LLM_EXTRACT_MAX_CONSECUTIVE_FAILURES, + 10 + ); + const topic = process.env.MEMORY_LLM_EXTRACT_TOPIC; + const memoryId = process.env.MEMORY_LLM_EXTRACT_MEMORY_ID; + const dryRun = parseBoolean(process.env.MEMORY_LLM_EXTRACT_DRY_RUN, false); + const force = parseBoolean(process.env.MEMORY_LLM_EXTRACT_FORCE, false); + const requireRaw = parseBoolean(process.env.MEMORY_LLM_EXTRACT_REQUIRE_RAW, true); + const requireVersion = parseNonNegativeInt(process.env.MEMORY_LLM_EXTRACT_REQUIRE_VERSION, 0); + const keepHistory = parseBoolean(process.env.MEMORY_LLM_EXTRACT_KEEP_HISTORY, true); + const historyLimit = parsePositiveInt(process.env.MEMORY_LLM_EXTRACT_HISTORY_LIMIT, 10); + const outputPath = + process.env.MEMORY_LLM_EXTRACT_OUTPUT_PATH || + resolve( + process.cwd(), + 'output', + 'memory-extractions', + `memory-llm-extract-${Date.now()}.jsonl` + ); + + const backend = (process.env.MEMORY_LLM_EXTRACT_BACKEND || 'direct').trim().toLowerCase(); + if (!['direct', 'claude', 'codex'].includes(backend)) { + throw new Error('MEMORY_LLM_EXTRACT_BACKEND must be one of: direct, claude, codex'); + } + const enabledKinds = getEnabledKinds({ batchAllKinds }); + const extractor = + backend === 'claude' || backend === 'codex' + ? new RunnerBackedMemoryExtractor(backend, enabledKinds) + : new MemoryLlmExtractor(); + const batchExtractor = extractor instanceof RunnerBackedMemoryExtractor ? extractor : null; + const useBatchExtraction = Boolean(batchExtractor && batchSize > 1); + if (!extractor.isEnabled()) { + throw new Error( + 'Memory LLM extraction is disabled. Set MEMORY_LLM_EXTRACTION_ENABLED=true and at least one per-type flag.' + ); + } + + const supabase = createSupabaseClient(); + await mkdir(dirname(outputPath), { recursive: true }); + await writeFile( + outputPath, + `${JSON.stringify({ + type: 'config', + userId, + memoryId: memoryId || null, + topic: topic || null, + limit, + offset, + pageSize, + batchAllKinds, + batchSize, + batchMaxChars, + useBatchExtraction, + maxConsecutiveFailures, + dryRun, + force, + requireRaw, + requireVersion, + keepHistory, + historyLimit, + backend, + enabledKinds: extractor.getEnabledKinds(), + startedAt: new Date().toISOString(), + })}\n` + ); + + console.log(`[memory-llm-extract] auditOutput=${outputPath}`); + + let extracted = 0; + let skipped = 0; + let processed = 0; + let loaded = 0; + let consecutiveNoOutput = 0; + const matchingCount = await countMatchingMemories(supabase, { + userId, + topic, + memoryId, + }); + const plannedTotal = + matchingCount === null ? limit : Math.min(limit, Math.max(0, matchingCount - offset)); + console.log( + `[memory-llm-extract] starting total=${plannedTotal} offset=${offset} limit=${limit} pageSize=${pageSize} batchSize=${useBatchExtraction ? batchSize : 1} batchMaxChars=${useBatchExtraction ? batchMaxChars : 0} extracted=${extracted} skipped=${skipped} backend=${backend} kinds=${extractor.getEnabledKinds().join(',')}` + ); + + const recordStatus = (status: ExtractionStatus, index: number) => { + if (status === 'extracted') { + extracted += 1; + consecutiveNoOutput = 0; + } else if (status === 'skip-existing') { + skipped += 1; + } else { + skipped += 1; + consecutiveNoOutput += 1; + } + console.log( + `[memory-llm-extract] counts processed=${index}/${plannedTotal} loaded=${loaded} extracted=${extracted} skipped=${skipped} consecutiveNoOutput=${consecutiveNoOutput} backend=${backend}` + ); + if (consecutiveNoOutput >= maxConsecutiveFailures) { + console.warn( + `[memory-llm-extract] stopping early after ${consecutiveNoOutput} consecutive no-output rows; maxConsecutiveFailures=${maxConsecutiveFailures}` + ); + return true; + } + return false; + }; + + while (processed < limit) { + const remaining = limit - processed; + const rows = await loadMemoryPage(supabase, { + userId, + topic, + memoryId, + offset: offset + processed, + limit: Math.min(pageSize, remaining), + }); + loaded += rows.length; + if (rows.length === 0) break; + + console.log( + `[memory-llm-extract] page loaded=${rows.length} pageStart=${offset + processed} processed=${processed}/${plannedTotal} extracted=${extracted} skipped=${skipped}` + ); + + if (useBatchExtraction) { + let batch: BatchItem[] = []; + let batchChars = 0; + let shouldStop = false; + + const flushBatch = async () => { + if (batch.length === 0) return; + const statuses = await processMemoryBatch({ + items: batch, + total: plannedTotal, + backend, + dryRun, + extractor: batchExtractor!, + outputPath, + supabase, + force, + keepHistory, + historyLimit, + }); + batch = []; + batchChars = 0; + for (const status of statuses) { + if (recordStatus(status.status, status.index)) { + shouldStop = true; + break; + } + } + }; + + for (const row of rows) { + processed += 1; + const metadata = (row.metadata as Record | null) || {}; + const existingExtractions = normalizeMemoryExtractions(metadata.llm_extractions); + if ( + !force && + hasAllEnabledKinds(existingExtractions, extractor.getEnabledKinds(), { + requireRaw, + requireVersion, + }) + ) { + console.log( + `[memory-llm-extract] progress processed=${processed}/${plannedTotal} backend=${backend} memory=${row.id} status=skip-existing` + ); + shouldStop = recordStatus('skip-existing', processed); + if (shouldStop) break; + continue; + } + + const rowChars = estimateBatchChars(row); + if ( + batch.length > 0 && + (batch.length >= batchSize || batchChars + rowChars > batchMaxChars) + ) { + await flushBatch(); + if (shouldStop) break; + } + batch.push({ row, index: processed, metadata, existingExtractions }); + batchChars += rowChars; + if (batch.length >= batchSize || batchChars >= batchMaxChars) { + await flushBatch(); + if (shouldStop) break; + } + } + await flushBatch(); + if (shouldStop) { + processed = limit; + break; + } + } else { + for (const row of rows) { + processed += 1; + const result = await processMemoryRow({ + row, + index: processed, + total: plannedTotal, + backend, + dryRun, + force, + requireRaw, + requireVersion, + extractor, + outputPath, + supabase, + keepHistory, + historyLimit, + }); + if (recordStatus(result, processed)) { + processed = limit; + break; + } + } + } + + if (rows.length < Math.min(pageSize, remaining)) break; + } + + await appendFile( + outputPath, + `${JSON.stringify({ + type: 'summary', + loaded, + processed, + total: plannedTotal, + extracted, + skipped, + dryRun, + backend, + batchAllKinds, + batchSize: useBatchExtraction ? batchSize : 1, + batchMaxChars: useBatchExtraction ? batchMaxChars : null, + requireRaw, + requireVersion, + keepHistory, + historyLimit, + completedAt: new Date().toISOString(), + })}\n` + ); + + console.log( + `[memory-llm-extract] complete loaded=${loaded} processed=${processed}/${plannedTotal} extracted=${extracted} skipped=${skipped} dryRun=${dryRun} backend=${backend} auditOutput=${outputPath}` + ); +} + +main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); +}); diff --git a/packages/api/src/services/embeddings/memory-chunks.test.ts b/packages/api/src/services/embeddings/memory-chunks.test.ts new file mode 100644 index 00000000..fa5dbf98 --- /dev/null +++ b/packages/api/src/services/embeddings/memory-chunks.test.ts @@ -0,0 +1,152 @@ +import { describe, expect, it } from 'vitest'; +import { + buildMemoryEmbeddingChunks, + countChunkViews, + inferChunkTypeFromMetadata, + MEMORY_EMBEDDING_CHUNKS_VERSION, +} from './memory-chunks'; +import { memoryExtractionsSchema } from '../memory-llm-extraction'; + +describe('memory chunk multi-view helpers', () => { + it('builds summary, fact, topic, entity, and content views when structured data is available', () => { + const chunks = buildMemoryEmbeddingChunks({ + summary: 'Policy B replaces Policy A for wound escalation.', + content: + 'Policy A required fax escalation. Policy B replaces Policy A and requires portal escalation within 24 hours. UCLA Care Team owns the policy review.', + topicKey: 'policy:wound-escalation', + topics: ['policy:wound-escalation', 'person:care-team'], + source: 'observation', + salience: 'high', + model: { maxInputChars: 1200 } as { maxInputChars: number }, + }); + + expect(chunks.some((chunk) => chunk.chunkType === 'summary')).toBe(true); + expect(chunks.some((chunk) => chunk.chunkType === 'fact')).toBe(true); + expect(chunks.some((chunk) => chunk.chunkType === 'topic')).toBe(true); + expect(chunks.some((chunk) => chunk.chunkType === 'entity')).toBe(true); + expect(chunks.some((chunk) => chunk.chunkType === 'content')).toBe(true); + + const viewCounts = countChunkViews(chunks); + expect(viewCounts.summary).toBe(1); + expect(viewCounts.current_state).toBe(0); + expect(viewCounts.content).toBeGreaterThan(0); + + const metadata = { + embedding_chunks: { + version: MEMORY_EMBEDDING_CHUNKS_VERSION, + viewCounts, + }, + }; + + expect(inferChunkTypeFromMetadata(0, metadata)).toBe('summary'); + }); + + it('prefers llm-derived summary, durable fact, entity, and current state chunks when provided', () => { + const llmExtractions = memoryExtractionsSchema.parse({ + version: 1, + provider: 'openai', + model: 'gpt-4.1-mini', + extractedAt: '2026-04-18T12:00:00.000Z', + summary: { + summary: 'The benchmark doc discussion established the current feature-flag plan.', + keyPoints: ['feature flags gate extraction', 'current state is first-class'], + actionRelevance: 'Helps future agents route retrieval experiments correctly.', + }, + durable_fact: { + durableFacts: [ + { + fact: 'Current-state memory should be indexed separately from durable facts.', + category: 'decision', + subject: 'current_state index', + object: 'durable_fact index', + evidence: + 'Current state should stay separate from durable facts because it is volatile.', + }, + ], + }, + entity: { + entities: [ + { + name: 'Wren', + aliases: ['wren'], + entityType: 'person', + description: 'Reviewer providing memory-system feedback.', + evidence: 'Wren forgot that we were discussing a specific document.', + }, + ], + }, + current_state: { + state: 'The dev server auto-restarts on file change.', + scope: 'local dev server', + status: 'running', + volatility: 'volatile', + evidence: + 'Current state is also important: like the currently running dev server will autorestart.', + }, + }); + + const chunks = buildMemoryEmbeddingChunks({ + summary: 'Fallback summary that should not be used', + content: + 'Current state is also important: like the currently running dev server will autorestart.', + topicKey: 'spec:memory-benchmark-notes', + topics: ['person:wren'], + source: 'observation', + salience: 'high', + model: { maxInputChars: 1200 } as { maxInputChars: number }, + llmExtractions, + extractionMode: 'llm', + }); + + expect(chunks.find((chunk) => chunk.chunkType === 'summary')?.text).toContain( + 'action relevance' + ); + expect(chunks.find((chunk) => chunk.chunkType === 'fact')?.text).toContain('durable fact:'); + expect(chunks.find((chunk) => chunk.chunkType === 'entity')?.text).toContain('entity: Wren'); + expect(chunks.find((chunk) => chunk.chunkType === 'current_state')?.text).toContain( + 'current state:' + ); + + const viewCounts = countChunkViews(chunks); + expect(viewCounts.summary).toBe(1); + expect(viewCounts.fact).toBe(1); + expect(viewCounts.entity).toBe(1); + expect(viewCounts.current_state).toBe(1); + }); + + it('uses heuristic extraction mode by default even when llm metadata exists', () => { + const llmExtractions = memoryExtractionsSchema.parse({ + version: 1, + provider: 'openai', + model: 'gpt-4.1-mini', + extractedAt: '2026-04-18T12:00:00.000Z', + current_state: { + state: 'The dev server auto-restarts on file change.', + scope: 'local dev server', + status: 'running', + volatility: 'volatile', + evidence: 'The current dev server auto-restarts when files change.', + }, + }); + + const chunks = buildMemoryEmbeddingChunks({ + summary: 'Policy B replaces Policy A for wound escalation.', + content: 'Policy B replaces Policy A and requires portal escalation within 24 hours.', + model: { maxInputChars: 1200 } as { maxInputChars: number }, + llmExtractions, + }); + + expect(chunks.some((chunk) => chunk.chunkType === 'current_state')).toBe(false); + expect(chunks.some((chunk) => chunk.chunkType === 'fact')).toBe(true); + }); + + it('sanitizes unpaired unicode surrogates before chunk persistence', () => { + const chunks = buildMemoryEmbeddingChunks({ + content: 'A benchmark transcript contained a broken low surrogate \udc00 in the text.', + model: { maxInputChars: 1200 } as { maxInputChars: number }, + }); + + expect(chunks.some((chunk) => chunk.text.includes('\udc00'))).toBe(false); + expect(chunks.some((chunk) => chunk.text.includes('�'))).toBe(true); + }); +}); diff --git a/packages/api/src/services/embeddings/memory-chunks.ts b/packages/api/src/services/embeddings/memory-chunks.ts index 8f000b12..8615b136 100644 --- a/packages/api/src/services/embeddings/memory-chunks.ts +++ b/packages/api/src/services/embeddings/memory-chunks.ts @@ -1,14 +1,38 @@ import type { Json, TablesInsert } from '../../data/supabase/types'; +import { MEMORY_EMBEDDING_CHUNKS_VERSION } from '../memory-benchmark-constants'; +import { + buildCurrentStateEmbeddingTexts, + buildDurableFactEmbeddingTexts, + buildEntityEmbeddingTexts, + buildSummaryEmbeddingTexts, + normalizeMemoryExtractions, + type MemoryExtractions, +} from '../memory-llm-extraction'; import type { EmbeddingResult } from './router'; import { type VettedEmbeddingModel } from './vetted-models'; -export const MEMORY_EMBEDDING_CHUNKS_VERSION = 1; +export { MEMORY_EMBEDDING_CHUNKS_VERSION }; const DEFAULT_MAX_CHARS = 1000; const DEFAULT_OVERLAP_CHARS = 150; +const MAX_FACT_CHUNKS = 3; +const MAX_ENTITY_CHUNKS = 2; +const MIN_FACT_SENTENCE_CHARS = 48; +const MAX_FACT_SENTENCE_CHARS = 280; + +export type MemoryChunkType = 'summary' | 'fact' | 'topic' | 'entity' | 'current_state' | 'content'; +export type MemoryExtractionChunkMode = 'heuristic' | 'llm' | 'merged'; +const CHUNK_TYPE_ORDER: MemoryChunkType[] = [ + 'summary', + 'fact', + 'topic', + 'entity', + 'current_state', + 'content', +]; export interface MemoryEmbeddingChunk { chunkIndex: number; - chunkType: 'summary' | 'content'; + chunkType: MemoryChunkType; text: string; startOffset: number; endOffset: number; @@ -18,6 +42,26 @@ export interface EmbeddedMemoryChunk extends MemoryEmbeddingChunk { embedding: EmbeddingResult; } +export interface MemoryChunkViewCounts { + summary: number; + fact: number; + topic: number; + entity: number; + current_state: number; + content: number; +} + +function emptyViewCounts(): MemoryChunkViewCounts { + return { + summary: 0, + fact: 0, + topic: 0, + entity: 0, + current_state: 0, + content: 0, + }; +} + function pickMaxChunkChars(model: VettedEmbeddingModel | null): number { if (!model?.maxInputChars) return DEFAULT_MAX_CHARS; return Math.max(200, model.maxInputChars - 100); @@ -43,7 +87,7 @@ function buildContentChunks( maxChars: number, overlapChars: number ): MemoryEmbeddingChunk[] { - const normalized = text.trim(); + const normalized = sanitizeChunkText(text.trim()); if (!normalized) return []; const chunks: MemoryEmbeddingChunk[] = []; @@ -73,34 +117,326 @@ function buildContentChunks( return chunks; } +function normalizeWhitespace(text: string): string { + return replaceUnpairedSurrogates(text.replace(/\s+/g, ' ').trim()); +} + +function replaceUnpairedSurrogates(text: string): string { + let sanitized = ''; + + for (let i = 0; i < text.length; i += 1) { + const code = text.charCodeAt(i); + const isHighSurrogate = code >= 0xd800 && code <= 0xdbff; + const isLowSurrogate = code >= 0xdc00 && code <= 0xdfff; + + if (isHighSurrogate) { + const next = text.charCodeAt(i + 1); + if (next >= 0xdc00 && next <= 0xdfff) { + sanitized += text[i] + text[i + 1]; + i += 1; + } else { + sanitized += '�'; + } + continue; + } + + sanitized += isLowSurrogate ? '�' : text[i]; + } + + return sanitized; +} + +function sanitizeChunkText(text: string): string { + return replaceUnpairedSurrogates(text); +} + +function splitIntoSentences(text: string): string[] { + const normalized = text + .replace(/\r/g, '\n') + .split(/\n+/) + .flatMap((line) => line.split(/(?<=[.!?])\s+/)) + .map(normalizeWhitespace) + .filter(Boolean); + + return normalized; +} + +function sentenceScore(sentence: string): number { + const lowered = sentence.toLowerCase(); + const cueWords = [ + ' must ', + ' should ', + ' decided ', + ' because ', + ' prefer ', + ' important ', + ' override', + ' replace', + ' policy', + ' convention', + ' requires ', + ' means ', + ]; + + let score = 0; + if (/\d/.test(sentence)) score += 0.3; + if (cueWords.some((cue) => lowered.includes(cue.trim()) || lowered.includes(cue))) score += 0.4; + if (/[A-Z][a-z]+(?:\s+[A-Z][a-z]+)+/.test(sentence) || /\b[A-Z]{2,}\b/.test(sentence)) { + score += 0.2; + } + + const tokens = lowered.split(/[^a-z0-9]+/).filter((token) => token.length > 2); + const uniqueTokens = new Set(tokens); + if (tokens.length > 0) score += Math.min(0.3, uniqueTokens.size / Math.max(tokens.length, 1) / 2); + + return score; +} + +function buildFactChunks(text: string): MemoryEmbeddingChunk[] { + const sentences = splitIntoSentences(text) + .filter( + (sentence) => + sentence.length >= MIN_FACT_SENTENCE_CHARS && sentence.length <= MAX_FACT_SENTENCE_CHARS + ) + .map((sentence) => ({ sentence, score: sentenceScore(sentence) })) + .filter((entry) => entry.score > 0.2) + .sort((a, b) => b.score - a.score || b.sentence.length - a.sentence.length); + + const seen = new Set(); + const chunks: MemoryEmbeddingChunk[] = []; + + for (const entry of sentences) { + const normalized = entry.sentence.toLowerCase(); + if (seen.has(normalized)) continue; + seen.add(normalized); + chunks.push({ + chunkIndex: chunks.length, + chunkType: 'fact', + text: entry.sentence, + startOffset: 0, + endOffset: entry.sentence.length, + }); + if (chunks.length >= MAX_FACT_CHUNKS) break; + } + + return chunks; +} + +function buildTopicChunks(params: { + topicKey?: string | null; + topics?: string[] | null; + source?: string | null; + salience?: string | null; +}): MemoryEmbeddingChunk[] { + const lines: string[] = []; + if (params.topicKey?.trim()) lines.push(`topic key: ${params.topicKey.trim()}`); + const normalizedTopics = (params.topics || []).map((topic) => topic.trim()).filter(Boolean); + if (normalizedTopics.length > 0) lines.push(`topics: ${normalizedTopics.join(', ')}`); + if (params.source?.trim()) lines.push(`source: ${params.source.trim()}`); + if (params.salience?.trim()) lines.push(`salience: ${params.salience.trim()}`); + const text = lines.join('\n').trim(); + if (!text) return []; + return [ + { + chunkIndex: 0, + chunkType: 'topic', + text, + startOffset: 0, + endOffset: text.length, + }, + ]; +} + +function buildChunksFromTexts(chunkType: MemoryChunkType, texts: string[]): MemoryEmbeddingChunk[] { + return texts + .map((text) => sanitizeChunkText(normalizeWhitespace(text))) + .filter(Boolean) + .map((text, index) => ({ + chunkIndex: index, + chunkType, + text, + startOffset: 0, + endOffset: text.length, + })); +} + +function extractEntityPhrases(text: string): string[] { + const matches = [ + ...text.matchAll(/\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+){0,3}\b/g), + ...text.matchAll(/\b[A-Z]{2,}(?:\s+[A-Z]{2,})*\b/g), + ].map((match) => normalizeWhitespace(match[0] || '')); + + const unique = new Set(); + for (const phrase of matches) { + if (phrase.length < 3) continue; + unique.add(phrase); + if (unique.size >= 8) break; + } + + return Array.from(unique); +} + +function buildEntityChunks(params: { + summary?: string | null; + content: string; + topicKey?: string | null; + topics?: string[] | null; +}): MemoryEmbeddingChunk[] { + const phrases = new Set(); + + for (const topic of params.topics || []) { + const normalized = topic + .split(':') + .map((part) => part.trim()) + .filter(Boolean) + .join(' '); + if (normalized) phrases.add(normalized); + } + + if (params.topicKey?.trim()) { + const normalized = params.topicKey + .split(':') + .map((part) => part.trim()) + .filter(Boolean) + .join(' '); + if (normalized) phrases.add(normalized); + } + + for (const phrase of extractEntityPhrases(`${params.summary || ''}\n${params.content}`)) { + phrases.add(phrase); + if (phrases.size >= 10) break; + } + + const entries = Array.from(phrases) + .map(normalizeWhitespace) + .filter(Boolean) + .slice(0, MAX_ENTITY_CHUNKS); + + return entries.map((entry, index) => ({ + chunkIndex: index, + chunkType: 'entity', + text: `entity focus: ${entry}`, + startOffset: 0, + endOffset: entry.length, + })); +} + +function reindexChunks(chunks: MemoryEmbeddingChunk[], startIndex: number): MemoryEmbeddingChunk[] { + return chunks.map((chunk, index) => ({ + ...chunk, + chunkIndex: startIndex + index, + })); +} + +export function countChunkViews(chunks: MemoryEmbeddingChunk[]): MemoryChunkViewCounts { + const counts = emptyViewCounts(); + for (const chunk of chunks) counts[chunk.chunkType] += 1; + return counts; +} + +export function inferChunkTypeFromMetadata( + chunkIndex: number | null | undefined, + metadata: Record | null | undefined +): MemoryChunkType | null { + if ( + typeof chunkIndex !== 'number' || + chunkIndex < 0 || + !metadata || + typeof metadata !== 'object' + ) { + return null; + } + + const embeddingChunks = + 'embedding_chunks' in metadata && + metadata.embedding_chunks && + typeof metadata.embedding_chunks === 'object' + ? (metadata.embedding_chunks as Record) + : null; + const viewCounts = + embeddingChunks && + 'viewCounts' in embeddingChunks && + embeddingChunks.viewCounts && + typeof embeddingChunks.viewCounts === 'object' + ? (embeddingChunks.viewCounts as Record) + : null; + + if (!viewCounts) return null; + + let offset = 0; + for (const chunkType of CHUNK_TYPE_ORDER) { + const rawCount = viewCounts[chunkType]; + const count = typeof rawCount === 'number' ? rawCount : 0; + if (chunkIndex < offset + count) return chunkType; + offset += count; + } + + return null; +} + export function buildMemoryEmbeddingChunks(params: { summary?: string | null; content: string; + topicKey?: string | null; + topics?: string[] | null; + source?: string | null; + salience?: string | null; model?: VettedEmbeddingModel | null; + llmExtractions?: MemoryExtractions | Record | null; + extractionMode?: MemoryExtractionChunkMode; }): MemoryEmbeddingChunk[] { - const { summary, content, model = null } = params; - const chunks: MemoryEmbeddingChunk[] = []; + const { summary, content, topicKey, topics, source, salience, model = null } = params; const maxChars = pickMaxChunkChars(model); + const chunks: MemoryEmbeddingChunk[] = []; + const llmExtractions = normalizeMemoryExtractions(params.llmExtractions); + const extractionMode = params.extractionMode || 'heuristic'; + const includeHeuristic = extractionMode === 'heuristic' || extractionMode === 'merged'; + const includeLlm = extractionMode === 'llm' || extractionMode === 'merged'; + const extractedSummaryTexts = llmExtractions?.summary + ? buildSummaryEmbeddingTexts(llmExtractions.summary) + : []; const normalizedSummary = summary?.trim(); - if (normalizedSummary) { - chunks.push({ - chunkIndex: 0, - chunkType: 'summary', - text: normalizedSummary, - startOffset: 0, - endOffset: normalizedSummary.length, - }); - } + const summaryTexts = [ + ...(includeLlm ? extractedSummaryTexts : []), + ...(includeHeuristic && normalizedSummary ? [normalizedSummary] : []), + ]; + chunks.push(...reindexChunks(buildChunksFromTexts('summary', summaryTexts), chunks.length)); - const contentChunks = buildContentChunks(content, maxChars, DEFAULT_OVERLAP_CHARS).map( - (chunk) => ({ - ...chunk, - chunkIndex: chunk.chunkIndex + chunks.length, - }) + const durableFactTexts = llmExtractions?.durable_fact + ? buildDurableFactEmbeddingTexts(llmExtractions.durable_fact) + : []; + const factChunks = [ + ...(includeLlm ? buildChunksFromTexts('fact', durableFactTexts) : []), + ...(includeHeuristic ? buildFactChunks(`${normalizedSummary || ''}\n${content}`) : []), + ]; + chunks.push(...reindexChunks(factChunks, chunks.length)); + chunks.push( + ...reindexChunks(buildTopicChunks({ topicKey, topics, source, salience }), chunks.length) ); - return [...chunks, ...contentChunks]; + const entityTexts = llmExtractions?.entity + ? buildEntityEmbeddingTexts(llmExtractions.entity) + : []; + const entityChunks = [ + ...(includeLlm ? buildChunksFromTexts('entity', entityTexts) : []), + ...(includeHeuristic ? buildEntityChunks({ summary, content, topicKey, topics }) : []), + ]; + chunks.push(...reindexChunks(entityChunks, chunks.length)); + + const currentStateTexts = llmExtractions?.current_state + ? buildCurrentStateEmbeddingTexts(llmExtractions.current_state) + : []; + if (includeLlm) { + chunks.push( + ...reindexChunks(buildChunksFromTexts('current_state', currentStateTexts), chunks.length) + ); + } + chunks.push( + ...reindexChunks(buildContentChunks(content, maxChars, DEFAULT_OVERLAP_CHARS), chunks.length) + ); + + return chunks; } export function formatVectorLiteral(vector: number[]): string { @@ -119,7 +455,7 @@ export function buildChunkRows(params: { user_id: userId, chunk_index: chunk.chunkIndex, chunk_type: chunk.chunkType, - chunk_text: chunk.text, + chunk_text: sanitizeChunkText(chunk.text), embedding: formatVectorLiteral(chunk.embedding.vector), metadata: { embedding: { @@ -138,16 +474,20 @@ export function buildChunkMetadataUpdate(params: { provider: string; model: string; chunkCount: number; + viewCounts: MemoryChunkViewCounts; + extractionMode?: MemoryExtractionChunkMode; existingMetadata?: Record | null; }): Record { - const { provider, model, chunkCount, existingMetadata } = params; + const { provider, model, chunkCount, viewCounts, extractionMode, existingMetadata } = params; return { ...(existingMetadata || {}), embedding_chunks: { provider, model, version: MEMORY_EMBEDDING_CHUNKS_VERSION, + ...(extractionMode ? { extractionMode } : {}), chunkCount, + viewCounts, updatedAt: new Date().toISOString(), }, }; diff --git a/packages/api/src/services/memory-benchmark-constants.ts b/packages/api/src/services/memory-benchmark-constants.ts new file mode 100644 index 00000000..4877a84a --- /dev/null +++ b/packages/api/src/services/memory-benchmark-constants.ts @@ -0,0 +1,3 @@ +export const MEMORY_EXTRACTION_VERSION = 2; +export const MEMORY_EMBEDDING_CHUNKS_VERSION = 2; +export const DEFAULT_MEMORY_LLM_MODEL = 'gpt-4.1-mini'; diff --git a/packages/api/src/services/memory-dreaming.test.ts b/packages/api/src/services/memory-dreaming.test.ts new file mode 100644 index 00000000..b7a6fb8d --- /dev/null +++ b/packages/api/src/services/memory-dreaming.test.ts @@ -0,0 +1,110 @@ +import { describe, expect, it } from 'vitest'; +import { + computeChronologyAwareBoost, + extractDreamDurableFacts, + findDreamDuplicateCandidates, + findDreamSupersessionCandidates, + queryHasChronologyIntent, +} from './memory-dreaming'; + +describe('memory-dreaming', () => { + it('detects chronology intent in current-policy style queries', () => { + expect(queryHasChronologyIntent('what is the current escalation policy?')).toBe(true); + expect(queryHasChronologyIntent('find the latest override')).toBe(true); + expect(queryHasChronologyIntent('who owns the notebook')).toBe(false); + }); + + it('boosts newer override memories when chronology intent is present', () => { + const minCreatedAt = new Date('2026-03-01T00:00:00Z'); + const maxCreatedAt = new Date('2026-03-20T00:00:00Z'); + + const olderBoost = computeChronologyAwareBoost({ + query: 'what is the current policy override', + memory: { + content: 'Older policy for wound-care escalation.', + summary: 'Old escalation policy', + topicKey: 'policy:wound-care', + createdAt: minCreatedAt, + }, + minCreatedAt, + maxCreatedAt, + }); + + const newerBoost = computeChronologyAwareBoost({ + query: 'what is the current policy override', + memory: { + content: 'Current policy overrides the previous wound-care escalation steps.', + summary: 'Current escalation policy', + topicKey: 'policy:wound-care', + createdAt: maxCreatedAt, + }, + minCreatedAt, + maxCreatedAt, + }); + + expect(newerBoost).toBeGreaterThan(olderBoost); + expect(newerBoost).toBeGreaterThan(0); + }); + + it('extracts durable fact candidates from memory text', () => { + const facts = extractDreamDurableFacts({ + summary: 'Current escalation policy', + content: + 'The new wound-care escalation policy replaces the previous triage flow. It requires notifying the inpatient lead within 15 minutes because the older path is deprecated.', + }); + + expect(facts.length).toBeGreaterThan(0); + expect(facts[0]?.text.toLowerCase()).toContain('policy'); + }); + + it('detects likely duplicate memories', () => { + const duplicates = findDreamDuplicateCandidates([ + { + id: 'mem-1', + summary: 'Current escalation policy', + content: 'Current escalation policy for wound-care requires inpatient lead notification.', + topicKey: 'policy:wound-care', + }, + { + id: 'mem-2', + summary: 'Current escalation policy', + content: 'Current escalation policy for wound-care requires inpatient lead notification.', + topicKey: 'policy:wound-care', + }, + ]); + + expect(duplicates).toEqual([ + expect.objectContaining({ + canonicalId: 'mem-1', + duplicateId: 'mem-2', + }), + ]); + }); + + it('detects likely supersession candidates within the same topic', () => { + const supersessions = findDreamSupersessionCandidates([ + { + id: 'mem-1', + summary: 'Old escalation policy', + content: 'Old escalation policy for wound-care uses the triage lead.', + topicKey: 'policy:wound-care', + createdAt: new Date('2026-03-01T00:00:00Z'), + }, + { + id: 'mem-2', + summary: 'Current escalation policy', + content: + 'Current escalation policy replaces the prior wound-care triage flow and now uses the inpatient lead.', + topicKey: 'policy:wound-care', + createdAt: new Date('2026-03-20T00:00:00Z'), + }, + ]); + + expect(supersessions).toEqual([ + expect.objectContaining({ + newerId: 'mem-2', + olderId: 'mem-1', + }), + ]); + }); +}); diff --git a/packages/api/src/services/memory-dreaming.ts b/packages/api/src/services/memory-dreaming.ts new file mode 100644 index 00000000..4404dc20 --- /dev/null +++ b/packages/api/src/services/memory-dreaming.ts @@ -0,0 +1,212 @@ +import type { Memory } from '../data/models/memory'; + +export interface DreamFactCandidate { + text: string; + score: number; +} + +export interface DreamDuplicateCandidate { + canonicalId: string; + duplicateId: string; + similarity: number; +} + +export interface DreamSupersessionCandidate { + newerId: string; + olderId: string; + confidence: number; + reason: 'override-cue' | 'replacement-cue'; +} + +const CHRONOLOGY_QUERY_CUES = [ + 'latest', + 'current', + 'newest', + 'recent', + 'now', + 'override', + 'overrides', + 'supersede', + 'supersedes', + 'replace', + 'replaces', + 'replaced', + 'deprecated', +]; + +const FORWARD_LOOKING_MEMORY_CUES = [ + 'override', + 'overrides', + 'supersede', + 'supersedes', + 'replace', + 'replaces', + 'replaced', + 'current', + 'new policy', + 'now uses', + 'instead of', +]; + +const STALE_MEMORY_CUES = ['deprecated', 'old policy', 'previous policy', 'former']; + +function normalize(text: string): string { + return text.toLowerCase().replace(/\s+/g, ' ').trim(); +} + +function tokenize(text: string): string[] { + return normalize(text) + .split(/[^a-z0-9]+/g) + .filter((token) => token.length > 2); +} + +function jaccard(a: string[], b: string[]): number { + if (a.length === 0 || b.length === 0) return 0; + const aSet = new Set(a); + const bSet = new Set(b); + let intersection = 0; + for (const token of aSet) { + if (bSet.has(token)) intersection += 1; + } + const union = new Set([...aSet, ...bSet]).size; + return union === 0 ? 0 : intersection / union; +} + +function sentenceScore(sentence: string): number { + const lowered = normalize(sentence); + let score = 0; + if (/\d/.test(sentence)) score += 0.2; + if (FORWARD_LOOKING_MEMORY_CUES.some((cue) => lowered.includes(cue))) score += 0.4; + if (/\b(must|should|requires|important|decided|because|policy|convention)\b/.test(lowered)) { + score += 0.3; + } + const tokens = tokenize(lowered); + const uniqueTokens = new Set(tokens); + score += Math.min(0.2, uniqueTokens.size / Math.max(tokens.length, 1)); + return score; +} + +export function queryHasChronologyIntent(query: string): boolean { + const lowered = normalize(query); + return CHRONOLOGY_QUERY_CUES.some((cue) => lowered.includes(cue)); +} + +export function computeChronologyAwareBoost(params: { + query: string; + memory: Pick; + minCreatedAt: Date; + maxCreatedAt: Date; +}): number { + if (!queryHasChronologyIntent(params.query)) return 0; + + const combined = normalize( + [params.memory.summary || '', params.memory.topicKey || '', params.memory.content].join('\n') + ); + const spanMs = Math.max(1, params.maxCreatedAt.getTime() - params.minCreatedAt.getTime()); + const recencyRatio = + (params.memory.createdAt.getTime() - params.minCreatedAt.getTime()) / spanMs; + + let boost = recencyRatio * 0.08; + if (FORWARD_LOOKING_MEMORY_CUES.some((cue) => combined.includes(cue))) boost += 0.05; + if (STALE_MEMORY_CUES.some((cue) => combined.includes(cue))) boost -= 0.03; + + return Math.max(-0.05, Math.min(0.15, boost)); +} + +export function extractDreamDurableFacts(memory: Pick): DreamFactCandidate[] { + const source = [memory.summary || '', memory.content] + .join('\n') + .replace(/\r/g, '\n') + .split(/\n+/) + .flatMap((line) => line.split(/(?<=[.!?])\s+/)) + .map((part) => part.trim()) + .filter((part) => part.length >= 32 && part.length <= 280); + + const ranked = source + .map((text) => ({ text, score: sentenceScore(text) })) + .filter((candidate) => candidate.score > 0.25) + .sort((a, b) => b.score - a.score || b.text.length - a.text.length); + + const deduped: DreamFactCandidate[] = []; + const seen = new Set(); + for (const candidate of ranked) { + const key = normalize(candidate.text); + if (seen.has(key)) continue; + seen.add(key); + deduped.push(candidate); + if (deduped.length >= 5) break; + } + return deduped; +} + +export function findDreamDuplicateCandidates( + memories: Array> +): DreamDuplicateCandidate[] { + const duplicates: DreamDuplicateCandidate[] = []; + + for (let i = 0; i < memories.length; i += 1) { + for (let j = i + 1; j < memories.length; j += 1) { + const left = memories[i]; + const right = memories[j]; + if (left.topicKey && right.topicKey && left.topicKey !== right.topicKey) continue; + + const similarity = jaccard( + tokenize(`${left.summary || ''} ${left.content}`), + tokenize(`${right.summary || ''} ${right.content}`) + ); + if (similarity < 0.82) continue; + + duplicates.push({ + canonicalId: left.id, + duplicateId: right.id, + similarity: Number(similarity.toFixed(4)), + }); + } + } + + return duplicates; +} + +export function findDreamSupersessionCandidates( + memories: Array> +): DreamSupersessionCandidate[] { + const candidates: DreamSupersessionCandidate[] = []; + + const byTopic = new Map>>(); + for (const memory of memories) { + if (!memory.topicKey) continue; + const bucket = byTopic.get(memory.topicKey) || []; + bucket.push(memory); + byTopic.set(memory.topicKey, bucket); + } + + for (const bucket of byTopic.values()) { + const ordered = [...bucket].sort((a, b) => a.createdAt.getTime() - b.createdAt.getTime()); + for (let i = 1; i < ordered.length; i += 1) { + const older = ordered[i - 1]; + const newer = ordered[i]; + const combined = normalize(`${newer.summary || ''} ${newer.content}`); + + const reason = combined.includes('override') || combined.includes('supersede') + ? 'override-cue' + : combined.includes('replace') || combined.includes('instead of') + ? 'replacement-cue' + : null; + if (!reason) continue; + + const tokenSimilarity = jaccard( + tokenize(`${older.summary || ''} ${older.content}`), + tokenize(`${newer.summary || ''} ${newer.content}`) + ); + + candidates.push({ + newerId: newer.id, + olderId: older.id, + confidence: Number(Math.min(0.99, 0.55 + tokenSimilarity * 0.35).toFixed(4)), + reason, + }); + } + } + + return candidates; +} diff --git a/packages/api/src/services/memory-llm-extraction.test.ts b/packages/api/src/services/memory-llm-extraction.test.ts new file mode 100644 index 00000000..58ef4b11 --- /dev/null +++ b/packages/api/src/services/memory-llm-extraction.test.ts @@ -0,0 +1,303 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { + batchMemoryExtractionResponseSchema, + buildBatchExtractionPrompt, + buildCurrentStateEmbeddingTexts, + buildCurrentStateExtractionPrompt, + buildDurableFactEmbeddingTexts, + buildDurableFactExtractionPrompt, + buildEntityEmbeddingTexts, + buildEntityExtractionPrompt, + buildSummaryEmbeddingTexts, + buildSummaryExtractionPrompt, + coerceExtractionPayload, + currentStateExtractionSchema, + durableFactExtractionSchema, + entityExtractionSchema, + MemoryLlmExtractor, + memoryExtractionsSchema, + normalizeMemoryExtractions, + summaryExtractionSchema, +} from './memory-llm-extraction'; + +describe('memory-llm-extraction', () => { + const source = { + summary: 'Discussed dev server behavior and merge process', + content: + 'The current dev server auto-restarts when files change. We decided that every PR needs sibling review before merging to main. Wren and Lumen discussed the memory benchmark document and current architecture.', + topicKey: 'project:ink/memory-benchmarks', + topics: ['person:wren', 'person:lumen', 'process:pr-review'], + source: 'observation', + salience: 'high', + }; + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('builds an entity prompt with explicit extraction guidance', () => { + const prompt = buildEntityExtractionPrompt(source); + expect(prompt.kind).toBe('entity'); + expect(prompt.systemPrompt).toContain('strict JSON'); + expect(prompt.userPrompt).toContain('Return at most 8 entities'); + expect(prompt.userPrompt).toContain('Memory text'); + }); + + it('builds a durable fact prompt with decision and process emphasis', () => { + const prompt = buildDurableFactExtractionPrompt(source); + expect(prompt.kind).toBe('durable_fact'); + expect(prompt.userPrompt).toContain('decision'); + expect(prompt.userPrompt).toContain('process'); + expect(prompt.schemaDescription).toContain('durableFacts'); + }); + + it('builds a summary prompt oriented to actionability', () => { + const prompt = buildSummaryExtractionPrompt(source); + expect(prompt.kind).toBe('summary'); + expect(prompt.userPrompt).toContain('actionRelevance'); + expect(prompt.systemPrompt).toContain('decision support'); + }); + + it('builds a current-state prompt for volatile operational status', () => { + const prompt = buildCurrentStateExtractionPrompt(source); + expect(prompt.kind).toBe('current_state'); + expect(prompt.userPrompt).toContain('present or near-present operational state'); + expect(prompt.userPrompt).toContain('dev server auto-restarts'); + }); + + it('builds a batch extraction prompt that keeps memories independent', () => { + const prompt = buildBatchExtractionPrompt( + [ + { memoryId: 'memory-a', source }, + { + memoryId: 'memory-b', + source: { + ...source, + content: 'A separate memory mentioned Conor and the benchmark plan.', + }, + }, + ], + ['entity', 'durable_fact', 'summary'] + ); + + expect(prompt.systemPrompt).toContain('Treat each memory independently'); + expect(prompt.userPrompt).toContain('memory-a'); + expect(prompt.userPrompt).toContain('memory-b'); + expect(prompt.userPrompt).toContain('summarize only that single source memory'); + expect(prompt.schemaDescription).toContain('"results"'); + expect(prompt.schemaDescription).toContain('"durable_fact"'); + }); + + it('formats embedding texts from structured entity extraction', () => { + const parsed = entityExtractionSchema.parse({ + entities: [ + { + name: 'Wren', + aliases: ['wren'], + entityType: 'person', + description: 'Collaborator reviewing benchmark architecture', + evidence: 'Wren and Lumen discussed the memory benchmark document.', + }, + ], + }); + + expect(buildEntityEmbeddingTexts(parsed)).toEqual([ + expect.stringContaining('entity: Wren; type: person'), + ]); + }); + + it('formats embedding texts from durable fact extraction', () => { + const parsed = durableFactExtractionSchema.parse({ + durableFacts: [ + { + fact: 'Every PR needs sibling review before merging to main.', + category: 'process', + subject: 'PR', + object: 'sibling review', + evidence: 'We decided that every PR needs sibling review before merging to main.', + }, + ], + }); + + expect(buildDurableFactEmbeddingTexts(parsed)).toEqual([ + expect.stringContaining('category: process'), + ]); + }); + + it('formats embedding texts from summary extraction', () => { + const parsed = summaryExtractionSchema.parse({ + summary: 'The team discussed benchmark architecture and merge process constraints.', + keyPoints: ['dev server auto-restarts', 'sibling review before merge'], + actionRelevance: 'Helps future agents follow merge process and interpret current dev state.', + }); + + expect(buildSummaryEmbeddingTexts(parsed)[0]).toContain('action relevance'); + }); + + it('formats embedding texts from current-state extraction', () => { + const parsed = currentStateExtractionSchema.parse({ + state: 'Dev server auto-restarts on file change.', + scope: 'local dev server', + status: 'running', + volatility: 'volatile', + evidence: 'The current dev server auto-restarts when files change.', + }); + + expect(buildCurrentStateEmbeddingTexts(parsed)[0]).toContain('volatility: volatile'); + }); + + it('normalizes extraction metadata', () => { + const normalized = normalizeMemoryExtractions({ + version: 1, + provider: 'openai', + model: 'gpt-4.1-mini', + extractedAt: '2026-04-18T12:00:00.000Z', + summary: { + summary: 'Summary text', + keyPoints: [], + actionRelevance: 'Useful later', + }, + }); + + expect(normalized).toEqual( + memoryExtractionsSchema.parse({ + version: 1, + provider: 'openai', + model: 'gpt-4.1-mini', + extractedAt: '2026-04-18T12:00:00.000Z', + summary: { + summary: 'Summary text', + keyPoints: [], + actionRelevance: 'Useful later', + }, + }) + ); + }); + + it('parses batched extraction responses', () => { + const parsed = batchMemoryExtractionResponseSchema.parse({ + results: [ + { + memoryId: 'memory-a', + entity: { entities: [] }, + durable_fact: { durableFacts: [] }, + summary: { + summary: 'A memory about benchmark architecture.', + keyPoints: ['benchmark architecture'], + actionRelevance: 'Helps retrieve the benchmark discussion later.', + }, + }, + ], + }); + + expect(parsed.results[0]?.memoryId).toBe('memory-a'); + expect(parsed.results[0]?.summary?.summary).toContain('benchmark architecture'); + }); + + it('coerces common runner schema drift instead of failing whole batches', () => { + const rawSummary = { + summary: 'A memory about meal prep.', + keyPoints: ['a', 'b', 'c', 'd', 'e', 'f', 'g'], + actionRelevance: 'Helps answer food planning questions.', + }; + const entity = coerceExtractionPayload('entity', { + entities: [ + { + name: 'Insurance underwriting process', + aliases: [ + 'underwriting', + 'risk review', + 'extra alias', + 'another', + 'fifth', + 'sixth', + 'seventh', + ], + entityType: 'process', + description: 'The insurer risk review flow.', + evidence: 'The insurer is completing underwriting.', + }, + ], + }); + const summary = coerceExtractionPayload('summary', rawSummary); + + expect(entity?.entities[0]?.entityType).toBe('other'); + expect(entity?.entities[0]?.aliases).toHaveLength(6); + expect(summary?.keyPoints).toHaveLength(6); + expect(rawSummary.keyPoints).toHaveLength(7); + }); + + it('runs enabled extraction kinds and returns typed metadata', async () => { + const fetchMock = vi + .spyOn(globalThis, 'fetch') + .mockResolvedValueOnce( + new Response( + JSON.stringify({ + choices: [ + { + message: { + content: JSON.stringify({ + entities: [ + { + name: 'Wren', + aliases: ['wren'], + entityType: 'person', + description: 'Reviewer', + evidence: 'Wren reviewed the benchmark.', + }, + ], + }), + }, + }, + ], + }), + { status: 200 } + ) + ) + .mockResolvedValueOnce( + new Response( + JSON.stringify({ + choices: [ + { + message: { + content: JSON.stringify({ + summary: 'Benchmark review covered feature flags.', + keyPoints: [ + 'feature flags', + 'typed indexes', + 'entity view', + 'durable fact view', + 'summary view', + 'current state view', + 'raw overflow should persist', + ], + actionRelevance: 'Helps route future experiments.', + }), + }, + }, + ], + }), + { status: 200 } + ) + ); + + const extractor = new MemoryLlmExtractor({ + enabled: true, + model: 'gpt-4.1-mini', + baseUrl: 'https://api.openai.com', + hasApiKey: true, + maxInputChars: 5000, + enabledKinds: ['entity', 'summary'], + }); + + const result = await extractor.extract(source); + + expect(fetchMock).toHaveBeenCalledTimes(2); + expect(result?.provider).toBe('openai'); + expect(result?.entity?.entities[0]?.name).toBe('Wren'); + expect(result?.summary?.summary).toContain('Benchmark review'); + expect(result?.summary?.keyPoints).toHaveLength(6); + expect((result?.raw?.summary as { keyPoints?: string[] })?.keyPoints).toHaveLength(7); + expect(result?.durable_fact).toBeUndefined(); + }); +}); diff --git a/packages/api/src/services/memory-llm-extraction.ts b/packages/api/src/services/memory-llm-extraction.ts new file mode 100644 index 00000000..153580ef --- /dev/null +++ b/packages/api/src/services/memory-llm-extraction.ts @@ -0,0 +1,681 @@ +import { env } from '../config/env'; +import { logger } from '../utils/logger'; +import { z } from 'zod'; +import { DEFAULT_MEMORY_LLM_MODEL, MEMORY_EXTRACTION_VERSION } from './memory-benchmark-constants'; + +export { DEFAULT_MEMORY_LLM_MODEL, MEMORY_EXTRACTION_VERSION }; + +export const entityExtractionItemSchema = z.object({ + name: z.string().min(1), + aliases: z.array(z.string().min(1)).max(6).default([]), + entityType: z.enum([ + 'person', + 'org', + 'project', + 'product', + 'place', + 'policy', + 'service', + 'file', + 'other', + ]), + description: z.string().min(1), + evidence: z.string().min(1), +}); + +export const durableFactExtractionItemSchema = z.object({ + fact: z.string().min(1), + category: z.enum([ + 'identity', + 'preference', + 'decision', + 'constraint', + 'process', + 'status', + 'ownership', + 'relationship', + 'other', + ]), + subject: z.string().optional(), + object: z.string().optional(), + evidence: z.string().min(1), +}); + +export const summaryExtractionSchema = z.object({ + summary: z.string().min(1), + keyPoints: z.array(z.string().min(1)).max(6).default([]), + actionRelevance: z.string().min(1), +}); + +export const currentStateExtractionSchema = z.object({ + state: z.string().min(1), + scope: z.string().min(1), + status: z.string().min(1), + volatility: z.enum(['volatile', 'semi-stable', 'stable']), + evidence: z.string().min(1), +}); + +export const entityExtractionSchema = z.object({ + entities: z.array(entityExtractionItemSchema).max(8), +}); + +export const durableFactExtractionSchema = z.object({ + durableFacts: z.array(durableFactExtractionItemSchema).max(10), +}); + +export const memoryExtractionsSchema = z.object({ + version: z.number().int().default(MEMORY_EXTRACTION_VERSION), + provider: z.string().min(1), + model: z.string().min(1), + extractedAt: z.string().min(1), + entity: entityExtractionSchema.optional(), + durable_fact: durableFactExtractionSchema.optional(), + summary: summaryExtractionSchema.optional(), + current_state: currentStateExtractionSchema.optional(), + raw: z + .object({ + provider: z.string().optional(), + model: z.string().optional(), + extractedAt: z.string().optional(), + entity: z.unknown().optional(), + durable_fact: z.unknown().optional(), + summary: z.unknown().optional(), + current_state: z.unknown().optional(), + }) + .passthrough() + .optional(), +}); + +export type MemoryExtractions = z.infer; + +export const batchMemoryExtractionResultSchema = z + .object({ + memoryId: z.string().min(1), + }) + .passthrough(); + +export const batchMemoryExtractionResponseSchema = z.object({ + results: z.array(batchMemoryExtractionResultSchema), +}); + +export interface MemoryExtractionSource { + summary?: string | null; + content: string; + topicKey?: string | null; + topics?: string[] | null; + source?: string | null; + salience?: string | null; +} + +export type ExtractionKind = 'entity' | 'durable_fact' | 'summary' | 'current_state'; + +export interface BatchMemoryExtractionSource { + memoryId: string; + source: MemoryExtractionSource; +} + +export interface ExtractionPromptBundle { + kind: ExtractionKind; + systemPrompt: string; + userPrompt: string; + schemaDescription: string; +} + +export interface ExtractionRuntimeConfig { + enabled: boolean; + model: string; + baseUrl: string; + hasApiKey: boolean; + maxInputChars: number; + enabledKinds: ExtractionKind[]; +} + +const DEFAULT_OPENAI_BASE_URL = 'https://api.openai.com'; + +function buildSourceBlock(source: MemoryExtractionSource): string { + const parts: string[] = []; + if (source.summary?.trim()) parts.push(`Summary:\n${source.summary.trim()}`); + if (source.topicKey?.trim()) parts.push(`Topic key: ${source.topicKey.trim()}`); + const topics = (source.topics || []).map((t) => t.trim()).filter(Boolean); + if (topics.length > 0) parts.push(`Topics: ${topics.join(', ')}`); + if (source.source?.trim()) parts.push(`Source: ${source.source.trim()}`); + if (source.salience?.trim()) parts.push(`Salience: ${source.salience.trim()}`); + parts.push(`Memory text:\n${source.content.trim()}`); + return parts.join('\n\n'); +} + +function clampSourceText(text: string, maxChars: number): string { + if (text.length <= maxChars) return text; + return `${text.slice(0, Math.max(0, maxChars - 1)).trimEnd()}…`; +} + +function compactWhitespace(text: string): string { + return text.replace(/\s+/g, ' ').trim(); +} + +function quote(text: string, maxChars = 220): string { + const normalized = compactWhitespace(text); + if (normalized.length <= maxChars) return normalized; + return `${normalized.slice(0, maxChars - 1).trimEnd()}…`; +} + +export function buildEntityExtractionPrompt( + source: MemoryExtractionSource +): ExtractionPromptBundle { + return { + kind: 'entity', + systemPrompt: + 'You extract explicit entity memory from a single memory record. Return strict JSON only. Do not speculate. Only extract entities clearly supported by the text. Prefer entities that help answer future who/what questions. Each entity must include a short grounded description and a direct evidence quote.', + schemaDescription: + 'JSON schema: {"entities": [{"name": string, "aliases": string[], "entityType": "person"|"org"|"project"|"product"|"place"|"policy"|"service"|"file"|"other", "description": string, "evidence": string}]}', + userPrompt: [ + 'Extraction type: entity', + 'Task:', + '- Extract the main people, orgs, projects, products, places, policies, services, or files explicitly mentioned.', + '- Preserve distinctive item-property relationships in descriptions when useful, e.g. an app that uses mnemonics, a tool used for a task, a named dish with its ingredients, or a named document with its purpose.', + '- Include aliases only if the text supports them.', + '- Ignore generic nouns that are not useful routing anchors.', + '- Return at most 8 entities.', + '', + buildSourceBlock(source), + ].join('\n'), + }; +} + +export function buildDurableFactExtractionPrompt( + source: MemoryExtractionSource +): ExtractionPromptBundle { + return { + kind: 'durable_fact', + systemPrompt: + 'You extract durable facts from a single memory record. Return strict JSON only. Durable facts are long-lived facts, decisions, constraints, process rules, status conditions, ownership facts, relationship facts, or preferences likely to matter later. Do not include fleeting chatter. Do not speculate. Every fact must quote evidence from the memory.', + schemaDescription: + 'JSON schema: {"durableFacts": [{"fact": string, "category": "identity"|"preference"|"decision"|"constraint"|"process"|"status"|"ownership"|"relationship"|"other", "subject"?: string, "object"?: string, "evidence": string}]}', + userPrompt: [ + 'Extraction type: durable_fact', + 'Task:', + '- Extract stable, decision-relevant facts from the memory.', + '- Also extract precise answerable details from assistant responses when they may be asked about later: named recommendations, list-item mappings, durations, quantities, colors, URLs, ordinal entries, procedures, constraints, code/chords, or exact quoted values.', + '- Prefer facts that would help answer who / what / why / constraint / process / status questions later.', + '- Use the most specific category available, including decision and process when applicable.', + '- Return at most 10 durable facts.', + '', + buildSourceBlock(source), + ].join('\n'), + }; +} + +export function buildSummaryExtractionPrompt( + source: MemoryExtractionSource +): ExtractionPromptBundle { + return { + kind: 'summary', + systemPrompt: + 'You write a compact retrieval-oriented summary for a single memory record. Return strict JSON only. The summary should optimize for future decision support and actionability, not literary style. Keep it source-grounded.', + schemaDescription: + 'JSON schema: {"summary": string, "keyPoints": string[], "actionRelevance": string}', + userPrompt: [ + 'Extraction type: summary', + 'Task:', + '- Produce a short holistic recap of the memory.', + '- Keep the summary focused on what happened, what matters, and why it may matter later.', + '- keyPoints should capture the most important supporting points, including exact values, names, durations, URLs, or list-item mappings when those are likely future follow-up targets.', + '- actionRelevance should state how this memory could help a future decision or action.', + '', + buildSourceBlock(source), + ].join('\n'), + }; +} + +export function buildCurrentStateExtractionPrompt( + source: MemoryExtractionSource +): ExtractionPromptBundle { + return { + kind: 'current_state', + systemPrompt: + 'You extract current-state memory from a single memory record. Return strict JSON only. Current state is volatile operational status that may change soon, such as server state, active branch state, current blocker, or live workflow status. Do not convert stable historical facts into current state. Include volatility and direct evidence.', + schemaDescription: + 'JSON schema: {"state": string, "scope": string, "status": string, "volatility": "volatile"|"semi-stable"|"stable", "evidence": string}', + userPrompt: [ + 'Extraction type: current_state', + 'Task:', + '- Extract only if the memory contains a present or near-present operational state.', + '- Good examples: dev server auto-restarts, current test server port, current blocker, current rollout status.', + '- Bad examples: old decisions or historical facts with no present-state implication.', + '', + buildSourceBlock(source), + ].join('\n'), + }; +} + +export function buildExtractionPrompt( + source: MemoryExtractionSource, + kind: ExtractionKind +): ExtractionPromptBundle { + switch (kind) { + case 'entity': + return buildEntityExtractionPrompt(source); + case 'durable_fact': + return buildDurableFactExtractionPrompt(source); + case 'summary': + return buildSummaryExtractionPrompt(source); + case 'current_state': + return buildCurrentStateExtractionPrompt(source); + } +} + +export function buildBatchExtractionPrompt( + items: BatchMemoryExtractionSource[], + kinds: ExtractionKind[] +): Omit { + const uniqueKinds = [...new Set(kinds)]; + const requestedSchemas = uniqueKinds.map((kind) => { + switch (kind) { + case 'entity': + return '"entity": {"entities": [{"name": string, "aliases": string[], "entityType": "person"|"org"|"project"|"product"|"place"|"policy"|"service"|"file"|"other", "description": string, "evidence": string}]}'; + case 'durable_fact': + return '"durable_fact": {"durableFacts": [{"fact": string, "category": "identity"|"preference"|"decision"|"constraint"|"process"|"status"|"ownership"|"relationship"|"other", "subject"?: string, "object"?: string, "evidence": string}]}'; + case 'summary': + return '"summary": {"summary": string, "keyPoints": string[], "actionRelevance": string}'; + case 'current_state': + return '"current_state": {"state": string, "scope": string, "status": string, "volatility": "volatile"|"semi-stable"|"stable", "evidence": string}'; + } + }); + + return { + systemPrompt: + 'You are a deterministic batched memory extraction worker. Return strict JSON only. Do not use tools. Treat each memory independently: do not synthesize across memories, do not infer from neighboring memories, and do not use benchmark labels. Each extracted item must be grounded in the memory it belongs to.', + schemaDescription: `JSON schema: {"results": [{"memoryId": string, ${requestedSchemas.join(', ')}}]}`, + userPrompt: [ + `Extraction types: ${uniqueKinds.join(', ')}`, + 'Task:', + '- Return exactly one result object for each input memoryId.', + '- Do not fill quotas. Extract only salient items likely to improve future retrieval or decision support.', + '- For entity extraction: extract at most 8 explicit people, orgs, projects, products, places, policies, services, or files per memory; prefer 2-5 high-signal entities and use an empty entities array if none are useful.', + '- Entity descriptions should preserve distinctive item-property relationships when useful, e.g. an app that uses mnemonics, a tool used for a task, a named dish with its ingredients, or a named document with its purpose.', + '- For durable_fact extraction: extract at most 10 long-lived facts, decisions, constraints, process rules, status conditions, ownership facts, relationship facts, or preferences per memory; prefer 2-6 high-signal facts and use an empty durableFacts array if none are useful.', + '- Durable facts should also preserve precise answerable details from assistant responses when they may be asked about later: named recommendations, list-item mappings, durations, quantities, colors, URLs, ordinal entries, procedures, constraints, code/chords, or exact quoted values.', + '- For summary extraction: summarize only that single source memory. Do not aggregate across the batch. Optimize for future retrieval and decision support.', + '- Summary keyPoints should include exact values, names, durations, URLs, or list-item mappings when those are likely future follow-up targets.', + '- For current_state extraction: include only present or near-present operational state supported by that memory.', + '- Evidence must quote or closely paraphrase text from the same memory.', + '', + 'Input memories JSON:', + JSON.stringify( + items.map(({ memoryId, source }) => ({ + memoryId, + summary: source.summary || null, + topicKey: source.topicKey || null, + topics: source.topics || [], + memorySource: source.source || null, + salience: source.salience || null, + content: source.content, + })), + null, + 2 + ), + ].join('\n'), + }; +} + +export function buildEntityEmbeddingTexts( + payload: z.infer +): string[] { + return payload.entities.map((entity) => + compactWhitespace( + `entity: ${entity.name}; type: ${entity.entityType}; aliases: ${entity.aliases.join(', ') || 'none'}; description: ${entity.description}; evidence: ${quote(entity.evidence)}` + ) + ); +} + +export function buildDurableFactEmbeddingTexts( + payload: z.infer +): string[] { + return payload.durableFacts.map((fact) => + compactWhitespace( + `durable fact: ${fact.fact}; category: ${fact.category}; subject: ${fact.subject || 'unknown'}; object: ${fact.object || 'unknown'}; evidence: ${quote(fact.evidence)}` + ) + ); +} + +export function buildSummaryEmbeddingTexts( + payload: z.infer +): string[] { + return [ + compactWhitespace( + `summary: ${payload.summary}; key points: ${payload.keyPoints.join(' | ') || 'none'}; action relevance: ${payload.actionRelevance}` + ), + ]; +} + +export function buildCurrentStateEmbeddingTexts( + payload: z.infer +): string[] { + return [ + compactWhitespace( + `current state: ${payload.state}; scope: ${payload.scope}; status: ${payload.status}; volatility: ${payload.volatility}; evidence: ${quote(payload.evidence)}` + ), + ]; +} + +export function normalizeMemoryExtractions(value: unknown): MemoryExtractions | null { + const parsed = memoryExtractionsSchema.safeParse(value); + return parsed.success ? parsed.data : null; +} + +const ENTITY_TYPES = new Set(entityExtractionItemSchema.shape.entityType.options); +const DURABLE_FACT_CATEGORIES = new Set(durableFactExtractionItemSchema.shape.category.options); +const VOLATILITY_VALUES = new Set(currentStateExtractionSchema.shape.volatility.options); + +function asRecord(value: unknown): Record { + return value && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : {}; +} + +function asNonEmptyString(value: unknown): string | null { + return typeof value === 'string' && value.trim() ? value.trim() : null; +} + +function asStringArray(value: unknown, maxItems: number): string[] { + if (!Array.isArray(value)) return []; + return value + .map((item) => asNonEmptyString(item)) + .filter((item): item is string => Boolean(item)) + .slice(0, maxItems); +} + +function coerceEntityExtraction(raw: unknown): z.infer { + const rawEntities = asRecord(raw).entities; + const entities = (Array.isArray(rawEntities) ? rawEntities : []) + .map((item) => { + const record = asRecord(item); + const name = asNonEmptyString(record.name); + const description = asNonEmptyString(record.description); + const evidence = asNonEmptyString(record.evidence); + if (!name || !description || !evidence) return null; + const rawType = asNonEmptyString(record.entityType); + return { + name, + aliases: asStringArray(record.aliases, 6), + entityType: rawType && ENTITY_TYPES.has(rawType as never) ? rawType : 'other', + description, + evidence, + }; + }) + .filter((item): item is z.infer => Boolean(item)) + .slice(0, 8); + return entityExtractionSchema.parse({ entities }); +} + +function coerceDurableFactExtraction(raw: unknown): z.infer { + const rawFacts = asRecord(raw).durableFacts; + const durableFacts = (Array.isArray(rawFacts) ? rawFacts : []) + .map((item) => { + const record = asRecord(item); + const fact = asNonEmptyString(record.fact); + const evidence = asNonEmptyString(record.evidence); + if (!fact || !evidence) return null; + const rawCategory = asNonEmptyString(record.category); + return { + fact, + category: + rawCategory && DURABLE_FACT_CATEGORIES.has(rawCategory as never) ? rawCategory : 'other', + ...(asNonEmptyString(record.subject) + ? { subject: asNonEmptyString(record.subject) as string } + : {}), + ...(asNonEmptyString(record.object) + ? { object: asNonEmptyString(record.object) as string } + : {}), + evidence, + }; + }) + .filter((item): item is z.infer => Boolean(item)) + .slice(0, 10); + return durableFactExtractionSchema.parse({ durableFacts }); +} + +function coerceSummaryExtraction(raw: unknown): z.infer { + const record = asRecord(raw); + const summary = asNonEmptyString(record.summary) || 'No salient summary extracted.'; + const actionRelevance = + asNonEmptyString(record.actionRelevance) || 'Useful for future retrieval and decision support.'; + return summaryExtractionSchema.parse({ + summary, + keyPoints: asStringArray(record.keyPoints, 6), + actionRelevance, + }); +} + +function coerceCurrentStateExtraction(raw: unknown): z.infer { + const record = asRecord(raw); + const state = asNonEmptyString(record.state) || 'No current state extracted.'; + const scope = asNonEmptyString(record.scope) || 'unknown'; + const status = asNonEmptyString(record.status) || 'unknown'; + const rawVolatility = asNonEmptyString(record.volatility); + return currentStateExtractionSchema.parse({ + state, + scope, + status, + volatility: + rawVolatility && VOLATILITY_VALUES.has(rawVolatility as never) + ? rawVolatility + : 'semi-stable', + evidence: asNonEmptyString(record.evidence) || state, + }); +} + +export function coerceExtractionPayload( + kind: ExtractionKind, + raw: unknown +): MemoryExtractions[ExtractionKind] { + switch (kind) { + case 'entity': + return coerceEntityExtraction(raw); + case 'durable_fact': + return coerceDurableFactExtraction(raw); + case 'summary': + return coerceSummaryExtraction(raw); + case 'current_state': + return coerceCurrentStateExtraction(raw); + } +} + +function assignExtractionPayload( + payload: Partial, + kind: ExtractionKind, + result: MemoryExtractions[ExtractionKind] +) { + switch (kind) { + case 'entity': + payload.entity = result as MemoryExtractions['entity']; + break; + case 'durable_fact': + payload.durable_fact = result as MemoryExtractions['durable_fact']; + break; + case 'summary': + payload.summary = result as MemoryExtractions['summary']; + break; + case 'current_state': + payload.current_state = result as MemoryExtractions['current_state']; + break; + } +} + +function assignRawExtractionPayload( + payload: Partial, + kind: ExtractionKind, + raw: unknown +) { + payload.raw = { + ...(payload.raw || {}), + [kind]: raw, + }; +} + +interface ExtractionResult { + normalized: MemoryExtractions[ExtractionKind]; + raw: unknown; +} + +function buildRuntimeConfig(): ExtractionRuntimeConfig { + const enabledKinds: ExtractionKind[] = []; + if (env.MEMORY_LLM_ENTITY_ENABLED) enabledKinds.push('entity'); + if (env.MEMORY_LLM_DURABLE_FACT_ENABLED) enabledKinds.push('durable_fact'); + if (env.MEMORY_LLM_SUMMARY_ENABLED) enabledKinds.push('summary'); + if (env.MEMORY_LLM_CURRENT_STATE_ENABLED) enabledKinds.push('current_state'); + + return { + enabled: env.MEMORY_LLM_EXTRACTION_ENABLED, + model: env.MEMORY_LLM_MODEL || DEFAULT_MEMORY_LLM_MODEL, + baseUrl: env.OPENAI_BASE_URL || DEFAULT_OPENAI_BASE_URL, + hasApiKey: Boolean(env.OPENAI_API_KEY), + maxInputChars: env.MEMORY_LLM_MAX_INPUT_CHARS, + enabledKinds, + }; +} + +function sanitizeSource( + source: MemoryExtractionSource, + maxInputChars: number +): MemoryExtractionSource { + return { + ...source, + summary: source.summary + ? clampSourceText(source.summary, Math.min(maxInputChars, 2000)) + : source.summary, + content: clampSourceText(source.content, maxInputChars), + }; +} + +function extractJsonObject(text: string): string { + const trimmed = text.trim(); + if (trimmed.startsWith('{') && trimmed.endsWith('}')) return trimmed; + const firstBrace = trimmed.indexOf('{'); + const lastBrace = trimmed.lastIndexOf('}'); + if (firstBrace === -1 || lastBrace === -1 || lastBrace <= firstBrace) { + throw new Error('LLM extraction response did not contain a JSON object'); + } + return trimmed.slice(firstBrace, lastBrace + 1); +} + +export class MemoryLlmExtractor { + private readonly config: ExtractionRuntimeConfig; + + constructor(config: ExtractionRuntimeConfig = buildRuntimeConfig()) { + this.config = config; + } + + isEnabled(): boolean { + return this.config.enabled && this.config.enabledKinds.length > 0; + } + + getEnabledKinds(): ExtractionKind[] { + return [...this.config.enabledKinds]; + } + + async extract(source: MemoryExtractionSource): Promise { + if (!this.isEnabled()) return null; + if (!this.config.hasApiKey) { + logger.warn('Memory LLM extraction enabled without OPENAI_API_KEY; skipping extraction', { + model: this.config.model, + enabledKinds: this.config.enabledKinds, + }); + return null; + } + + const sanitizedSource = sanitizeSource(source, this.config.maxInputChars); + const entries = await Promise.all( + this.config.enabledKinds.map( + async (kind) => [kind, await this.extractKind(kind, sanitizedSource)] as const + ) + ); + + const extractedAt = new Date().toISOString(); + const payload: Partial = { + version: MEMORY_EXTRACTION_VERSION, + provider: 'openai', + model: this.config.model, + extractedAt, + raw: { + provider: 'openai', + model: this.config.model, + extractedAt, + }, + }; + + for (const [kind, result] of entries) { + if (!result) continue; + assignExtractionPayload(payload, kind, result.normalized); + assignRawExtractionPayload(payload, kind, result.raw); + } + + const normalized = normalizeMemoryExtractions(payload); + return normalized && + Object.keys(normalized).some((key) => + ['entity', 'durable_fact', 'summary', 'current_state'].includes(key) + ) + ? normalized + : null; + } + + private async extractKind( + kind: ExtractionKind, + source: MemoryExtractionSource + ): Promise { + const prompt = buildExtractionPrompt(source, kind); + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), 20_000); + + try { + const response = await fetch(`${this.config.baseUrl}/v1/chat/completions`, { + method: 'POST', + headers: { + 'content-type': 'application/json', + authorization: `Bearer ${env.OPENAI_API_KEY}`, + }, + body: JSON.stringify({ + model: this.config.model, + temperature: 0, + response_format: { type: 'json_object' }, + messages: [ + { + role: 'system', + content: `${prompt.systemPrompt}\n${prompt.schemaDescription}`, + }, + { + role: 'user', + content: prompt.userPrompt, + }, + ], + }), + signal: controller.signal, + }); + + if (!response.ok) { + throw new Error(`Memory LLM extraction failed (${response.status})`); + } + + const data = (await response.json()) as { + choices?: Array<{ message?: { content?: string | null } }>; + }; + const content = data.choices?.[0]?.message?.content; + if (!content?.trim()) throw new Error('Memory LLM extraction returned empty content'); + + const parsedJson = JSON.parse(extractJsonObject(content)); + return { + normalized: coerceExtractionPayload(kind, parsedJson), + raw: parsedJson, + }; + } catch (error) { + logger.warn('Memory LLM extraction failed for kind', { + kind, + model: this.config.model, + error: error instanceof Error ? error.message : String(error), + }); + return null; + } finally { + clearTimeout(timeout); + } + } +} diff --git a/packages/benchmarks/README.md b/packages/benchmarks/README.md new file mode 100644 index 00000000..ee43503d --- /dev/null +++ b/packages/benchmarks/README.md @@ -0,0 +1,56 @@ +# Memory benchmark terms + +This package is for experimental memory-system benchmarks only. It must not be bundled into production packages. + +## Terms we use precisely + +- **Benchmark case**: one dataset question plus its source sessions/documents. Example: one LongMemEval question. +- **Source memory**: one memory row created from benchmark source text. For LongMemEval this is usually one session transcript from the case haystack. Source memories are raw evidence, not extracted facts. +- **Seed pass**: creates source memory rows for benchmark cases and records their memory IDs in a `*.seed.json` file. A seed pass does not mean entity/fact/summary extraction happened. +- **Seed ID**: a stable label used to group source memories from one seed pass. The script writes topics like `benchmark:memory-recall:` so later extraction/backfill jobs can target exactly that corpus. +- **Extraction pass**: reads source memories and asks a configured backend to produce structured JSON views: `entity`, `durable_fact`, `summary`, and/or `current_state`. +- **Extraction audit file**: JSONL output from `extract-memory-llm-views` containing the exact extracted payload and exact strings that will be embedded. This is the human-inspectable record of what the model produced. +- **Embedding/backfill pass**: embeds source memory content plus any saved extraction views and writes vectors/chunks. This is separate from seeding and extraction. +- **Recall pass**: runs benchmark queries against an already seeded and embedded corpus. + +Use `MEMORY_LLM_EXTRACT_MEMORY_ID=` or `BACKFILL_MEMORY_ID=` when an experiment should touch exactly one source memory. Use the seed/case topics when it should touch a whole corpus or case. + +## Recall mode definitions + +- **text**: lexical/text search over memory rows. +- **semantic**: vector search over the selected embedding chunks. +- **hybrid**: text search + one semantic search, merged and reranked by weighted text/semantic score. Hybrid does not mean multi-view routing. +- **multi-view router**: an experimental hybrid option that separately queries derived chunks and content chunks, then applies optional chunk-type, multi-view, and chronology boosts. This is not the default meaning of hybrid. + +## LLM extraction backends + +Extraction should normally use a subscription-backed CLI runner rather than direct provider API billing: + +```bash +MEMORY_LLM_EXTRACT_BACKEND=claude # or codex +MEMORY_LLM_EXTRACTION_ENABLED=true +MEMORY_LLM_ENTITY_ENABLED=true +MEMORY_LLM_DURABLE_FACT_ENABLED=false +MEMORY_LLM_SUMMARY_ENABLED=false +yarn workspace @inklabs/api extract:memory-llm-views +``` + +`MEMORY_LLM_EXTRACT_BACKEND=direct` is the direct OpenAI-compatible HTTP path and requires `OPENAI_API_KEY`. Runner-backed extraction uses the existing Claude/Codex CLI runners and writes the same audit JSONL. + +## Minimal controlled experiment shape + +The memory experiment pipeline has three conceptual steps: + +1. **Extract**: use our prompts to derive the salient view for one axis of investigation, e.g. `entity`, `durable_fact`, `summary`, or `current_state`. This is not a mechanical preprocessing step; it is part of the research surface and can strongly affect benchmark quality. +2. **Embed**: generate vectors for the exact extracted strings we plan to query later. +3. **Store for querying**: persist the extracted payloads and vectors in the database so recall can use them. Steps 2 and 3 can happen in the same backfill operation. + +Controlled benchmark flow: + +1. Seed source memories once for a benchmark corpus. +2. Run one extraction pass at a time, e.g. entity-only. +3. Inspect the extraction audit JSONL before trusting the vectors. +4. Backfill embeddings scoped by `BACKFILL_MEMORY_TOPIC=benchmark:memory-recall:`. +5. Run recall using explicit modes/variants and record the output/state files. + +Future recurrent/dream passes should treat prior extracted views as possible source material too. For example, durable facts may be summarized, deduplicated, contradicted, or consolidated against earlier durable facts rather than only extracted from raw episodic memories. diff --git a/packages/benchmarks/package.json b/packages/benchmarks/package.json new file mode 100644 index 00000000..c958cf87 --- /dev/null +++ b/packages/benchmarks/package.json @@ -0,0 +1,21 @@ +{ + "name": "@inklabs/benchmarks", + "version": "0.5.0", + "private": true, + "license": "SEE LICENSE IN LICENSE", + "description": "Benchmark harnesses for Inkwell memory and runtime evaluation", + "scripts": { + "benchmark:memory-recall": "tsx src/benchmark-memory-recall.ts", + "benchmark:bootstrap-relevance": "tsx src/benchmark-bootstrap-relevance.ts", + "test": "vitest run --config vitest.config.ts", + "benchmark:memory-answer": "tsx src/benchmark-memory-answer.ts", + "benchmark:memory-dream": "tsx src/benchmark-memory-dream.ts" + }, + "dependencies": { + "@inklabs/api": "workspace:*" + }, + "devDependencies": { + "tsx": "^4.20.6", + "vitest": "^4.0.18" + } +} diff --git a/packages/benchmarks/src/benchmark-answer-coverage.test.ts b/packages/benchmarks/src/benchmark-answer-coverage.test.ts new file mode 100644 index 00000000..68db5906 --- /dev/null +++ b/packages/benchmarks/src/benchmark-answer-coverage.test.ts @@ -0,0 +1,20 @@ +import { describe, expect, it } from 'vitest'; +import { answerTokenCoverage, hasAnswer, hasOptionalAnswer } from './benchmark-answer-coverage'; + +describe('benchmark answer coverage', () => { + it('matches exact normalized answer phrases', () => { + expect(hasAnswer('The user now has 38 pre-1920 coins.', '38')).toBe(true); + expect( + answerTokenCoverage('The latest plan is sibling review before merge.', 'sibling review') + ).toBe(1); + }); + + it('does not match short numeric answers inside opaque identifiers', () => { + expect(hasAnswer('memoryId=mem38 sessionId=s38 durableFactCount=1', '38')).toBe(false); + }); + + it('returns null for absent optional answers', () => { + expect(hasOptionalAnswer('anything', undefined)).toBeNull(); + expect(hasOptionalAnswer('anything', '')).toBeNull(); + }); +}); diff --git a/packages/benchmarks/src/benchmark-answer-coverage.ts b/packages/benchmarks/src/benchmark-answer-coverage.ts new file mode 100644 index 00000000..f5dd91ec --- /dev/null +++ b/packages/benchmarks/src/benchmark-answer-coverage.ts @@ -0,0 +1,109 @@ +const ANSWER_STOPWORDS = new Set([ + 'a', + 'an', + 'and', + 'are', + 'as', + 'at', + 'be', + 'by', + 'for', + 'from', + 'had', + 'has', + 'have', + 'he', + 'her', + 'his', + 'i', + 'in', + 'is', + 'it', + 'my', + 'of', + 'on', + 'or', + 'our', + 'she', + 'that', + 'the', + 'their', + 'they', + 'to', + 'was', + 'were', + 'with', + 'you', +]); + +export function normalizeAnswerText(text: string | number): string { + return String(text) + .toLowerCase() + .normalize('NFKD') + .replace(/[\u0300-\u036f]/g, '') + .replace(/[^a-z0-9]+/g, ' ') + .replace(/\s+/g, ' ') + .trim(); +} + +function normalizedTokens(text: string | number): string[] { + const normalized = normalizeAnswerText(text); + return normalized ? normalized.split(' ') : []; +} + +function containsNormalizedPhrase(text: string, answer: string | number): boolean { + const normalizedText = normalizeAnswerText(text); + const normalizedAnswer = normalizeAnswerText(answer); + if (!normalizedText || !normalizedAnswer) return false; + return ` ${normalizedText} `.includes(` ${normalizedAnswer} `); +} + +export function answerTokenCoverage(text: string, answer: string | number): number { + if (containsNormalizedPhrase(text, answer)) return 1; + + const textTokens = new Set(normalizedTokens(text)); + const answerTokens = normalizedTokens(answer) + .filter((token) => token.length >= 3 || /^\d+$/.test(token)) + .filter((token) => !ANSWER_STOPWORDS.has(token)); + + if (answerTokens.length === 0) return 0; + const hitCount = answerTokens.filter((token) => textTokens.has(token)).length; + return hitCount / answerTokens.length; +} + +export function hasAnswer(text: string, answer: string | number): boolean { + const coverage = answerTokenCoverage(text, answer); + if (coverage >= 0.8) return true; + + const normalizedAnswer = normalizeAnswerText(answer); + return normalizedAnswer.length >= 4 && containsNormalizedPhrase(text, answer); +} + +export function hasOptionalAnswer( + text: string, + answer: string | number | null | undefined +): boolean | null { + if (answer === null || answer === undefined) return null; + const normalizedAnswer = normalizeAnswerText(answer); + if (!normalizedAnswer) return null; + return hasAnswer(text, answer); +} + +export function compact(text: string, maxChars = 500): string { + const oneLine = text.replace(/\s+/g, ' ').trim(); + if (oneLine.length <= maxChars) return oneLine; + return `${oneLine.slice(0, Math.max(0, maxChars - 1)).trimEnd()}…`; +} + +export function snippetAroundAnswer(text: string, answer: string | number, maxChars = 700): string { + const normalizedAnswer = normalizeAnswerText(answer); + const normalizedText = normalizeAnswerText(text); + const index = normalizedAnswer ? normalizedText.indexOf(normalizedAnswer) : -1; + if (index < 0) return compact(text, maxChars); + + // Indexes after normalization are approximate; use a proportional source slice. + const ratio = index / Math.max(normalizedText.length, 1); + const sourceIndex = Math.floor(text.length * ratio); + const start = Math.max(0, sourceIndex - Math.floor(maxChars / 2)); + return compact(text.slice(start, start + maxChars), maxChars); +} diff --git a/packages/api/src/scripts/benchmark-bootstrap-relevance.ts b/packages/benchmarks/src/benchmark-bootstrap-relevance.ts similarity index 98% rename from packages/api/src/scripts/benchmark-bootstrap-relevance.ts rename to packages/benchmarks/src/benchmark-bootstrap-relevance.ts index 9e813b30..5fc656aa 100644 --- a/packages/api/src/scripts/benchmark-bootstrap-relevance.ts +++ b/packages/benchmarks/src/benchmark-bootstrap-relevance.ts @@ -1,8 +1,7 @@ import { mkdir, writeFile } from 'node:fs/promises'; import { dirname, resolve } from 'node:path'; import { randomUUID } from 'node:crypto'; -import { createSupabaseClient } from '../data/supabase/client'; -import { MemoryRepository } from '../data/repositories/memory-repository'; +import { createSupabaseClient, MemoryRepository } from '@inklabs/api/benchmarks'; import { getBootstrapRelevanceDataset } from './benchmark-data/datasets'; type Mode = 'baseline' | 'thread_aware'; diff --git a/packages/api/src/scripts/benchmark-data/datasets.ts b/packages/benchmarks/src/benchmark-data/datasets.ts similarity index 99% rename from packages/api/src/scripts/benchmark-data/datasets.ts rename to packages/benchmarks/src/benchmark-data/datasets.ts index dd27d899..1984a4fe 100644 --- a/packages/api/src/scripts/benchmark-data/datasets.ts +++ b/packages/benchmarks/src/benchmark-data/datasets.ts @@ -1,7 +1,8 @@ export interface BenchmarkCase { id: string; query: string; - targetContent: string; + targetContent?: string; + targetContents?: string[]; distractors: string[]; provenance?: string; } diff --git a/packages/api/src/scripts/benchmark-data/hf-loader.ts b/packages/benchmarks/src/benchmark-data/hf-loader.ts similarity index 100% rename from packages/api/src/scripts/benchmark-data/hf-loader.ts rename to packages/benchmarks/src/benchmark-data/hf-loader.ts diff --git a/packages/benchmarks/src/benchmark-data/locomo-loader.test.ts b/packages/benchmarks/src/benchmark-data/locomo-loader.test.ts new file mode 100644 index 00000000..a19776eb --- /dev/null +++ b/packages/benchmarks/src/benchmark-data/locomo-loader.test.ts @@ -0,0 +1,72 @@ +import { beforeEach, describe, expect, it } from 'vitest'; +import { mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { loadLoCoMoDataset } from './locomo-loader'; + +describe('loadLoCoMoDataset', () => { + const oldPath = process.env.LOCOMO_DATASET_PATH; + const oldLimit = process.env.LOCOMO_LIMIT; + const oldDistractors = process.env.LOCOMO_MAX_DISTRACTORS; + + beforeEach(() => { + if (oldPath === undefined) delete process.env.LOCOMO_DATASET_PATH; + else process.env.LOCOMO_DATASET_PATH = oldPath; + if (oldLimit === undefined) delete process.env.LOCOMO_LIMIT; + else process.env.LOCOMO_LIMIT = oldLimit; + if (oldDistractors === undefined) delete process.env.LOCOMO_MAX_DISTRACTORS; + else process.env.LOCOMO_MAX_DISTRACTORS = oldDistractors; + }); + + it('maps QA evidence sessions to target content and other sessions to distractors', async () => { + const dir = await mkdtemp(join(tmpdir(), 'locomo-')); + const file = join(dir, 'sample.json'); + + await writeFile( + file, + JSON.stringify([ + { + sample_id: 'sample-1', + conversation: { + speaker_a: 'Alex', + speaker_b: 'Sam', + session_1_date_time: '2024-01-01', + session_1: [ + { speaker: 'Alex', dia_id: 'D1:1', text: 'I started learning guitar last week.' }, + { speaker: 'Sam', dia_id: 'D1:2', text: 'That is exciting.' }, + ], + session_2_date_time: '2024-01-02', + session_2: [ + { speaker: 'Sam', dia_id: 'D2:1', text: 'Did you keep practicing?' }, + { speaker: 'Alex', dia_id: 'D2:2', text: 'Yes, I practiced every day.' }, + ], + }, + qa: [ + { + question: 'What instrument did Alex start learning?', + answer: 'guitar', + evidence: ['D1:1'], + category: 1, + }, + ], + }, + ]), + 'utf-8' + ); + + process.env.LOCOMO_DATASET_PATH = file; + process.env.LOCOMO_LIMIT = '10'; + process.env.LOCOMO_MAX_DISTRACTORS = '3'; + + const loaded = await loadLoCoMoDataset(); + + expect(loaded.cases).toHaveLength(1); + expect(loaded.cases[0].id).toBe('sample-1-qa-1'); + expect(loaded.cases[0].query).toContain('instrument'); + expect(loaded.cases[0].targetContent).toContain('session_1'); + expect(loaded.cases[0].targetContent).toContain('[evidence] Alex: I started learning guitar'); + expect(loaded.cases[0].distractors).toHaveLength(1); + expect(loaded.cases[0].distractors[0]).toContain('session_2'); + expect(loaded.cases[0].provenance).toContain('locomo:sample-1'); + }); +}); diff --git a/packages/benchmarks/src/benchmark-data/locomo-loader.ts b/packages/benchmarks/src/benchmark-data/locomo-loader.ts new file mode 100644 index 00000000..364055de --- /dev/null +++ b/packages/benchmarks/src/benchmark-data/locomo-loader.ts @@ -0,0 +1,195 @@ +import { readFile } from 'node:fs/promises'; +import type { BenchmarkCase } from './datasets'; + +const DEFAULT_LOCOMO_URL = + 'https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json'; + +type LoCoMoTurn = { + speaker?: string; + dia_id?: string; + text?: string; +}; + +type LoCoMoQuestion = { + question?: string; + answer?: string; + evidence?: string[]; + category?: number; +}; + +type LoCoMoConversation = Record & { + speaker_a?: string; + speaker_b?: string; +}; + +type LoCoMoSample = { + sample_id?: string; + conversation?: LoCoMoConversation; + qa?: LoCoMoQuestion[]; +}; + +type SessionRecord = { + key: string; + sessionNumber: number; + dateTime?: string; + turns: LoCoMoTurn[]; +}; + +function parsePositiveInt(raw: string | undefined, fallback: number): number { + if (!raw) return fallback; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) return fallback; + return Math.floor(parsed); +} + +function clampArray(items: T[], limit: number): T[] { + return items.slice(0, Math.max(0, limit)); +} + +function formatSession(session: SessionRecord, evidenceIds: Set): string | null { + const lines = session.turns + .map((turn) => { + const speaker = typeof turn.speaker === 'string' ? turn.speaker.trim() : 'unknown'; + const diaId = typeof turn.dia_id === 'string' ? turn.dia_id.trim() : ''; + const text = typeof turn.text === 'string' ? turn.text.trim() : ''; + if (!text) return null; + const prefix = evidenceIds.has(diaId) ? '[evidence] ' : ''; + return `${prefix}${speaker}: ${text}`; + }) + .filter((line): line is string => !!line); + + if (lines.length === 0) return null; + + const header = session.dateTime + ? `${session.key} @ ${session.dateTime}` + : `${session.key}`; + + return `${header}\n${lines.join('\n')}`; +} + +function extractSessions(conversation: LoCoMoConversation | undefined): SessionRecord[] { + if (!conversation) return []; + + const sessions: SessionRecord[] = []; + for (const [key, value] of Object.entries(conversation)) { + const match = /^session_(\d+)$/.exec(key); + if (!match || !Array.isArray(value)) continue; + + const sessionNumber = Number(match[1]); + const dateTimeKey = `session_${sessionNumber}_date_time`; + const dateTime = + typeof conversation[dateTimeKey] === 'string' ? String(conversation[dateTimeKey]) : undefined; + + sessions.push({ + key, + sessionNumber, + dateTime, + turns: value as LoCoMoTurn[], + }); + } + + return sessions.sort((a, b) => a.sessionNumber - b.sessionNumber); +} + +function mapSamplesToBenchmarkCases( + samples: LoCoMoSample[], + maxCases: number, + maxDistractors: number +): BenchmarkCase[] { + const cases: BenchmarkCase[] = []; + + for (const sample of samples) { + if (cases.length >= maxCases) break; + + const sampleId = typeof sample.sample_id === 'string' ? sample.sample_id : 'unknown-sample'; + const sessions = extractSessions(sample.conversation); + const qaItems = Array.isArray(sample.qa) ? sample.qa : []; + const sessionByNumber = new Map( + sessions.map((session) => [session.sessionNumber, session]) + ); + + for (let qaIndex = 0; qaIndex < qaItems.length && cases.length < maxCases; qaIndex += 1) { + const qa = qaItems[qaIndex]; + const question = typeof qa.question === 'string' ? qa.question.trim() : ''; + const evidence = Array.isArray(qa.evidence) ? qa.evidence.filter(Boolean) : []; + if (!question || evidence.length === 0) continue; + + const evidenceIds = new Set(evidence); + const evidenceSessionNumbers = new Set(); + for (const diaId of evidence) { + const match = /^D(\d+):/.exec(diaId); + if (match) evidenceSessionNumbers.add(Number(match[1])); + } + if (evidenceSessionNumbers.size === 0) continue; + + const targetSessions = [...evidenceSessionNumbers] + .map((num) => sessionByNumber.get(num) || null) + .filter((session): session is SessionRecord => !!session) + .map((session) => formatSession(session, evidenceIds)) + .filter((text): text is string => !!text); + + if (targetSessions.length === 0) continue; + + const distractors = sessions + .filter((session) => !evidenceSessionNumbers.has(session.sessionNumber)) + .map((session) => formatSession(session, new Set())) + .filter((text): text is string => !!text); + + if (distractors.length === 0) continue; + + cases.push({ + id: `${sampleId}-qa-${qaIndex + 1}`, + query: question, + targetContent: targetSessions.join('\n\n---\n\n'), + distractors: clampArray(distractors, maxDistractors), + provenance: `locomo:${sampleId}:category-${qa.category ?? 'unknown'}`, + }); + } + } + + return cases; +} + +async function fetchJson(url: string): Promise { + const response = await fetch(url); + if (!response.ok) { + throw new Error(`LoCoMo download failed (${response.status}): ${await response.text()}`); + } + return response.json(); +} + +async function loadSourceJson(): Promise { + const localPath = process.env.LOCOMO_DATASET_PATH; + if (localPath) { + const raw = await readFile(localPath, 'utf-8'); + return JSON.parse(raw); + } + + const url = process.env.LOCOMO_DATASET_URL || DEFAULT_LOCOMO_URL; + return fetchJson(url); +} + +export async function loadLoCoMoDataset(): Promise<{ + cases: BenchmarkCase[]; + source: string; +}> { + const limit = parsePositiveInt(process.env.LOCOMO_LIMIT, 200); + const maxDistractors = parsePositiveInt(process.env.LOCOMO_MAX_DISTRACTORS, 5); + const raw = await loadSourceJson(); + + if (!Array.isArray(raw)) { + throw new Error('LoCoMo dataset must be a JSON array of conversation samples.'); + } + + const cases = mapSamplesToBenchmarkCases(raw as LoCoMoSample[], limit, maxDistractors); + if (cases.length === 0) { + throw new Error('LoCoMo dataset loaded but produced 0 benchmark cases.'); + } + + return { + cases, + source: process.env.LOCOMO_DATASET_PATH + ? `file:${process.env.LOCOMO_DATASET_PATH}` + : `url:${process.env.LOCOMO_DATASET_URL || DEFAULT_LOCOMO_URL}`, + }; +} diff --git a/packages/benchmarks/src/benchmark-data/longmemeval-loader.test.ts b/packages/benchmarks/src/benchmark-data/longmemeval-loader.test.ts new file mode 100644 index 00000000..3eab17f3 --- /dev/null +++ b/packages/benchmarks/src/benchmark-data/longmemeval-loader.test.ts @@ -0,0 +1,205 @@ +import { beforeEach, describe, expect, it } from 'vitest'; +import { mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { loadLongMemEvalDataset, loadLongMemEvalDreamDataset } from './longmemeval-loader'; + +describe('loadLongMemEvalDataset', () => { + const oldPath = process.env.LONGMEMEVAL_DATASET_PATH; + const oldLimit = process.env.LONGMEMEVAL_LIMIT; + const oldOffset = process.env.LONGMEMEVAL_OFFSET; + const oldDistractors = process.env.LONGMEMEVAL_MAX_DISTRACTORS; + + beforeEach(() => { + if (oldPath === undefined) delete process.env.LONGMEMEVAL_DATASET_PATH; + else process.env.LONGMEMEVAL_DATASET_PATH = oldPath; + if (oldLimit === undefined) delete process.env.LONGMEMEVAL_LIMIT; + else process.env.LONGMEMEVAL_LIMIT = oldLimit; + if (oldOffset === undefined) delete process.env.LONGMEMEVAL_OFFSET; + else process.env.LONGMEMEVAL_OFFSET = oldOffset; + if (oldDistractors === undefined) delete process.env.LONGMEMEVAL_MAX_DISTRACTORS; + else process.env.LONGMEMEVAL_MAX_DISTRACTORS = oldDistractors; + }); + + it('maps answer sessions to distinct target documents and non-answer sessions to distractors', async () => { + const dir = await mkdtemp(join(tmpdir(), 'longmemeval-')); + const file = join(dir, 'sample.json'); + await writeFile( + file, + JSON.stringify([ + { + question_id: 'q1', + question_type: 'multi-session', + question: 'What database backend did the user settle on?', + question_date: '2025-01-10', + haystack_session_ids: ['s1', 's2', 's3'], + answer_session_ids: ['s2', 's3'], + haystack_sessions: [ + [ + { role: 'user', content: 'I am still deciding between sqlite and postgres.' }, + { role: 'assistant', content: 'Let us compare both options.' }, + ], + [ + { role: 'user', content: 'I think postgres is the safer backend.' }, + { role: 'assistant', content: 'That sounds like the current preference.' }, + ], + [{ role: 'user', content: 'Decision made: we are standardizing on postgres.' }], + ], + }, + ]), + 'utf-8' + ); + + process.env.LONGMEMEVAL_DATASET_PATH = file; + process.env.LONGMEMEVAL_LIMIT = '10'; + process.env.LONGMEMEVAL_MAX_DISTRACTORS = '2'; + + const loaded = await loadLongMemEvalDataset(); + + expect(loaded.cases).toHaveLength(1); + expect(loaded.cases[0].id).toBe('q1'); + expect(loaded.cases[0].query).toContain('What database backend'); + expect(loaded.cases[0].targetContents).toEqual([ + expect.stringContaining('session s2'), + expect.stringContaining('session s3'), + ]); + expect(loaded.cases[0].distractors).toHaveLength(1); + expect(loaded.cases[0].distractors[0]).toContain('session s1'); + expect(loaded.cases[0].provenance).toContain('multi-session'); + }); + + it('uses the full haystack when LONGMEMEVAL_MAX_DISTRACTORS is not set', async () => { + const dir = await mkdtemp(join(tmpdir(), 'longmemeval-full-')); + const file = join(dir, 'sample-full.json'); + await writeFile( + file, + JSON.stringify([ + { + question_id: 'q2', + question_type: 'single-session-user', + question: 'What city did I move to?', + question_date: '2025-01-11', + haystack_session_ids: ['s1', 's2', 's3', 's4'], + answer_session_ids: ['s4'], + haystack_sessions: [ + [{ role: 'user', content: 'This is distractor one.' }], + [{ role: 'user', content: 'This is distractor two.' }], + [{ role: 'user', content: 'This is distractor three.' }], + [{ role: 'user', content: 'I moved to Portland last summer.' }], + ], + }, + ]), + 'utf-8' + ); + + process.env.LONGMEMEVAL_DATASET_PATH = file; + process.env.LONGMEMEVAL_LIMIT = '10'; + delete process.env.LONGMEMEVAL_MAX_DISTRACTORS; + + const loaded = await loadLongMemEvalDataset(); + + expect(loaded.cases).toHaveLength(1); + expect(loaded.cases[0].distractors).toHaveLength(3); + expect(loaded.cases[0].targetContents).toEqual([expect.stringContaining('session s4')]); + }); + + it('supports LONGMEMEVAL_OFFSET for batch seeding later windows', async () => { + const dir = await mkdtemp(join(tmpdir(), 'longmemeval-offset-')); + const file = join(dir, 'sample-offset.json'); + await writeFile( + file, + JSON.stringify([ + { + question_id: 'q1', + question_type: 'type-a', + question: 'first?', + question_date: '2025-01-01', + haystack_session_ids: ['s1', 's2'], + answer_session_ids: ['s2'], + haystack_sessions: [[{ role: 'user', content: 'd1' }], [{ role: 'user', content: 't1' }]], + }, + { + question_id: 'q2', + question_type: 'type-b', + question: 'second?', + question_date: '2025-01-02', + haystack_session_ids: ['s3', 's4'], + answer_session_ids: ['s4'], + haystack_sessions: [[{ role: 'user', content: 'd2' }], [{ role: 'user', content: 't2' }]], + }, + { + question_id: 'q3', + question_type: 'type-c', + question: 'third?', + question_date: '2025-01-03', + haystack_session_ids: ['s5', 's6'], + answer_session_ids: ['s6'], + haystack_sessions: [[{ role: 'user', content: 'd3' }], [{ role: 'user', content: 't3' }]], + }, + ]), + 'utf-8' + ); + + process.env.LONGMEMEVAL_DATASET_PATH = file; + process.env.LONGMEMEVAL_LIMIT = '1'; + process.env.LONGMEMEVAL_OFFSET = '1'; + delete process.env.LONGMEMEVAL_MAX_DISTRACTORS; + + const loaded = await loadLongMemEvalDataset(); + + expect(loaded.cases).toHaveLength(1); + expect(loaded.cases[0].id).toBe('q2'); + expect(loaded.cases[0].query).toBe('second?'); + }); + + it('loads dream cases in chronological haystack order without splitting targets first', async () => { + const dir = await mkdtemp(join(tmpdir(), 'longmemeval-dream-')); + const file = join(dir, 'sample-dream.json'); + await writeFile( + file, + JSON.stringify([ + { + question_id: 'q-dream', + question_type: 'knowledge-update', + question: 'How many coins do I have?', + answer: '38', + question_date: '2025-01-12', + haystack_session_ids: ['older', 'newer'], + haystack_dates: ['2025-01-01', '2025-01-05'], + answer_session_ids: ['older', 'newer'], + haystack_sessions: [ + [{ role: 'user', content: 'I have 37 pre-1920 coins.', has_answer: true }], + [ + { + role: 'user', + content: 'I added one more 1915-S Barber quarter.', + has_answer: true, + }, + ], + ], + }, + ]), + 'utf-8' + ); + + process.env.LONGMEMEVAL_DATASET_PATH = file; + process.env.LONGMEMEVAL_LIMIT = '10'; + delete process.env.LONGMEMEVAL_MAX_DISTRACTORS; + + const loaded = await loadLongMemEvalDreamDataset(); + + expect(loaded.cases).toHaveLength(1); + expect(loaded.cases[0].id).toBe('q-dream'); + expect(loaded.cases[0].answer).toBe('38'); + expect(loaded.cases[0].sessions.map((session) => session.sessionId)).toEqual([ + 'older', + 'newer', + ]); + expect(loaded.cases[0].sessions[0]).toMatchObject({ + date: '2025-01-01', + hasAnswer: true, + isAnswerSession: true, + }); + expect(loaded.cases[0].sessions[1].content).toContain('session newer'); + }); +}); diff --git a/packages/benchmarks/src/benchmark-data/longmemeval-loader.ts b/packages/benchmarks/src/benchmark-data/longmemeval-loader.ts new file mode 100644 index 00000000..b19aa173 --- /dev/null +++ b/packages/benchmarks/src/benchmark-data/longmemeval-loader.ts @@ -0,0 +1,289 @@ +import { readFile } from 'node:fs/promises'; +import type { BenchmarkCase } from './datasets'; + +const DEFAULT_LONGMEMEVAL_URL = + 'https://huggingface.co/datasets/xiaowu0162/longmemeval-cleaned/resolve/main/longmemeval_s_cleaned.json'; + +type LongMemEvalTurn = { + role?: string; + content?: string; + has_answer?: boolean; +}; + +type LongMemEvalInstance = { + question_id?: string; + question_type?: string; + question?: string; + answer?: string; + question_date?: string; + haystack_session_ids?: string[]; + haystack_dates?: string[]; + haystack_sessions?: LongMemEvalTurn[][]; + answer_session_ids?: string[]; +}; + +export interface LongMemEvalOrderedSession { + sessionId: string; + date?: string; + content: string; + hasAnswer: boolean; + isAnswerSession: boolean; + turnCount: number; +} + +export interface LongMemEvalDreamCase { + id: string; + query: string; + answer?: string; + questionType?: string; + questionDate?: string; + answerSessionIds: string[]; + sessions: LongMemEvalOrderedSession[]; +} + +function parsePositiveInt(raw: string | undefined, fallback: number): number { + if (!raw) return fallback; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) return fallback; + return Math.floor(parsed); +} + +function parseOptionalPositiveInt(raw: string | undefined): number | null { + if (!raw) return null; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) return null; + return Math.floor(parsed); +} + +function parseNonNegativeInt(raw: string | undefined, fallback: number): number { + if (!raw) return fallback; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed < 0) return fallback; + return Math.floor(parsed); +} + +function clampArray(items: T[], limit: number): T[] { + return items.slice(0, Math.max(0, limit)); +} + +export function formatLongMemEvalSession(turns: LongMemEvalTurn[]): string { + return turns + .map((turn) => { + const role = typeof turn.role === 'string' ? turn.role : 'unknown'; + const content = typeof turn.content === 'string' ? turn.content.trim() : ''; + if (!content) return null; + return `${role}: ${content}`; + }) + .filter((line): line is string => !!line) + .join('\n'); +} + +function buildTargetContents(instance: LongMemEvalInstance): string[] { + const sessionIds = Array.isArray(instance.haystack_session_ids) + ? instance.haystack_session_ids + : []; + const sessions = Array.isArray(instance.haystack_sessions) ? instance.haystack_sessions : []; + const answerIds = new Set( + Array.isArray(instance.answer_session_ids) ? instance.answer_session_ids : [] + ); + + const matched = sessionIds + .map((sessionId, idx) => ({ + sessionId, + turns: sessions[idx] || [], + })) + .filter(({ sessionId }) => answerIds.has(sessionId)) + .map(({ sessionId, turns }) => { + const formatted = formatLongMemEvalSession(turns); + return formatted ? `session ${sessionId}\n${formatted}` : null; + }) + .filter((text): text is string => !!text); + + return matched; +} + +function buildDistractors(instance: LongMemEvalInstance, maxDistractors: number): string[] { + const sessionIds = Array.isArray(instance.haystack_session_ids) + ? instance.haystack_session_ids + : []; + const sessions = Array.isArray(instance.haystack_sessions) ? instance.haystack_sessions : []; + const answerIds = new Set( + Array.isArray(instance.answer_session_ids) ? instance.answer_session_ids : [] + ); + + const distractors = sessionIds + .map((sessionId, idx) => ({ + sessionId, + turns: sessions[idx] || [], + })) + .filter(({ sessionId }) => !answerIds.has(sessionId)) + .map(({ sessionId, turns }) => { + const formatted = formatLongMemEvalSession(turns); + return formatted ? `session ${sessionId}\n${formatted}` : null; + }) + .filter((text): text is string => !!text); + + return clampArray(distractors, maxDistractors); +} + +function mapInstancesToBenchmarkCases( + instances: LongMemEvalInstance[], + offset: number, + maxCases: number, + maxDistractors: number +): BenchmarkCase[] { + const cases: BenchmarkCase[] = []; + + for (const instance of instances.slice(offset)) { + if (cases.length >= maxCases) break; + const id = typeof instance.question_id === 'string' ? instance.question_id : null; + const query = typeof instance.question === 'string' ? instance.question.trim() : null; + if (!id || !query) continue; + + const targetContents = buildTargetContents(instance); + if (targetContents.length === 0) continue; + + const distractors = buildDistractors(instance, maxDistractors); + if (distractors.length === 0) continue; + + cases.push({ + id, + query, + targetContents, + distractors, + provenance: `longmemeval:${instance.question_type || 'unknown'}:${instance.question_date || 'unknown-date'}`, + }); + } + + return cases; +} + +function mapInstancesToDreamCases( + instances: LongMemEvalInstance[], + offset: number, + maxCases: number +): LongMemEvalDreamCase[] { + const cases: LongMemEvalDreamCase[] = []; + + for (const instance of instances.slice(offset)) { + if (cases.length >= maxCases) break; + + const id = typeof instance.question_id === 'string' ? instance.question_id : null; + const query = typeof instance.question === 'string' ? instance.question.trim() : null; + if (!id || !query) continue; + + const sessionIds = Array.isArray(instance.haystack_session_ids) + ? instance.haystack_session_ids + : []; + const sessions = Array.isArray(instance.haystack_sessions) ? instance.haystack_sessions : []; + const dates = Array.isArray(instance.haystack_dates) ? instance.haystack_dates : []; + const answerSessionIds = Array.isArray(instance.answer_session_ids) + ? instance.answer_session_ids + : []; + const answerIds = new Set(answerSessionIds); + + const orderedSessions = sessionIds + .map((sessionId, idx) => { + const turns = sessions[idx] || []; + const formatted = formatLongMemEvalSession(turns); + if (!formatted.trim()) return null; + return { + sessionId, + date: typeof dates[idx] === 'string' ? dates[idx] : undefined, + content: `session ${sessionId}\n${formatted}`, + hasAnswer: turns.some((turn) => turn.has_answer === true), + isAnswerSession: answerIds.has(sessionId), + turnCount: turns.length, + }; + }) + .filter((session): session is LongMemEvalOrderedSession => Boolean(session)); + + if (orderedSessions.length === 0) continue; + + cases.push({ + id, + query, + answer: typeof instance.answer === 'string' ? instance.answer : undefined, + questionType: typeof instance.question_type === 'string' ? instance.question_type : undefined, + questionDate: typeof instance.question_date === 'string' ? instance.question_date : undefined, + answerSessionIds, + sessions: orderedSessions, + }); + } + + return cases; +} + +async function fetchJson(url: string): Promise { + const response = await fetch(url); + if (!response.ok) { + throw new Error(`LongMemEval download failed (${response.status}): ${await response.text()}`); + } + return response.json(); +} + +async function loadSourceJson(): Promise { + const localPath = process.env.LONGMEMEVAL_DATASET_PATH; + if (localPath) { + const raw = await readFile(localPath, 'utf-8'); + return JSON.parse(raw); + } + + const url = process.env.LONGMEMEVAL_DATASET_URL || DEFAULT_LONGMEMEVAL_URL; + return fetchJson(url); +} + +export async function loadLongMemEvalDataset(): Promise<{ + cases: BenchmarkCase[]; + source: string; +}> { + const limit = parsePositiveInt(process.env.LONGMEMEVAL_LIMIT, 100); + const offset = parseNonNegativeInt(process.env.LONGMEMEVAL_OFFSET, 0); + const maxDistractors = + parseOptionalPositiveInt(process.env.LONGMEMEVAL_MAX_DISTRACTORS) ?? Number.MAX_SAFE_INTEGER; + const raw = await loadSourceJson(); + if (!Array.isArray(raw)) { + throw new Error('LongMemEval dataset must be a JSON array of evaluation instances.'); + } + + const cases = mapInstancesToBenchmarkCases( + raw as LongMemEvalInstance[], + offset, + limit, + maxDistractors + ); + if (cases.length === 0) { + throw new Error('LongMemEval dataset loaded but produced 0 benchmark cases.'); + } + + return { + cases, + source: process.env.LONGMEMEVAL_DATASET_PATH + ? `file:${process.env.LONGMEMEVAL_DATASET_PATH}` + : `url:${process.env.LONGMEMEVAL_DATASET_URL || DEFAULT_LONGMEMEVAL_URL}`, + }; +} + +export async function loadLongMemEvalDreamDataset(): Promise<{ + cases: LongMemEvalDreamCase[]; + source: string; +}> { + const limit = parsePositiveInt(process.env.LONGMEMEVAL_LIMIT, 100); + const offset = parseNonNegativeInt(process.env.LONGMEMEVAL_OFFSET, 0); + const raw = await loadSourceJson(); + if (!Array.isArray(raw)) { + throw new Error('LongMemEval dataset must be a JSON array of evaluation instances.'); + } + + const cases = mapInstancesToDreamCases(raw as LongMemEvalInstance[], offset, limit); + if (cases.length === 0) { + throw new Error('LongMemEval dataset loaded but produced 0 dream cases.'); + } + + return { + cases, + source: process.env.LONGMEMEVAL_DATASET_PATH + ? `file:${process.env.LONGMEMEVAL_DATASET_PATH}` + : `url:${process.env.LONGMEMEVAL_DATASET_URL || DEFAULT_LONGMEMEVAL_URL}`, + }; +} diff --git a/packages/benchmarks/src/benchmark-data/public-benchmarks.ts b/packages/benchmarks/src/benchmark-data/public-benchmarks.ts new file mode 100644 index 00000000..0e4c70a3 --- /dev/null +++ b/packages/benchmarks/src/benchmark-data/public-benchmarks.ts @@ -0,0 +1,72 @@ +export type PublicBenchmarkFamily = 'longmemeval' | 'locomo' | 'convomem' | 'membench'; + +export interface PublicBenchmarkDescriptor { + family: PublicBenchmarkFamily; + displayName: string; + primaryQuestion: string; + whyItMatters: string; + recommendedMetrics: string[]; + implementationNotes: string[]; +} + +export const PUBLIC_BENCHMARKS: PublicBenchmarkDescriptor[] = [ + { + family: 'longmemeval', + displayName: 'LongMemEval', + primaryQuestion: 'Can the system retrieve the right conversational memory over long horizons?', + whyItMatters: + 'This is the cleanest first benchmark for Inkwell memory retrieval because it tests long-horizon conversational recall without being Ink-specific.', + recommendedMetrics: ['recall@1', 'recall@3', 'recall@5', 'mrr', 'ndcg', 'latency'], + implementationNotes: [ + 'Start here first.', + 'Evaluate raw text, semantic, hybrid, chunked, and optional rerank tiers separately.', + 'Keep a no-LLM baseline as the primary honest comparison point.', + ], + }, + { + family: 'locomo', + displayName: 'LoCoMo', + primaryQuestion: + 'Can the system retrieve and support questions that require temporal and multi-hop conversational reasoning?', + whyItMatters: + 'LoCoMo adds pressure from temporal reasoning and cross-session dependencies, which are central to continuity claims.', + recommendedMetrics: ['recall@5', 'recall@10', 'category breakdown', 'latency'], + implementationNotes: [ + 'Be careful with top-k settings so retrieval remains meaningful.', + 'Report category-level results, not just one overall score.', + ], + }, + { + family: 'convomem', + displayName: 'ConvoMem', + primaryQuestion: 'How well does the system perform on large-scale conversational memory retrieval?', + whyItMatters: + 'Useful for measuring scale and broader conversational coverage once LongMemEval and LoCoMo are stable.', + recommendedMetrics: ['recall@k', 'category breakdown', 'latency'], + implementationNotes: [ + 'Good follow-up benchmark after LongMemEval parity.', + 'Useful for testing whether dream-phase extraction helps or hurts at scale.', + ], + }, + { + family: 'membench', + displayName: 'MemBench / BEAM-style suites', + primaryQuestion: + 'How well does the system behave under broader noisy-memory and long-context retrieval pressure?', + whyItMatters: + 'This is useful for the quality-vs-efficiency story and for comparing long-context memory tradeoffs.', + recommendedMetrics: ['benchmark-specific score', 'latency', 'cost', 'token efficiency'], + implementationNotes: [ + 'Use this after the core conversational benchmarks are wired in.', + 'Helpful for positioning Ink against systems that emphasize token-efficient long-context memory.', + ], + }, +]; + +export function getPublicBenchmarkDescriptor( + family: PublicBenchmarkFamily +): PublicBenchmarkDescriptor { + const descriptor = PUBLIC_BENCHMARKS.find((entry) => entry.family === family); + if (!descriptor) throw new Error(`Unknown public benchmark family: ${family}`); + return descriptor; +} diff --git a/packages/benchmarks/src/benchmark-memory-answer.ts b/packages/benchmarks/src/benchmark-memory-answer.ts new file mode 100644 index 00000000..42e2e548 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-answer.ts @@ -0,0 +1,212 @@ +import { readFile, mkdir, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import { createSupabaseClient, MemoryRepository } from '@inklabs/api/benchmarks'; +import { + buildBenchmarkRecallOptions, + parseBenchmarkRecallVariant, +} from './benchmark-memory-recall.variant'; +import { answerTokenCoverage, hasAnswer, snippetAroundAnswer } from './benchmark-answer-coverage'; +import type { RecallMode } from './benchmark-memory-recall.types'; + +type LongMemEvalTurn = { + role?: string; + content?: string; +}; + +type LongMemEvalInstance = { + question_id?: string; + question_type?: string; + question?: string; + answer?: string | number; + question_date?: string; +}; + +type SeededCase = { + topic: string; + targetMemoryIds: string[]; +}; + +type SeedState = { + seedId: string; + seededCases: Record; +}; + +const TOP_K = 5; +const DEFAULT_LONGMEMEVAL_PATH = resolve(process.cwd(), '.cache', 'longmemeval_s_cleaned.json'); +const BENCHMARK_AGENT_ID = 'lumen'; + +function parsePositiveInt(raw: string | undefined, fallback: number): number { + if (!raw) return fallback; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) return fallback; + return Math.floor(parsed); +} + +function parseNonNegativeInt(raw: string | undefined, fallback: number): number { + if (!raw) return fallback; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed < 0) return fallback; + return Math.floor(parsed); +} + +function parseMode(raw?: string): RecallMode { + if (raw === 'text' || raw === 'semantic' || raw === 'hybrid' || raw === 'auto') return raw; + return 'semantic'; +} + +async function readJson(path: string): Promise { + return JSON.parse(await readFile(path, 'utf-8')) as T; +} + +async function writeJson(path: string, payload: unknown): Promise { + await mkdir(dirname(path), { recursive: true }); + await writeFile(path, JSON.stringify(payload, null, 2), 'utf-8'); +} + +function mean(values: number[]): number { + if (values.length === 0) return 0; + return values.reduce((acc, value) => acc + value, 0) / values.length; +} + +function round(value: number): number { + return Number(value.toFixed(4)); +} + +async function main() { + const userId = process.env.BENCHMARK_USER_ID; + if (!userId) { + throw new Error('BENCHMARK_USER_ID is required.'); + } + + const rawPath = process.env.LONGMEMEVAL_DATASET_PATH || DEFAULT_LONGMEMEVAL_PATH; + const seedPath = + process.env.MEMORY_BENCHMARK_SEED_PATH || + resolve( + process.cwd(), + 'output', + 'memory-benchmarks', + 'longmem-100c-all-distractors-20260427.seed.json' + ); + const outputPath = + process.env.MEMORY_ANSWER_BENCHMARK_OUTPUT_PATH || + resolve( + process.cwd(), + 'output', + 'memory-answer-benchmarks', + `longmem-answer-${Date.now()}.json` + ); + const offset = parseNonNegativeInt(process.env.LONGMEMEVAL_OFFSET, 0); + const limit = parsePositiveInt(process.env.LONGMEMEVAL_LIMIT, 100); + const variant = parseBenchmarkRecallVariant(process.env.MEMORY_BENCHMARK_VARIANT); + const mode = parseMode(process.env.MEMORY_BENCHMARK_MODE); + const progressEvery = parsePositiveInt(process.env.MEMORY_BENCHMARK_PROGRESS_EVERY, 10); + + const raw = await readJson(rawPath); + const seed = await readJson(seedPath); + const cases = raw.slice(offset, offset + limit).filter((item) => { + return ( + item.question_id && + item.question && + item.answer !== undefined && + seed.seededCases[item.question_id] + ); + }); + + const repo = new MemoryRepository(createSupabaseClient()); + const runs = []; + const startedAt = Date.now(); + + console.log( + `[memory-answer-benchmark] cases=${cases.length} offset=${offset} limit=${limit} variant=${variant} mode=${mode} seed=${seed.seedId}` + ); + + for (const [index, benchCase] of cases.entries()) { + const caseId = benchCase.question_id!; + const query = benchCase.question!; + const answer = benchCase.answer!; + const seeded = seed.seededCases[caseId]; + const recalled = await repo.recall( + userId, + query, + buildBenchmarkRecallOptions({ + mode, + variant, + limit: TOP_K, + agentId: BENCHMARK_AGENT_ID, + topics: [seeded.topic], + }) + ); + + const targetIds = new Set(seeded.targetMemoryIds); + const targetRank = recalled.findIndex((memory) => targetIds.has(memory.id)); + const top1Text = recalled[0]?.content || ''; + const top5Text = recalled.map((memory) => memory.content).join('\n\n---\n\n'); + const top1AnswerCoverage = answerTokenCoverage(top1Text, answer); + const top5AnswerCoverage = answerTokenCoverage(top5Text, answer); + const top1ContainsAnswer = hasAnswer(top1Text, answer); + const top5ContainsAnswer = hasAnswer(top5Text, answer); + + runs.push({ + caseId, + questionType: benchCase.question_type || null, + query, + answer: String(answer), + targetRank: targetRank >= 0 ? targetRank + 1 : null, + top1ContainsAnswer, + top5ContainsAnswer, + top1AnswerCoverage: round(top1AnswerCoverage), + top5AnswerCoverage: round(top5AnswerCoverage), + topSummaries: recalled.map((memory) => memory.summary || memory.content.slice(0, 80)), + top1Snippet: snippetAroundAnswer(top1Text, answer), + }); + + if ((index + 1) % progressEvery === 0 || index === cases.length - 1) { + const elapsedMs = Date.now() - startedAt; + console.log( + `[memory-answer-benchmark] progress ${index + 1}/${cases.length} elapsed=${Math.round( + elapsedMs / 1000 + )}s` + ); + } + } + + const top1Contains = runs.filter((run) => run.top1ContainsAnswer).length; + const top5Contains = runs.filter((run) => run.top5ContainsAnswer).length; + const targetAt1 = runs.filter((run) => run.targetRank === 1).length; + const targetAt5 = runs.filter((run) => run.targetRank !== null && run.targetRank <= TOP_K).length; + const payload = { + settings: { + dataset: 'longmemeval-s-cleaned', + rawPath, + seedPath, + seedId: seed.seedId, + offset, + limit, + variant, + mode, + topK: TOP_K, + outputPath, + }, + summary: { + cases: runs.length, + targetRecallAt1: round(targetAt1 / runs.length), + targetRecallAt5: round(targetAt5 / runs.length), + answerInTop1: round(top1Contains / runs.length), + answerInTop5: round(top5Contains / runs.length), + meanTop1AnswerCoverage: round(mean(runs.map((run) => run.top1AnswerCoverage))), + meanTop5AnswerCoverage: round(mean(runs.map((run) => run.top5AnswerCoverage))), + }, + misses: runs.filter((run) => !run.top5ContainsAnswer), + nonTop1Answers: runs.filter((run) => !run.top1ContainsAnswer), + runs, + }; + + await writeJson(outputPath, payload); + console.log(JSON.stringify(payload.summary, null, 2)); + console.log(`[memory-answer-benchmark] wrote ${outputPath}`); +} + +main().catch((error) => { + console.error('[memory-answer-benchmark] failed:', error); + process.exit(1); +}); diff --git a/packages/benchmarks/src/benchmark-memory-dream.logic.test.ts b/packages/benchmarks/src/benchmark-memory-dream.logic.test.ts new file mode 100644 index 00000000..9fb9b610 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-dream.logic.test.ts @@ -0,0 +1,272 @@ +import { describe, expect, it } from 'vitest'; +import { hasOptionalAnswer } from './benchmark-answer-coverage'; +import { + applyLocalDreamUpdate, + buildOnlineDreamPrompt, + buildOrderedDreamSessions, + createInitialDreamState, + parseLongMemSessionId, + renderDreamStateForAnswerCheck, + type DreamMemoryRow, +} from './benchmark-memory-dream.logic'; +import type { LongMemEvalDreamCase } from './benchmark-data/longmemeval-loader'; + +describe('benchmark-memory-dream logic', () => { + it('parses LongMemEval session ids from seeded memory content', () => { + expect(parseLongMemSessionId('session abc-123\nuser: hello')).toBe('abc-123'); + expect(parseLongMemSessionId('not a session\nuser: hello')).toBeNull(); + }); + + it('orders seeded memories by the dataset haystack order, not database order', () => { + const dreamCase: LongMemEvalDreamCase = { + id: 'case-1', + query: 'What changed?', + answerSessionIds: ['newer'], + sessions: [ + { + sessionId: 'older', + content: 'session older\nuser: old', + hasAnswer: false, + isAnswerSession: false, + turnCount: 1, + }, + { + sessionId: 'newer', + content: 'session newer\nuser: new', + hasAnswer: true, + isAnswerSession: true, + turnCount: 1, + }, + ], + }; + const rows: DreamMemoryRow[] = [ + { + id: 'mem-new', + content: 'session newer\nuser: new', + summary: null, + metadata: null, + created_at: '2026-01-02T00:00:00Z', + }, + { + id: 'mem-old', + content: 'session older\nuser: old', + summary: null, + metadata: null, + created_at: '2026-01-01T00:00:00Z', + }, + ]; + + const ordered = buildOrderedDreamSessions(dreamCase, rows); + + expect(ordered.missingSessionIds).toEqual([]); + expect(ordered.sessions.map((session) => session.memoryId)).toEqual(['mem-old', 'mem-new']); + }); + + it('integrates per-memory LLM views into a compact evidence-linked local dream state', () => { + const state = createInitialDreamState('case-coin', 'online'); + const [session] = buildOrderedDreamSessions( + { + id: 'case-coin', + query: 'How many pre-1920 coins do I have?', + answer: '38', + answerSessionIds: ['s1'], + sessions: [ + { + sessionId: 's1', + content: 'session s1\nuser: I now have 38 pre-1920 coins.', + hasAnswer: true, + isAnswerSession: true, + turnCount: 1, + }, + ], + }, + [ + { + id: 'mem-s1', + content: 'session s1\nuser: I now have 38 pre-1920 coins.', + summary: 'benchmark target', + created_at: '2026-01-01T00:00:00Z', + metadata: { + llm_extractions: { + durable_fact: { + durableFacts: [ + { + fact: 'The user now has 38 pre-1920 coins.', + category: 'status', + subject: 'user', + object: 'pre-1920 coin count', + evidence: 'I now have 38 pre-1920 coins.', + }, + ], + }, + summary: { + summary: 'The user updated their pre-1920 coin count to 38.', + keyPoints: ['38 pre-1920 coins'], + actionRelevance: 'Useful for answering collection count questions.', + }, + }, + }, + }, + ] + ).sessions; + + const updated = applyLocalDreamUpdate(state, session); + const rendered = renderDreamStateForAnswerCheck(updated); + + expect(updated.sessionCount).toBe(1); + expect(updated.durableFacts).toHaveLength(1); + expect(updated.durableFacts[0].evidenceMemoryIds).toEqual(['mem-s1']); + expect(hasOptionalAnswer(rendered, '38')).toBe(true); + }); + + it('keeps richer entity descriptions when later mentions are terse', () => { + const state = createInitialDreamState('case-entity', 'online'); + const [detailed, terse] = buildOrderedDreamSessions( + { + id: 'case-entity', + query: 'Who is Riley?', + answerSessionIds: ['s2'], + sessions: [ + { + sessionId: 's1', + content: 'session s1\nuser: Riley is my neighbor who restores vintage bikes.', + hasAnswer: false, + isAnswerSession: false, + turnCount: 1, + }, + { + sessionId: 's2', + content: 'session s2\nuser: Riley stopped by.', + hasAnswer: true, + isAnswerSession: true, + turnCount: 1, + }, + ], + }, + [ + { + id: 'mem-s1', + content: 'session s1\nuser: Riley is my neighbor who restores vintage bikes.', + summary: null, + created_at: '2026-01-01T00:00:00Z', + metadata: { + llm_extractions: { + entity: { + entities: [ + { + name: 'Riley', + entityType: 'person', + description: 'Neighbor who restores vintage bikes.', + aliases: [], + evidence: 'Riley is my neighbor who restores vintage bikes.', + }, + ], + }, + }, + }, + }, + { + id: 'mem-s2', + content: 'session s2\nuser: Riley stopped by.', + summary: null, + created_at: '2026-01-02T00:00:00Z', + metadata: { + llm_extractions: { + entity: { + entities: [ + { + name: 'Riley', + entityType: 'person', + description: 'Riley stopped by.', + aliases: [], + evidence: 'Riley stopped by.', + }, + ], + }, + }, + }, + }, + ] + ).sessions; + + const updated = applyLocalDreamUpdate(applyLocalDreamUpdate(state, detailed), terse); + + expect(updated.entities).toHaveLength(1); + expect(updated.entities[0].description).toBe('Neighbor who restores vintage bikes.'); + expect(updated.entities[0].evidenceSessionIds).toEqual(['s1', 's2']); + }); + + it('does not count lineage metadata as answer evidence when rendering for coverage', () => { + const state = createInitialDreamState('case-38', 'online'); + const rendered = renderDreamStateForAnswerCheck({ + ...state, + evidenceSessionIds: ['38'], + evidenceMemoryIds: ['mem-38'], + entities: [ + { + name: 'Coin collection', + entityType: 'collection', + description: 'A vintage collection.', + aliases: [], + evidence: 'A vintage collection was mentioned.', + evidenceMemoryIds: ['mem-38'], + evidenceSessionIds: ['38'], + firstSeenSessionId: '38', + lastSeenSessionId: '38', + }, + ], + temporalEvents: [ + { + sessionId: '38', + memoryId: 'mem-38', + orderIndex: 38, + summary: 'The user discussed a collection.', + isAnswerSession: false, + }, + ], + }); + + expect(rendered).not.toContain('mem-38'); + expect(hasOptionalAnswer(rendered, '38')).toBe(false); + }); + + it('builds an online dream prompt that uses prior state plus one new episode', () => { + const state = createInitialDreamState('case-1', 'online'); + const [session] = buildOrderedDreamSessions( + { + id: 'case-1', + query: 'What process should I follow?', + answerSessionIds: ['s1'], + sessions: [ + { + sessionId: 's1', + content: 'session s1\nuser: Always request sibling review before merge.', + hasAnswer: true, + isAnswerSession: true, + turnCount: 1, + }, + ], + }, + [ + { + id: 'mem-s1', + content: 'session s1\nuser: Always request sibling review before merge.', + summary: null, + metadata: null, + created_at: '2026-01-01T00:00:00Z', + }, + ] + ).sessions; + + const prompt = buildOnlineDreamPrompt({ + caseId: 'case-1', + question: 'What process should I follow?', + previousState: state, + nextSession: session, + }); + + expect(prompt.systemPrompt).toContain('one new chronological episode'); + expect(prompt.userPrompt).toContain('Previous compact dream state JSON'); + expect(prompt.userPrompt).toContain('session s1'); + }); +}); diff --git a/packages/benchmarks/src/benchmark-memory-dream.logic.ts b/packages/benchmarks/src/benchmark-memory-dream.logic.ts new file mode 100644 index 00000000..f9123975 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-dream.logic.ts @@ -0,0 +1,539 @@ +import type { LongMemEvalDreamCase } from './benchmark-data/longmemeval-loader'; + +export type DreamMode = 'online' | 'batch'; + +export interface DreamMemoryRow { + id: string; + content: string; + summary: string | null; + metadata: Record | null; + created_at: string; +} + +export interface DreamEntity { + name: string; + entityType?: string; + description: string; + aliases: string[]; + evidence: string; + evidenceMemoryIds: string[]; + evidenceSessionIds: string[]; + firstSeenSessionId: string; + lastSeenSessionId: string; +} + +export interface DreamDurableFact { + key: string; + fact: string; + category: string; + subject?: string; + object?: string; + evidence: string; + status: 'active' | 'superseded' | 'uncertain'; + evidenceMemoryIds: string[]; + evidenceSessionIds: string[]; + firstSeenSessionId: string; + lastSeenSessionId: string; +} + +export interface DreamCurrentState { + key: string; + state: string; + scope: string; + status: string; + volatility: string; + evidence: string; + evidenceMemoryIds: string[]; + evidenceSessionIds: string[]; + lastSeenSessionId: string; +} + +export interface DreamTemporalEvent { + sessionId: string; + memoryId: string; + date?: string; + orderIndex: number; + summary: string; + isAnswerSession: boolean; +} + +export interface DreamState { + caseId: string; + mode: DreamMode; + sessionCount: number; + stateSummary: string; + entities: DreamEntity[]; + durableFacts: DreamDurableFact[]; + currentStates: DreamCurrentState[]; + temporalEvents: DreamTemporalEvent[]; + evidenceMemoryIds: string[]; + evidenceSessionIds: string[]; + updatedAt: string; +} + +export interface OrderedDreamSession { + caseId: string; + sessionId: string; + memoryId: string; + content: string; + date?: string; + hasAnswer: boolean; + isAnswerSession: boolean; + createdAt: string; + sourceSummary: string | null; + extractions: DreamExtractionViews; +} + +export interface DreamExtractionViews { + entities: ExtractedEntity[]; + durableFacts: ExtractedDurableFact[]; + summary: ExtractedSummary | null; + currentState: ExtractedCurrentState | null; +} + +export interface ExtractedEntity { + name: string; + aliases: string[]; + entityType?: string; + description: string; + evidence: string; +} + +export interface ExtractedDurableFact { + fact: string; + category: string; + subject?: string; + object?: string; + evidence: string; +} + +export interface ExtractedSummary { + summary: string; + keyPoints: string[]; + actionRelevance: string; +} + +export interface ExtractedCurrentState { + state: string; + scope: string; + status: string; + volatility: string; + evidence: string; +} + +export interface BuildDreamSessionsResult { + sessions: OrderedDreamSession[]; + missingSessionIds: string[]; + extraMemoryIds: string[]; +} + +export interface DreamLimits { + maxEntities: number; + maxDurableFacts: number; + maxCurrentStates: number; + maxTemporalEvents: number; + maxStateSummaryChars: number; +} + +const DEFAULT_LIMITS: DreamLimits = { + maxEntities: 80, + maxDurableFacts: 160, + maxCurrentStates: 60, + maxTemporalEvents: 80, + maxStateSummaryChars: 2500, +}; + +function asRecord(value: unknown): Record | null { + return value && typeof value === 'object' && !Array.isArray(value) + ? (value as Record) + : null; +} + +function asString(value: unknown): string | null { + return typeof value === 'string' && value.trim() ? value.trim() : null; +} + +function asArray(value: unknown): unknown[] { + return Array.isArray(value) ? value : []; +} + +function compactWhitespace(text: string): string { + return text.replace(/\s+/g, ' ').trim(); +} + +function clampText(text: string, maxChars: number): string { + const compacted = compactWhitespace(text); + if (compacted.length <= maxChars) return compacted; + return `${compacted.slice(0, Math.max(0, maxChars - 1)).trimEnd()}…`; +} + +function stripSessionHeader(content: string): string { + return content.replace(/^session\s+[^\r\n]+[\r\n]+/i, ''); +} + +function normalizeKey(text: string): string { + return compactWhitespace(text).toLowerCase(); +} + +function uniqueStrings(items: string[]): string[] { + return Array.from(new Set(items.map(compactWhitespace).filter(Boolean))); +} + +function mergeEvidenceIds(existing: string[], next: string): string[] { + return uniqueStrings([...existing, next]); +} + +function extractEntities(raw: unknown): ExtractedEntity[] { + const record = asRecord(raw); + return asArray(record?.entities) + .map((item) => { + const entity = asRecord(item); + const name = asString(entity?.name); + const description = asString(entity?.description); + const evidence = asString(entity?.evidence); + if (!name || !description || !evidence) return null; + return { + name, + aliases: asArray(entity?.aliases) + .map(asString) + .filter((alias): alias is string => Boolean(alias)), + entityType: asString(entity?.entityType) || undefined, + description, + evidence, + }; + }) + .filter((entity): entity is ExtractedEntity => Boolean(entity)); +} + +function extractDurableFacts(raw: unknown): ExtractedDurableFact[] { + const record = asRecord(raw); + return asArray(record?.durableFacts) + .map((item) => { + const fact = asRecord(item); + const factText = asString(fact?.fact); + const evidence = asString(fact?.evidence); + if (!factText || !evidence) return null; + return { + fact: factText, + category: asString(fact?.category) || 'other', + subject: asString(fact?.subject) || undefined, + object: asString(fact?.object) || undefined, + evidence, + }; + }) + .filter((fact): fact is ExtractedDurableFact => Boolean(fact)); +} + +function extractSummary(raw: unknown): ExtractedSummary | null { + const record = asRecord(raw); + const summary = asString(record?.summary); + const actionRelevance = asString(record?.actionRelevance); + if (!summary || !actionRelevance) return null; + return { + summary, + keyPoints: asArray(record?.keyPoints) + .map(asString) + .filter((point): point is string => Boolean(point)), + actionRelevance, + }; +} + +function extractCurrentState(raw: unknown): ExtractedCurrentState | null { + const record = asRecord(raw); + const state = asString(record?.state); + const scope = asString(record?.scope); + const status = asString(record?.status); + const evidence = asString(record?.evidence); + if (!state || !scope || !status || !evidence) return null; + return { + state, + scope, + status, + volatility: asString(record?.volatility) || 'semi-stable', + evidence, + }; +} + +export function parseLongMemSessionId(content: string): string | null { + const match = content.match(/^session\s+([^\r\n]+)[\r\n]+/i); + return match?.[1]?.trim() || null; +} + +export function extractDreamViews(metadata: Record | null): DreamExtractionViews { + const llmExtractions = asRecord(metadata?.llm_extractions); + return { + entities: extractEntities(llmExtractions?.entity), + durableFacts: extractDurableFacts(llmExtractions?.durable_fact), + summary: extractSummary(llmExtractions?.summary), + currentState: extractCurrentState(llmExtractions?.current_state), + }; +} + +export function buildOrderedDreamSessions( + dreamCase: LongMemEvalDreamCase, + memoryRows: DreamMemoryRow[] +): BuildDreamSessionsResult { + const bySessionId = new Map(); + + for (const row of memoryRows) { + const sessionId = parseLongMemSessionId(row.content); + if (!sessionId) continue; + bySessionId.set(sessionId, row); + } + + const sessions: OrderedDreamSession[] = []; + const missingSessionIds: string[] = []; + const consumedMemoryIds = new Set(); + + for (const rawSession of dreamCase.sessions) { + const memory = bySessionId.get(rawSession.sessionId); + if (!memory) { + missingSessionIds.push(rawSession.sessionId); + continue; + } + consumedMemoryIds.add(memory.id); + sessions.push({ + caseId: dreamCase.id, + sessionId: rawSession.sessionId, + memoryId: memory.id, + content: memory.content, + date: rawSession.date, + hasAnswer: rawSession.hasAnswer, + isAnswerSession: rawSession.isAnswerSession, + createdAt: memory.created_at, + sourceSummary: memory.summary, + extractions: extractDreamViews(memory.metadata), + }); + } + + const extraMemoryIds = memoryRows + .map((row) => row.id) + .filter((memoryId) => !consumedMemoryIds.has(memoryId)); + + return { sessions, missingSessionIds, extraMemoryIds }; +} + +export function createInitialDreamState(caseId: string, mode: DreamMode): DreamState { + return { + caseId, + mode, + sessionCount: 0, + stateSummary: '', + entities: [], + durableFacts: [], + currentStates: [], + temporalEvents: [], + evidenceMemoryIds: [], + evidenceSessionIds: [], + updatedAt: new Date(0).toISOString(), + }; +} + +function factKey(fact: ExtractedDurableFact): string { + // Local dreams are intentionally heuristic accumulators: including the fact + // text keeps differently worded observations distinct, so this reducer does + // not infer semantic supersession across "37 coins" -> "38 coins" updates. + // Real supersession is deferred to the future LLM dream updater. + return normalizeKey( + [fact.category, fact.subject || 'unknown-subject', fact.object || 'unknown-object', fact.fact] + .filter(Boolean) + .join('|') + ); +} + +function currentStateKey(state: ExtractedCurrentState): string { + return normalizeKey([state.scope, state.status, state.state].join('|')); +} + +function updateStateSummary(params: { + previousSummary: string; + session: OrderedDreamSession; + limits: DreamLimits; +}): string { + const sessionSummary = params.session.extractions.summary; + const nextLine = sessionSummary + ? `summary: ${sessionSummary.summary}` + : `summary: ${clampText(stripSessionHeader(params.session.content), 260)}`; + const combined = [params.previousSummary, nextLine].filter(Boolean).join('\n'); + if (combined.length <= params.limits.maxStateSummaryChars) return combined; + // Keep the tail deliberately: the local online reducer is recency-biased for + // "current state" questions, while raw evidence links preserve chronology. + return combined.slice(combined.length - params.limits.maxStateSummaryChars).trimStart(); +} + +function chooseEntityDescription( + existing: DreamEntity | undefined, + nextDescription: string +): string { + if (!existing) return nextDescription; + return nextDescription.length > existing.description.length + ? nextDescription + : existing.description; +} + +function moveToEnd(map: Map, key: TKey, value: TValue): void { + map.delete(key); + map.set(key, value); +} + +export function applyLocalDreamUpdate( + state: DreamState, + session: OrderedDreamSession, + limits: Partial = {} +): DreamState { + const resolvedLimits = { ...DEFAULT_LIMITS, ...limits }; + const entityMap = new Map(state.entities.map((entity) => [normalizeKey(entity.name), entity])); + const factMap = new Map(state.durableFacts.map((fact) => [fact.key, fact])); + const currentStateMap = new Map(state.currentStates.map((item) => [item.key, item])); + + for (const entity of session.extractions.entities) { + const key = normalizeKey(entity.name); + const existing = entityMap.get(key); + moveToEnd(entityMap, key, { + name: entity.name, + entityType: entity.entityType || existing?.entityType, + description: chooseEntityDescription(existing, entity.description), + aliases: uniqueStrings([...(existing?.aliases || []), ...entity.aliases]), + evidence: entity.evidence, + evidenceMemoryIds: mergeEvidenceIds(existing?.evidenceMemoryIds || [], session.memoryId), + evidenceSessionIds: mergeEvidenceIds(existing?.evidenceSessionIds || [], session.sessionId), + firstSeenSessionId: existing?.firstSeenSessionId || session.sessionId, + lastSeenSessionId: session.sessionId, + }); + } + + for (const fact of session.extractions.durableFacts) { + const key = factKey(fact); + const existing = factMap.get(key); + moveToEnd(factMap, key, { + key, + fact: fact.fact, + category: fact.category, + subject: fact.subject, + object: fact.object, + evidence: fact.evidence, + status: existing?.status || 'active', + evidenceMemoryIds: mergeEvidenceIds(existing?.evidenceMemoryIds || [], session.memoryId), + evidenceSessionIds: mergeEvidenceIds(existing?.evidenceSessionIds || [], session.sessionId), + firstSeenSessionId: existing?.firstSeenSessionId || session.sessionId, + lastSeenSessionId: session.sessionId, + }); + } + + if (session.extractions.currentState) { + const currentState = session.extractions.currentState; + const key = currentStateKey(currentState); + const existing = currentStateMap.get(key); + moveToEnd(currentStateMap, key, { + key, + state: currentState.state, + scope: currentState.scope, + status: currentState.status, + volatility: currentState.volatility, + evidence: currentState.evidence, + evidenceMemoryIds: mergeEvidenceIds(existing?.evidenceMemoryIds || [], session.memoryId), + evidenceSessionIds: mergeEvidenceIds(existing?.evidenceSessionIds || [], session.sessionId), + lastSeenSessionId: session.sessionId, + }); + } + + const summary = session.extractions.summary; + const eventSummary = summary + ? [summary.summary, ...summary.keyPoints.slice(0, 3)].join(' | ') + : clampText(stripSessionHeader(session.content), 400); + + const temporalEvents = [ + ...state.temporalEvents, + { + sessionId: session.sessionId, + memoryId: session.memoryId, + date: session.date, + orderIndex: state.sessionCount, + summary: eventSummary, + isAnswerSession: session.isAnswerSession, + }, + ].slice(-resolvedLimits.maxTemporalEvents); + + return { + ...state, + sessionCount: state.sessionCount + 1, + stateSummary: updateStateSummary({ + previousSummary: state.stateSummary, + session, + limits: resolvedLimits, + }), + entities: Array.from(entityMap.values()).slice(-resolvedLimits.maxEntities), + durableFacts: Array.from(factMap.values()).slice(-resolvedLimits.maxDurableFacts), + currentStates: Array.from(currentStateMap.values()).slice(-resolvedLimits.maxCurrentStates), + temporalEvents, + evidenceMemoryIds: mergeEvidenceIds(state.evidenceMemoryIds, session.memoryId), + evidenceSessionIds: mergeEvidenceIds(state.evidenceSessionIds, session.sessionId), + updatedAt: new Date().toISOString(), + }; +} + +export function renderDreamStateForAnswerCheck(state: DreamState): string { + // Render only semantic content for answer checks. Evidence ids, session ids, + // and counters are intentionally omitted so short answers like "38" do not + // get credit from lineage metadata instead of remembered content. + return [ + state.stateSummary, + ...state.entities.map( + (entity) => `${entity.name}: ${entity.description}; evidence: ${entity.evidence}` + ), + ...state.durableFacts.map( + (fact) => + `${fact.fact}; category: ${fact.category}; subject: ${fact.subject || ''}; object: ${ + fact.object || '' + }; evidence: ${fact.evidence}` + ), + ...state.currentStates.map( + (item) => + `${item.state}; scope: ${item.scope}; status: ${item.status}; evidence: ${item.evidence}` + ), + ...state.temporalEvents.map((event) => event.summary), + ].join('\n'); +} + +export function buildOnlineDreamPrompt(params: { + caseId: string; + question: string; + previousState: DreamState; + nextSession: OrderedDreamSession; +}): { systemPrompt: string; userPrompt: string; schemaDescription: string } { + return { + systemPrompt: + 'You are an online memory-dream worker. Integrate one new chronological episode into a compact, evidence-grounded state ledger. Return strict JSON only. Do not use benchmark answer labels. Do not review all past raw episodes; use the prior compact state plus this one new episode.', + schemaDescription: + 'JSON schema: {"stateSummary": string, "entities": [{"name": string, "entityType"?: string, "description": string, "aliases": string[], "evidenceMemoryIds": string[], "evidenceSessionIds": string[]}], "durableFacts": [{"key": string, "fact": string, "category": string, "subject"?: string, "object"?: string, "status": "active"|"superseded"|"uncertain", "evidenceMemoryIds": string[], "evidenceSessionIds": string[]}], "currentStates": [{"key": string, "state": string, "scope": string, "status": string, "volatility": string, "evidenceMemoryIds": string[], "evidenceSessionIds": string[]}], "temporalEvents": [{"sessionId": string, "memoryId": string, "date"?: string, "summary": string}], "notes": string[]}', + userPrompt: [ + `caseId: ${params.caseId}`, + `question for later evaluation only, not a label: ${params.question}`, + '', + 'Integration rules:', + '- Treat episodes as chronological within this case.', + '- Preserve current state updates, quantities, decisions, constraints, process rules, list/table mappings, and exact values when evidence supports them.', + '- If a new episode updates an old value, mark the old fact superseded and write the new active value with evidence links.', + '- If a value requires arithmetic or accumulation, perform the update and keep both evidence session ids.', + '- Keep the ledger compact. Do not copy the full transcript.', + '- Keep cases isolated: never infer from other cases.', + '', + 'Previous compact dream state JSON:', + JSON.stringify(params.previousState, null, 2), + '', + 'New episode JSON:', + JSON.stringify( + { + sessionId: params.nextSession.sessionId, + memoryId: params.nextSession.memoryId, + date: params.nextSession.date || null, + sourceSummary: params.nextSession.sourceSummary, + extractedViews: params.nextSession.extractions, + content: params.nextSession.content, + }, + null, + 2 + ), + ].join('\n'), + }; +} diff --git a/packages/benchmarks/src/benchmark-memory-dream.ts b/packages/benchmarks/src/benchmark-memory-dream.ts new file mode 100644 index 00000000..af7f7d57 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-dream.ts @@ -0,0 +1,267 @@ +import { mkdir, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { createSupabaseClient } from '@inklabs/api/benchmarks'; +import { loadLongMemEvalDreamDataset } from './benchmark-data/longmemeval-loader'; +import { hasOptionalAnswer } from './benchmark-answer-coverage'; +import { loadBenchmarkSeedState } from './benchmark-memory-recall.state'; +import { + applyLocalDreamUpdate, + buildOrderedDreamSessions, + createInitialDreamState, + renderDreamStateForAnswerCheck, + type DreamMemoryRow, + type DreamMode, + type DreamState, +} from './benchmark-memory-dream.logic'; + +const BENCHMARK_TOPIC = 'benchmark:memory-recall'; +const DEFAULT_PROGRESS_EVERY = 5; + +interface DreamCaseResult { + caseId: string; + query: string; + answer?: string; + questionType?: string; + questionDate?: string; + answerSessionIds: string[]; + sessionCount: number; + processedSessionCount: number; + missingSessionIds: string[]; + extraMemoryIds: string[]; + answerInDream: boolean | null; + answerInSource: boolean | null; + finalState: DreamState; + steps: DreamStepSummary[]; +} + +interface DreamStepSummary { + sessionId: string; + memoryId: string; + index: number; + hasAnswer: boolean; + isAnswerSession: boolean; + entityCount: number; + durableFactCount: number; + currentStateCount: number; + temporalEventCount: number; +} + +function parseBoolean(raw: string | undefined, defaultValue: boolean): boolean { + if (raw === undefined) return defaultValue; + return ['1', 'true', 'yes', 'on'].includes(raw.toLowerCase()); +} + +function parsePositiveInt(raw: string | undefined, defaultValue: number): number { + if (!raw) return defaultValue; + const parsed = Number(raw); + return Number.isFinite(parsed) && parsed > 0 ? Math.floor(parsed) : defaultValue; +} + +function parseDreamMode(raw: string | undefined): DreamMode { + const normalized = (raw || 'online').trim().toLowerCase(); + if (normalized === 'online' || normalized === 'batch') return normalized; + console.warn(`[memory-dream] unknown MEMORY_DREAM_MODE=${raw}; falling back to online`); + return 'online'; +} + +async function writeJsonOutput(outputPath: string, payload: unknown): Promise { + await mkdir(dirname(outputPath), { recursive: true }); + await writeFile(outputPath, JSON.stringify(payload, null, 2), 'utf-8'); +} + +async function loadCaseMemories(params: { + supabase: ReturnType; + userId: string; + topic: string; + limit: number; +}): Promise { + const { data, error } = await params.supabase + .from('memories') + .select('id,content,summary,metadata,created_at') + .eq('user_id', params.userId) + .contains('topics', [params.topic]) + .order('created_at', { ascending: true }) + .limit(params.limit); + + if (error) throw new Error(`Failed to load dream memories for ${params.topic}: ${error.message}`); + return (data || []) as DreamMemoryRow[]; +} + +function buildSourceText(sessions: { content: string }[]): string { + return sessions + .map((session) => session.content.replace(/^session\s+[^\r\n]+[\r\n]+/i, '')) + .join('\n\n'); +} + +function logProgress(params: { + completedCases: number; + totalCases: number; + processedSessions: number; + lastCaseId: string; + lastMs: number; +}) { + console.log( + `[memory-dream] progress cases=${params.completedCases}/${params.totalCases} ` + + `sessions=${params.processedSessions} lastCase=${params.lastCaseId} lastMs=${params.lastMs}` + ); +} + +async function main() { + const userId = process.env.BENCHMARK_USER_ID || process.env.MEMORY_DREAM_USER_ID; + if (!userId) { + throw new Error('BENCHMARK_USER_ID or MEMORY_DREAM_USER_ID is required'); + } + + const mode = parseDreamMode(process.env.MEMORY_DREAM_MODE); + if (mode === 'batch') { + console.warn( + '[memory-dream] MEMORY_DREAM_MODE=batch is reserved for a future batch reducer; current runner still applies the online local reducer' + ); + } + const strictSeed = parseBoolean(process.env.MEMORY_DREAM_STRICT_SEED, true); + const writeSteps = parseBoolean(process.env.MEMORY_DREAM_WRITE_STEPS, true); + const memoryLoadLimit = parsePositiveInt(process.env.MEMORY_DREAM_MEMORY_LOAD_LIMIT, 1000); + const progressEvery = parsePositiveInt( + process.env.MEMORY_DREAM_PROGRESS_EVERY, + DEFAULT_PROGRESS_EVERY + ); + const seedPath = + process.env.MEMORY_DREAM_SEED_PATH || + process.env.MEMORY_BENCHMARK_SEED_PATH || + resolve(process.cwd(), 'output', 'memory-benchmarks', 'memory-benchmark.seed.json'); + const runId = process.env.MEMORY_DREAM_RUN_ID || `memory-dream-${Date.now()}-${randomUUID()}`; + const outputPath = + process.env.MEMORY_DREAM_OUTPUT_PATH || + resolve(process.cwd(), 'output', 'memory-dreams', `${runId}.json`); + + const [{ cases, source }, seedState] = await Promise.all([ + loadLongMemEvalDreamDataset(), + loadBenchmarkSeedState(seedPath), + ]); + if (!seedState) throw new Error(`Missing seed state at ${seedPath}`); + + const supabase = createSupabaseClient(); + const results: DreamCaseResult[] = []; + let processedSessions = 0; + + console.log( + `[memory-dream] start mode=${mode} cases=${cases.length} seedId=${seedState.seedId} seedPath=${seedPath}` + ); + console.log(`[memory-dream] source=${source} outputPath=${outputPath} writeSteps=${writeSteps}`); + + for (const [caseIndex, dreamCase] of cases.entries()) { + const startedAt = Date.now(); + const seededCase = seedState.seededCases[dreamCase.id]; + if (!seededCase) { + const message = `Seed state does not include case ${dreamCase.id}`; + if (strictSeed) throw new Error(message); + console.warn(`[memory-dream] ${message}; skipping`); + continue; + } + + const rows = await loadCaseMemories({ + supabase, + userId, + topic: seededCase.topic || `${BENCHMARK_TOPIC}:${seedState.seedId}:${dreamCase.id}`, + limit: memoryLoadLimit, + }); + const ordered = buildOrderedDreamSessions(dreamCase, rows); + if (strictSeed && ordered.missingSessionIds.length > 0) { + throw new Error( + `Case ${dreamCase.id} is missing seeded sessions: ${ordered.missingSessionIds.join(', ')}` + ); + } + + let state = createInitialDreamState(dreamCase.id, mode); + const steps: DreamStepSummary[] = []; + + for (const [sessionIndex, session] of ordered.sessions.entries()) { + state = applyLocalDreamUpdate(state, session); + processedSessions += 1; + if (writeSteps) { + steps.push({ + sessionId: session.sessionId, + memoryId: session.memoryId, + index: sessionIndex, + hasAnswer: session.hasAnswer, + isAnswerSession: session.isAnswerSession, + entityCount: state.entities.length, + durableFactCount: state.durableFacts.length, + currentStateCount: state.currentStates.length, + temporalEventCount: state.temporalEvents.length, + }); + } + } + + const renderedDream = renderDreamStateForAnswerCheck(state); + const sourceText = buildSourceText(ordered.sessions); + results.push({ + caseId: dreamCase.id, + query: dreamCase.query, + answer: dreamCase.answer, + questionType: dreamCase.questionType, + questionDate: dreamCase.questionDate, + answerSessionIds: dreamCase.answerSessionIds, + sessionCount: dreamCase.sessions.length, + processedSessionCount: ordered.sessions.length, + missingSessionIds: ordered.missingSessionIds, + extraMemoryIds: ordered.extraMemoryIds, + answerInDream: hasOptionalAnswer(renderedDream, dreamCase.answer), + answerInSource: hasOptionalAnswer(sourceText, dreamCase.answer), + finalState: state, + steps, + }); + + if ((caseIndex + 1) % progressEvery === 0 || caseIndex === cases.length - 1) { + logProgress({ + completedCases: caseIndex + 1, + totalCases: cases.length, + processedSessions, + lastCaseId: dreamCase.id, + lastMs: Date.now() - startedAt, + }); + } + } + + const answerable = results.filter((result) => result.answerInSource === true); + const payload = { + runId, + settings: { + mode, + dataset: 'longmemeval-s-cleaned', + source, + seedId: seedState.seedId, + seedPath, + outputPath, + caseCount: cases.length, + strictSeed, + writeSteps, + memoryLoadLimit, + note: 'First-pass dream run uses existing per-memory LLM extraction views and a local online reducer. It does not write new DB memories or embeddings. Batch mode is accepted for future experiments but currently uses the same online reducer.', + }, + summary: { + cases: results.length, + processedSessions, + missingSessionCount: results.reduce( + (sum, result) => sum + result.missingSessionIds.length, + 0 + ), + answerInSource: results.filter((result) => result.answerInSource === true).length, + answerInDream: results.filter((result) => result.answerInDream === true).length, + answerInDreamWhenSourceHasAnswer: answerable.filter((result) => result.answerInDream === true) + .length, + answerableCases: answerable.length, + }, + results, + }; + + await writeJsonOutput(outputPath, payload); + console.log(JSON.stringify(payload.summary, null, 2)); + console.log(`[memory-dream] complete outputPath=${outputPath}`); +} + +main().catch((error) => { + console.error('[memory-dream] failed:', error); + process.exit(1); +}); diff --git a/packages/benchmarks/src/benchmark-memory-recall.config.test.ts b/packages/benchmarks/src/benchmark-memory-recall.config.test.ts new file mode 100644 index 00000000..3e5957f4 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.config.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, it, vi } from 'vitest'; +import { buildRepresentationKey, parseBenchmarkPhase } from './benchmark-memory-recall.config'; + +describe('benchmark-memory-recall config helpers', () => { + it('parses benchmark phase values', () => { + expect(parseBenchmarkPhase('seed')).toBe('seed'); + expect(parseBenchmarkPhase('recall')).toBe('recall'); + expect(parseBenchmarkPhase('all')).toBe('all'); + }); + + it('warns and falls back to all on unknown benchmark phase', () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined); + expect(parseBenchmarkPhase('oops')).toBe('all'); + expect(warn).toHaveBeenCalledWith( + expect.stringContaining('Unknown MEMORY_BENCHMARK_PHASE=oops') + ); + warn.mockRestore(); + }); + + it('includes chunk, extraction, and model dimensions in representation key', () => { + const key = buildRepresentationKey({ + MEMORY_EMBEDDINGS_ENABLED: 'true', + MEMORY_EMBEDDING_PROVIDER: 'openai', + MEMORY_EMBEDDING_MODEL: 'text-embedding-3-large', + MEMORY_EXTRACTION_MODE: 'llm', + MEMORY_LLM_EXTRACTION_ENABLED: 'true', + MEMORY_LLM_MODEL: 'gpt-5-mini', + MEMORY_LLM_ENTITY_ENABLED: 'true', + MEMORY_LLM_DURABLE_FACT_ENABLED: 'false', + MEMORY_LLM_SUMMARY_ENABLED: 'true', + MEMORY_LLM_CURRENT_STATE_ENABLED: 'false', + } as NodeJS.ProcessEnv); + + expect(key).toContain('chunks-v'); + expect(key).toContain('extract-v'); + expect(key).toContain('gpt-5'); + expect(key).toContain('llm'); + expect(key).toContain('text-embedding-3-large'); + }); +}); diff --git a/packages/benchmarks/src/benchmark-memory-recall.config.ts b/packages/benchmarks/src/benchmark-memory-recall.config.ts new file mode 100644 index 00000000..e99dd73d --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.config.ts @@ -0,0 +1,48 @@ +import { + DEFAULT_MEMORY_LLM_MODEL, + MEMORY_EMBEDDING_CHUNKS_VERSION, + MEMORY_EXTRACTION_VERSION, +} from '@inklabs/api/benchmark-constants'; + +export type BenchmarkPhase = 'all' | 'seed' | 'recall'; + +export function parseBenchmarkPhase(raw?: string): BenchmarkPhase { + const normalized = raw?.trim().toLowerCase(); + if (normalized === 'seed' || normalized === 'recall' || normalized === 'all') return normalized; + if (normalized) { + console.warn(`[memory-benchmark] Unknown MEMORY_BENCHMARK_PHASE=${raw}; using all`); + } + return 'all'; +} + +export function slugify(value: string): string { + return value + .toLowerCase() + .replace(/[^a-z0-9]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 80); +} + +/** + * Any setting that changes how memories are chunked, embedded, or extracted must be + * represented here. Missing dimensions cause stale seed reuse across config changes, + * which silently corrupts ablation comparisons. + */ +export function buildRepresentationKey(env: NodeJS.ProcessEnv = process.env): string { + const parts = [ + 'chunked', + `chunks-v${MEMORY_EMBEDDING_CHUNKS_VERSION}`, + env.MEMORY_EMBEDDINGS_ENABLED || 'default', + env.MEMORY_EMBEDDING_PROVIDER || 'default', + env.MEMORY_EMBEDDING_MODEL || 'default', + env.MEMORY_EXTRACTION_MODE || 'heuristic', + `extract-v${MEMORY_EXTRACTION_VERSION}`, + env.MEMORY_LLM_EXTRACTION_ENABLED || 'false', + env.MEMORY_LLM_MODEL || DEFAULT_MEMORY_LLM_MODEL, + env.MEMORY_LLM_ENTITY_ENABLED || 'false', + env.MEMORY_LLM_DURABLE_FACT_ENABLED || 'false', + env.MEMORY_LLM_SUMMARY_ENABLED || 'false', + env.MEMORY_LLM_CURRENT_STATE_ENABLED || 'false', + ]; + return slugify(parts.join('-')); +} diff --git a/packages/benchmarks/src/benchmark-memory-recall.state.test.ts b/packages/benchmarks/src/benchmark-memory-recall.state.test.ts new file mode 100644 index 00000000..48c6abb7 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.state.test.ts @@ -0,0 +1,144 @@ +import { mkdtemp, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { + createInitialBenchmarkSeedState, + createInitialBenchmarkRunState, + estimateRemainingDuration, + formatDurationMs, + loadBenchmarkRunState, + loadBenchmarkSeedState, +} from './benchmark-memory-recall.state'; + +describe('benchmark-memory-recall state helpers', () => { + it('creates an empty initial state', () => { + const state = createInitialBenchmarkRunState({ + runId: 'membench-test', + seedId: 'seed-test', + dataset: 'longmemeval-s-cleaned', + datasetSource: 'url:test', + benchmarkFamily: 'longmemeval', + variant: 'default', + userId: 'user-123', + modes: ['text', 'semantic', 'hybrid'], + outputPath: '/tmp/out.json', + }); + + expect(state.runId).toBe('membench-test'); + expect(state.seedId).toBe('seed-test'); + expect(state.seededCases).toEqual({}); + expect(state.completedRuns).toEqual({}); + expect(state.timings).toEqual({ + seedCaseCount: 0, + seedTotalMs: 0, + recallCaseCount: 0, + recallTotalMs: 0, + }); + }); + + it('creates an empty initial seed state', () => { + const state = createInitialBenchmarkSeedState({ + seedId: 'seed-test', + dataset: 'longmemeval-s-cleaned', + datasetSource: 'url:test', + benchmarkFamily: 'longmemeval', + userId: 'user-123', + representationKey: 'chunked-default', + }); + + expect(state.seedId).toBe('seed-test'); + expect(state.representationKey).toBe('chunked-default'); + expect(state.seededCases).toEqual({}); + }); + + it('formats durations for logs', () => { + expect(formatDurationMs(250)).toBe('250ms'); + expect(formatDurationMs(1500)).toBe('1.5s'); + expect(formatDurationMs(65000)).toBe('1m 5s'); + }); + + it('estimates remaining duration from average case time', () => { + expect( + estimateRemainingDuration({ + completed: 20, + total: 100, + averageMs: 2000, + }) + ).toBe('2m 40s'); + }); + + it('normalizes legacy seeded state with singular targetMemoryId', async () => { + const dir = await mkdtemp(join(tmpdir(), 'membench-state-')); + const file = join(dir, 'state.json'); + await writeFile( + file, + JSON.stringify({ + runId: 'legacy', + dataset: 'longmemeval-s-cleaned', + datasetSource: 'url:test', + benchmarkFamily: 'longmemeval', + userId: 'user-123', + modes: ['semantic'], + outputPath: '/tmp/out.json', + seededCases: { + caseA: { + caseId: 'caseA', + topic: 'benchmark:caseA', + targetMemoryId: 'memory-1', + distractorMemoryIds: ['memory-2'], + seedMs: 100, + }, + }, + completedRuns: {}, + timings: { + seedCaseCount: 1, + seedTotalMs: 100, + recallCaseCount: 0, + recallTotalMs: 0, + }, + }), + 'utf-8' + ); + + const state = await loadBenchmarkRunState(file); + expect(state?.variant).toBe('default'); + expect(state?.seedId).toBe('legacy'); + expect(state?.seededCases.caseA.targetMemoryIds).toEqual(['memory-1']); + }); + + it('loads benchmark seed state with legacy singular targetMemoryId', async () => { + const dir = await mkdtemp(join(tmpdir(), 'membench-seed-')); + const file = join(dir, 'seed.json'); + await writeFile( + file, + JSON.stringify({ + seedId: 'seed-legacy', + dataset: 'longmemeval-s-cleaned', + datasetSource: 'url:test', + benchmarkFamily: 'longmemeval', + userId: 'user-123', + representationKey: 'chunked-default', + seededCases: { + caseA: { + caseId: 'caseA', + topic: 'benchmark:caseA', + targetMemoryId: 'memory-1', + distractorMemoryIds: ['memory-2'], + seedMs: 100, + }, + }, + timings: { + seedCaseCount: 1, + seedTotalMs: 100, + recallCaseCount: 0, + recallTotalMs: 0, + }, + }), + 'utf-8' + ); + + const state = await loadBenchmarkSeedState(file); + expect(state?.seededCases.caseA.targetMemoryIds).toEqual(['memory-1']); + }); +}); diff --git a/packages/benchmarks/src/benchmark-memory-recall.state.ts b/packages/benchmarks/src/benchmark-memory-recall.state.ts new file mode 100644 index 00000000..6b77ffe3 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.state.ts @@ -0,0 +1,196 @@ +import { mkdir, readFile, writeFile } from 'node:fs/promises'; +import { dirname } from 'node:path'; +import type { RecallMode } from './benchmark-memory-recall.types'; +import type { BenchmarkRecallVariant } from './benchmark-memory-recall.variant'; + +export interface SeededCaseState { + caseId: string; + topic: string; + targetMemoryIds: string[]; + distractorMemoryIds: string[]; + seedMs: number; +} + +export interface BenchmarkSeedState { + seedId: string; + dataset: string; + datasetSource: string; + benchmarkFamily: string | null; + userId: string; + representationKey: string; + seededCases: Record; + timings: BenchmarkTimingState; +} + +export interface CompletedCaseRunState { + rank: number | null; + topSummaries: string[]; + recallMs: number; +} + +export interface BenchmarkTimingState { + seedCaseCount: number; + seedTotalMs: number; + recallCaseCount: number; + recallTotalMs: number; +} + +export interface BenchmarkRunState { + runId: string; + seedId: string; + dataset: string; + datasetSource: string; + benchmarkFamily: string | null; + variant: BenchmarkRecallVariant; + userId: string; + modes: RecallMode[]; + outputPath: string; + seededCases: Record; + completedRuns: Partial>>; + timings: BenchmarkTimingState; +} + +export function createInitialBenchmarkSeedState(params: { + seedId: string; + dataset: string; + datasetSource: string; + benchmarkFamily: string | null; + userId: string; + representationKey: string; +}): BenchmarkSeedState { + return { + ...params, + seededCases: {}, + timings: { + seedCaseCount: 0, + seedTotalMs: 0, + recallCaseCount: 0, + recallTotalMs: 0, + }, + }; +} + +export function createInitialBenchmarkRunState(params: { + runId: string; + seedId: string; + dataset: string; + datasetSource: string; + benchmarkFamily: string | null; + variant: BenchmarkRecallVariant; + userId: string; + modes: RecallMode[]; + outputPath: string; +}): BenchmarkRunState { + return { + ...params, + seededCases: {}, + completedRuns: {}, + timings: { + seedCaseCount: 0, + seedTotalMs: 0, + recallCaseCount: 0, + recallTotalMs: 0, + }, + }; +} + +export async function loadBenchmarkSeedState( + statePath: string +): Promise { + try { + const raw = await readFile(statePath, 'utf-8'); + const parsed = JSON.parse(raw) as BenchmarkSeedState & { + seededCases?: Record< + string, + SeededCaseState & { + targetMemoryId?: string; + } + >; + }; + + if (parsed.seededCases) { + for (const seededCase of Object.values(parsed.seededCases)) { + if (!Array.isArray(seededCase.targetMemoryIds)) { + seededCase.targetMemoryIds = seededCase.targetMemoryId ? [seededCase.targetMemoryId] : []; + } + } + } + + return parsed as BenchmarkSeedState; + } catch (error) { + const nodeError = error as NodeJS.ErrnoException; + if (nodeError?.code === 'ENOENT') return null; + throw error; + } +} + +export async function loadBenchmarkRunState(statePath: string): Promise { + try { + const raw = await readFile(statePath, 'utf-8'); + const parsed = JSON.parse(raw) as BenchmarkRunState & { + seededCases?: Record< + string, + SeededCaseState & { + targetMemoryId?: string; + } + >; + }; + + if (parsed.seededCases) { + for (const seededCase of Object.values(parsed.seededCases)) { + if (!Array.isArray(seededCase.targetMemoryIds)) { + seededCase.targetMemoryIds = seededCase.targetMemoryId ? [seededCase.targetMemoryId] : []; + } + } + } + + if (!parsed.variant) { + parsed.variant = 'default'; + } + + if (!parsed.seedId) { + parsed.seedId = parsed.runId; + } + + return parsed as BenchmarkRunState; + } catch (error) { + const nodeError = error as NodeJS.ErrnoException; + if (nodeError?.code === 'ENOENT') return null; + throw error; + } +} + +export async function writeBenchmarkRunState( + statePath: string, + state: BenchmarkRunState +): Promise { + await mkdir(dirname(statePath), { recursive: true }); + await writeFile(statePath, JSON.stringify(state, null, 2), 'utf-8'); +} + +export async function writeBenchmarkSeedState( + statePath: string, + state: BenchmarkSeedState +): Promise { + await mkdir(dirname(statePath), { recursive: true }); + await writeFile(statePath, JSON.stringify(state, null, 2), 'utf-8'); +} + +export function formatDurationMs(durationMs: number): string { + if (durationMs < 1000) return `${durationMs}ms`; + const seconds = durationMs / 1000; + if (seconds < 60) return `${seconds.toFixed(1)}s`; + const minutes = Math.floor(seconds / 60); + const remainingSeconds = Math.round(seconds % 60); + return `${minutes}m ${remainingSeconds}s`; +} + +export function estimateRemainingDuration(params: { + completed: number; + total: number; + averageMs: number; +}): string { + const remaining = Math.max(0, params.total - params.completed); + if (remaining === 0 || params.averageMs <= 0) return '0ms'; + return formatDurationMs(Math.round(remaining * params.averageMs)); +} diff --git a/packages/benchmarks/src/benchmark-memory-recall.ts b/packages/benchmarks/src/benchmark-memory-recall.ts new file mode 100644 index 00000000..54edc3c9 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.ts @@ -0,0 +1,659 @@ +import { mkdir, writeFile } from 'node:fs/promises'; +import { dirname, resolve } from 'node:path'; +import { randomUUID } from 'node:crypto'; +import { createSupabaseClient, MemoryRepository } from '@inklabs/api/benchmarks'; +import { + buildRepresentationKey, + parseBenchmarkPhase, + slugify, +} from './benchmark-memory-recall.config'; +import { getBenchmarkDataset } from './benchmark-data/datasets'; +import { loadHfBenchmarkDataset } from './benchmark-data/hf-loader'; +import { loadLoCoMoDataset } from './benchmark-data/locomo-loader'; +import { loadLongMemEvalDataset } from './benchmark-data/longmemeval-loader'; +import { + PUBLIC_BENCHMARKS, + type PublicBenchmarkFamily, + getPublicBenchmarkDescriptor, +} from './benchmark-data/public-benchmarks'; +import { + buildBenchmarkRecallOptions, + describeBenchmarkRecallVariant, + parseBenchmarkRecallVariant, +} from './benchmark-memory-recall.variant'; +import { + createInitialBenchmarkSeedState, + createInitialBenchmarkRunState, + estimateRemainingDuration, + formatDurationMs, + loadBenchmarkRunState, + loadBenchmarkSeedState, + writeBenchmarkSeedState, + writeBenchmarkRunState, +} from './benchmark-memory-recall.state'; +import type { RecallMode } from './benchmark-memory-recall.types'; + +function parseBenchmarkFamily(raw?: string): PublicBenchmarkFamily | null { + if (!raw) return null; + const normalized = raw.trim().toLowerCase(); + const match = PUBLIC_BENCHMARKS.find((entry) => entry.family === normalized); + return match ? match.family : null; +} + +interface CaseRun { + caseId: string; + query: string; + mode: RecallMode; + rank: number | null; + topSummaries: string[]; +} + +interface SummaryMetric { + mode: RecallMode; + cases: number; + recallAt1: number; + recallAt3: number; + recallAt5: number; + mrr: number; +} + +const TOP_K = 5; +const BENCHMARK_TOPIC = 'benchmark:memory-recall'; +const BENCHMARK_AGENT_ID = 'lumen'; +const DEFAULT_DATASET = 'internal-gold-v1'; +const RETRY_ATTEMPTS = 3; +const DEFAULT_PROGRESS_EVERY = 25; + +function parseModes(raw?: string): RecallMode[] { + if (!raw) return ['text', 'semantic', 'hybrid']; + const parsed = raw + .split(',') + .map((m) => m.trim()) + .filter(Boolean) as RecallMode[]; + return parsed.length > 0 ? parsed : ['text', 'semantic', 'hybrid']; +} + +function mean(values: number[]): number { + if (values.length === 0) return 0; + return values.reduce((acc, v) => acc + v, 0) / values.length; +} + +function round(value: number): number { + return Number(value.toFixed(4)); +} + +function parseBoolean(raw: string | undefined, defaultValue: boolean): boolean { + if (raw === undefined) return defaultValue; + return ['1', 'true', 'yes', 'on'].includes(raw.toLowerCase()); +} + +function parsePositiveInt(raw: string | undefined, defaultValue: number): number { + if (!raw) return defaultValue; + const parsed = Number(raw); + if (!Number.isFinite(parsed) || parsed <= 0) return defaultValue; + return Math.floor(parsed); +} + +function sleep(ms: number): Promise { + return new Promise((resolve) => setTimeout(resolve, ms)); +} + +function getTargetDocuments(benchCase: { + targetContent?: string; + targetContents?: string[]; +}): string[] { + if (Array.isArray(benchCase.targetContents) && benchCase.targetContents.length > 0) { + return benchCase.targetContents; + } + + if (typeof benchCase.targetContent === 'string' && benchCase.targetContent.trim().length > 0) { + return [benchCase.targetContent]; + } + + throw new Error('Benchmark case must define targetContent or targetContents.'); +} + +async function withRetries(label: string, fn: () => Promise): Promise { + let lastError: unknown = null; + + for (let attempt = 1; attempt <= RETRY_ATTEMPTS; attempt += 1) { + try { + return await fn(); + } catch (error) { + lastError = error; + if (attempt === RETRY_ATTEMPTS) break; + console.warn( + `[memory-benchmark] ${label} failed on attempt ${attempt}/${RETRY_ATTEMPTS}; retrying...`, + error + ); + await sleep(250 * attempt); + } + } + + throw lastError; +} + +function buildSummaryMetrics(modes: RecallMode[], runs: CaseRun[]): SummaryMetric[] { + return modes.map((mode) => { + const modeRuns = runs.filter((r) => r.mode === mode); + const reciprocalRanks = modeRuns.map((r) => (r.rank ? 1 / r.rank : 0)); + const hitsAt1 = modeRuns.filter((r) => r.rank === 1).length / modeRuns.length; + const hitsAt3 = modeRuns.filter((r) => r.rank !== null && r.rank <= 3).length / modeRuns.length; + const hitsAt5 = modeRuns.filter((r) => r.rank !== null && r.rank <= 5).length / modeRuns.length; + + return { + mode, + cases: modeRuns.length, + recallAt1: round(hitsAt1), + recallAt3: round(hitsAt3), + recallAt5: round(hitsAt5), + mrr: round(mean(reciprocalRanks)), + }; + }); +} + +async function persistRun( + supabase: any, + params: { + runId: string; + userId: string; + dataset: string; + topK: number; + caseCount: number; + modes: RecallMode[]; + summary: SummaryMetric[]; + runs: CaseRun[]; + datasetSource: string; + benchmarkFamily: PublicBenchmarkFamily | null; + variantName: string; + } +): Promise { + const { + runId, + userId, + dataset, + topK, + caseCount, + modes, + summary, + runs, + datasetSource, + benchmarkFamily, + variantName, + } = params; + + const modeRows = summary.map((metric) => ({ + run_id: runId, + mode: metric.mode, + cases: metric.cases, + recall_at_1: metric.recallAt1, + recall_at_3: metric.recallAt3, + recall_at_5: metric.recallAt5, + mrr: metric.mrr, + })); + + const caseRows = runs.map((run) => ({ + run_id: runId, + case_id: run.caseId, + mode: run.mode, + query: run.query, + rank: run.rank, + top_summaries: run.topSummaries, + })); + + const runRow = { + run_id: runId, + user_id: userId, + dataset, + provider: process.env.MEMORY_EMBEDDING_PROVIDER || 'default', + model: process.env.MEMORY_EMBEDDING_MODEL || 'default', + embeddings_enabled: parseBoolean(process.env.MEMORY_EMBEDDINGS_ENABLED, false), + top_k: topK, + case_count: caseCount, + modes, + summary, + metadata: { + benchmarkTopic: BENCHMARK_TOPIC, + benchmarkAgentId: BENCHMARK_AGENT_ID, + datasetSource, + benchmarkFamily, + variant: variantName, + benchmarkFamilyDescriptor: benchmarkFamily + ? getPublicBenchmarkDescriptor(benchmarkFamily) + : null, + }, + }; + + const { error: runError } = await supabase.from('memory_recall_benchmark_runs').insert(runRow); + if (runError) throw new Error(`Failed to persist benchmark run: ${runError.message}`); + + const { error: metricsError } = await supabase + .from('memory_recall_benchmark_metrics') + .insert(modeRows); + if (metricsError) throw new Error(`Failed to persist benchmark metrics: ${metricsError.message}`); + + const { error: caseError } = await supabase + .from('memory_recall_benchmark_case_results') + .insert(caseRows); + if (caseError) throw new Error(`Failed to persist benchmark case results: ${caseError.message}`); +} + +async function writeJsonOutput(outputPath: string, payload: unknown): Promise { + await mkdir(dirname(outputPath), { recursive: true }); + await writeFile(outputPath, JSON.stringify(payload, null, 2), 'utf-8'); +} + +function logProgress(params: { + label: string; + completed: number; + total: number; + durationMs: number; + averageMs: number; +}) { + console.log( + `[memory-benchmark] ${params.label} ${params.completed}/${params.total} ` + + `last=${formatDurationMs(params.durationMs)} avg=${formatDurationMs(Math.round(params.averageMs))} ` + + `eta=${estimateRemainingDuration({ + completed: params.completed, + total: params.total, + averageMs: params.averageMs, + })}` + ); +} + +async function loadBenchmarkCases(dataset: string) { + if (dataset === 'hf') { + const hf = await loadHfBenchmarkDataset(); + return { cases: hf.cases, source: hf.source }; + } + + if (dataset === 'longmemeval-s-cleaned') { + const longmemeval = await loadLongMemEvalDataset(); + return { cases: longmemeval.cases, source: longmemeval.source }; + } + + if (dataset === 'locomo10') { + const locomo = await loadLoCoMoDataset(); + return { cases: locomo.cases, source: locomo.source }; + } + + return { cases: getBenchmarkDataset(dataset), source: `builtin:${dataset}` }; +} + +async function main() { + const userId = process.env.BENCHMARK_USER_ID; + if (!userId) { + throw new Error( + 'BENCHMARK_USER_ID is required. Example: BENCHMARK_USER_ID= yarn benchmark:memory-recall' + ); + } + + const dataset = process.env.MEMORY_BENCHMARK_DATASET || DEFAULT_DATASET; + const { cases: benchmarkCases, source: datasetSource } = await loadBenchmarkCases(dataset); + const benchmarkFamily = parseBenchmarkFamily(process.env.MEMORY_BENCHMARK_FAMILY); + const variant = parseBenchmarkRecallVariant(process.env.MEMORY_BENCHMARK_VARIANT); + const modes = parseModes(process.env.MEMORY_BENCHMARK_MODES); + const persistResults = parseBoolean(process.env.MEMORY_BENCHMARK_PERSIST, true); + const writeOutputFile = parseBoolean(process.env.MEMORY_BENCHMARK_WRITE_FILE, true); + const progressEvery = parsePositiveInt( + process.env.MEMORY_BENCHMARK_PROGRESS_EVERY, + DEFAULT_PROGRESS_EVERY + ); + const phase = parseBenchmarkPhase(process.env.MEMORY_BENCHMARK_PHASE); + const representationKey = buildRepresentationKey(); + + const requestedRunId = process.env.MEMORY_BENCHMARK_RUN_ID; + const runId = requestedRunId || `membench-${Date.now()}-${randomUUID().slice(0, 8)}`; + const seedId = + process.env.MEMORY_BENCHMARK_SEED_ID || + `${slugify(dataset)}-${benchmarkFamily || 'custom'}-${representationKey}`; + const outputPath = + process.env.MEMORY_BENCHMARK_OUTPUT_PATH || + resolve(process.cwd(), 'output', 'memory-benchmarks', `${runId}.json`); + const statePath = + process.env.MEMORY_BENCHMARK_STATE_PATH || + resolve(process.cwd(), 'output', 'memory-benchmarks', `${runId}.state.json`); + const seedPath = + process.env.MEMORY_BENCHMARK_SEED_PATH || + resolve(process.cwd(), 'output', 'memory-benchmarks', `${seedId}.seed.json`); + const existingSeedState = await loadBenchmarkSeedState(seedPath); + if ( + existingSeedState?.representationKey !== undefined && + existingSeedState.representationKey !== representationKey + ) { + console.warn( + `[memory-benchmark] loaded seed has representationKey=${existingSeedState.representationKey} ` + + `but current config is ${representationKey}. Results may not reflect current memory pipeline.` + ); + } + const existingState = await loadBenchmarkRunState(statePath); + const reuseSeeded = parseBoolean( + process.env.MEMORY_BENCHMARK_REUSE_SEEDED, + !!existingSeedState || !!existingState + ); + const keepSeeded = parseBoolean(process.env.MEMORY_BENCHMARK_KEEP_SEEDED, true); + const variantDescriptor = describeBenchmarkRecallVariant(variant); + + const supabase = createSupabaseClient(); + const repo = new MemoryRepository(supabase); + const createdMemoryIds: string[] = []; + + const caseTargets: Record = {}; + const caseTopics: Record = {}; + const seedState = + existingSeedState || + createInitialBenchmarkSeedState({ + seedId, + dataset, + datasetSource, + benchmarkFamily, + userId, + representationKey, + }); + const runState = + existingState || + createInitialBenchmarkRunState({ + runId, + seedId, + dataset, + datasetSource, + benchmarkFamily, + variant, + userId, + modes, + outputPath, + }); + + if (!existingSeedState) { + await writeBenchmarkSeedState(seedPath, seedState); + } + + if (existingState) { + console.log( + `[memory-benchmark] Resuming run ${runState.runId} from ${statePath} ` + + `(seeded=${Object.keys(runState.seededCases).length}, ` + + `completed=${Object.values(runState.completedRuns).reduce((acc, runs) => acc + Object.keys(runs || {}).length, 0)})` + ); + } else { + await writeBenchmarkRunState(statePath, runState); + } + + console.log( + `[memory-benchmark] phase=${phase} seedId=${seedId} representation=${representationKey} seedPath=${seedPath}` + ); + console.log( + `[memory-benchmark] variant=${variantDescriptor.name} ` + + `semanticChunkTypes=${Array.isArray(variantDescriptor.semanticChunkTypes) ? variantDescriptor.semanticChunkTypes.join('|') : variantDescriptor.semanticChunkTypes} ` + + `hybridChunkStrategy=${variantDescriptor.hybridChunkStrategy} ` + + `chunkBoost=${variantDescriptor.applyChunkTypeBoosts} multiView=${variantDescriptor.applyMultiViewBoost} chronology=${variantDescriptor.applyChronologyBoost}` + ); + + try { + for (const [index, benchCase] of benchmarkCases.entries()) { + const seedTopic = `${BENCHMARK_TOPIC}:${seedState.seedId}`; + const caseTopic = `${BENCHMARK_TOPIC}:${seedState.seedId}:${benchCase.id}`; + caseTopics[benchCase.id] = [caseTopic]; + + const seededCase = seedState.seededCases[benchCase.id] || runState.seededCases[benchCase.id]; + if (reuseSeeded && seededCase) { + caseTargets[benchCase.id] = seededCase.targetMemoryIds; + caseTopics[benchCase.id] = [seededCase.topic]; + runState.seededCases[benchCase.id] = seededCase; + continue; + } + + if (phase === 'recall') { + throw new Error( + `Missing seeded case for ${benchCase.id} in recall-only mode. Seed path: ${seedPath}` + ); + } + + const seedStartedAt = Date.now(); + const targetMemoryIds: string[] = []; + const targetDocuments = getTargetDocuments(benchCase); + + for (let i = 0; i < targetDocuments.length; i += 1) { + const target = await withRetries( + `remember target ${benchCase.id}${targetDocuments.length > 1 ? ` #${i + 1}` : ''}`, + () => + repo.remember({ + userId, + agentId: BENCHMARK_AGENT_ID, + content: targetDocuments[i], + summary: + targetDocuments.length > 1 + ? `benchmark target ${benchCase.id} #${i + 1}` + : `benchmark target ${benchCase.id}`, + source: 'observation', + salience: 'low', + topicKey: BENCHMARK_TOPIC, + topics: [BENCHMARK_TOPIC, seedTopic, caseTopic], + }) + ); + createdMemoryIds.push(target.id); + targetMemoryIds.push(target.id); + } + caseTargets[benchCase.id] = targetMemoryIds; + const distractorIds: string[] = []; + + for (let i = 0; i < benchCase.distractors.length; i += 1) { + const distractor = await withRetries(`remember distractor ${benchCase.id} #${i + 1}`, () => + repo.remember({ + userId, + agentId: BENCHMARK_AGENT_ID, + content: benchCase.distractors[i], + summary: `benchmark distractor ${benchCase.id} #${i + 1}`, + source: 'observation', + salience: 'low', + topicKey: BENCHMARK_TOPIC, + topics: [BENCHMARK_TOPIC, seedTopic, caseTopic], + }) + ); + createdMemoryIds.push(distractor.id); + distractorIds.push(distractor.id); + } + + const seedMs = Date.now() - seedStartedAt; + const seededCaseState = { + caseId: benchCase.id, + topic: caseTopic, + targetMemoryIds, + distractorMemoryIds: distractorIds, + seedMs, + }; + seedState.seededCases[benchCase.id] = seededCaseState; + runState.seededCases[benchCase.id] = seededCaseState; + seedState.timings.seedCaseCount += 1; + seedState.timings.seedTotalMs += seedMs; + runState.timings.seedCaseCount = seedState.timings.seedCaseCount; + runState.timings.seedTotalMs = seedState.timings.seedTotalMs; + await writeBenchmarkSeedState(seedPath, seedState); + await writeBenchmarkRunState(statePath, runState); + + if ((index + 1) % progressEvery === 0 || index === benchmarkCases.length - 1) { + logProgress({ + label: 'seeded cases', + completed: Object.keys(seedState.seededCases).length, + total: benchmarkCases.length, + durationMs: seedMs, + averageMs: seedState.timings.seedTotalMs / Math.max(1, seedState.timings.seedCaseCount), + }); + } + } + + if (phase === 'seed') { + console.log( + JSON.stringify( + { + seedId, + seedPath, + dataset, + datasetSource, + benchmarkFamily, + representationKey, + seededCaseCount: Object.keys(seedState.seededCases).length, + timings: { + seedCaseCount: seedState.timings.seedCaseCount, + seedTotalMs: seedState.timings.seedTotalMs, + seedAverageMs: + seedState.timings.seedTotalMs / Math.max(1, seedState.timings.seedCaseCount), + }, + }, + null, + 2 + ) + ); + return; + } + + const runs: CaseRun[] = []; + + for (const mode of modes) { + const completedForMode = (runState.completedRuns[mode] ||= {}); + console.log( + `[memory-benchmark] starting recall mode=${mode} completed=${Object.keys(completedForMode).length}/${benchmarkCases.length}` + ); + + for (const [index, benchCase] of benchmarkCases.entries()) { + const resumed = completedForMode[benchCase.id]; + if (resumed) { + runs.push({ + caseId: benchCase.id, + query: benchCase.query, + mode, + rank: resumed.rank, + topSummaries: resumed.topSummaries, + }); + continue; + } + + const recallStartedAt = Date.now(); + const results = await withRetries(`recall ${benchCase.id} (${mode})`, () => + repo.recall( + userId, + benchCase.query, + buildBenchmarkRecallOptions({ + mode, + variant, + limit: TOP_K, + agentId: BENCHMARK_AGENT_ID, + topics: caseTopics[benchCase.id], + }) + ) + ); + + const expectedIds = new Set(caseTargets[benchCase.id]); + const rank = results.findIndex((m) => expectedIds.has(m.id)); + const recallMs = Date.now() - recallStartedAt; + + const caseRun: CaseRun = { + caseId: benchCase.id, + query: benchCase.query, + mode, + rank: rank >= 0 ? rank + 1 : null, + topSummaries: results.map((m) => m.summary || m.content.slice(0, 80)), + }; + runs.push(caseRun); + completedForMode[benchCase.id] = { + rank: caseRun.rank, + topSummaries: caseRun.topSummaries, + recallMs, + }; + runState.timings.recallCaseCount += 1; + runState.timings.recallTotalMs += recallMs; + await writeBenchmarkRunState(statePath, runState); + + const completedCount = Object.keys(completedForMode).length; + if (completedCount % progressEvery === 0 || index === benchmarkCases.length - 1) { + logProgress({ + label: `recalled ${mode}`, + completed: completedCount, + total: benchmarkCases.length, + durationMs: recallMs, + averageMs: + runState.timings.recallTotalMs / Math.max(1, runState.timings.recallCaseCount), + }); + } + } + } + + const summary = buildSummaryMetrics(modes, runs); + + if (persistResults) { + await persistRun(supabase as any, { + runId, + userId, + dataset, + topK: TOP_K, + caseCount: benchmarkCases.length, + modes, + summary, + runs, + datasetSource, + benchmarkFamily, + variantName: variantDescriptor.name, + }); + } + + const payload = { + runId, + settings: { + dataset, + model: process.env.MEMORY_EMBEDDING_MODEL || 'default', + provider: process.env.MEMORY_EMBEDDING_PROVIDER || 'default', + embeddingsEnabled: process.env.MEMORY_EMBEDDINGS_ENABLED || 'default', + topK: TOP_K, + benchmarkCases: benchmarkCases.length, + persistResults, + datasetSource, + benchmarkFamily, + benchmarkFamilyDescriptor: benchmarkFamily + ? getPublicBenchmarkDescriptor(benchmarkFamily) + : null, + variant: variantDescriptor, + statePath, + seedPath, + reuseSeeded, + keepSeeded, + phase, + seedCaseCount: Object.keys(seedState.seededCases).length, + timings: { + seedCaseCount: seedState.timings.seedCaseCount, + seedTotalMs: seedState.timings.seedTotalMs, + seedAverageMs: + seedState.timings.seedTotalMs / Math.max(1, seedState.timings.seedCaseCount), + recallCaseCount: runState.timings.recallCaseCount, + recallTotalMs: runState.timings.recallTotalMs, + recallAverageMs: + runState.timings.recallTotalMs / Math.max(1, runState.timings.recallCaseCount), + }, + }, + summary, + runs, + outputPath: writeOutputFile ? outputPath : null, + statePath, + seedPath, + }; + + if (writeOutputFile) { + await writeJsonOutput(outputPath, payload); + } + + console.log(JSON.stringify(payload, null, 2)); + } finally { + if (!keepSeeded) { + for (const memoryId of createdMemoryIds) { + try { + await repo.forget(memoryId, userId); + } catch { + // best-effort cleanup + } + } + } + } +} + +main().catch((error) => { + console.error('[memory-benchmark] failed:', error); + process.exit(1); +}); diff --git a/packages/benchmarks/src/benchmark-memory-recall.types.ts b/packages/benchmarks/src/benchmark-memory-recall.types.ts new file mode 100644 index 00000000..58bcef78 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.types.ts @@ -0,0 +1 @@ +export type RecallMode = 'text' | 'semantic' | 'hybrid' | 'auto'; diff --git a/packages/benchmarks/src/benchmark-memory-recall.variant.test.ts b/packages/benchmarks/src/benchmark-memory-recall.variant.test.ts new file mode 100644 index 00000000..4e60088f --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.variant.test.ts @@ -0,0 +1,234 @@ +import { describe, expect, it } from 'vitest'; +import { + buildBenchmarkRecallOptions, + describeBenchmarkRecallVariant, + parseBenchmarkRecallVariant, +} from './benchmark-memory-recall.variant'; + +describe('benchmark-memory-recall variants', () => { + it('parses friendly aliases', () => { + expect(parseBenchmarkRecallVariant(undefined)).toBe('default'); + expect(parseBenchmarkRecallVariant('raw')).toBe('content-only'); + expect(parseBenchmarkRecallVariant('content+entity')).toBe('content-plus-entity'); + expect(parseBenchmarkRecallVariant('parallel-content-entity')).toBe( + 'content-plus-entity-parallel' + ); + expect(parseBenchmarkRecallVariant('content+entity+fact')).toBe('content-plus-entity-fact'); + expect(parseBenchmarkRecallVariant('content+derived')).toBe('content-plus-derived'); + expect(parseBenchmarkRecallVariant('entities')).toBe('entity-only'); + expect(parseBenchmarkRecallVariant('durable-facts')).toBe('fact-only'); + expect(parseBenchmarkRecallVariant('derived')).toBe('derived-only'); + expect(parseBenchmarkRecallVariant('no-chrono')).toBe('multiview-no-chrono'); + expect(parseBenchmarkRecallVariant('unknown')).toBe('default'); + }); + + it('builds content-only hybrid options without boosts', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'hybrid', + variant: 'content-only', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-1'], + }) + ).toMatchObject({ + recallMode: 'hybrid', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-1'], + hybridChunkStrategy: 'content-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + }); + + it('builds semantic derived-only options', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'derived-only', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-2'], + }) + ).toMatchObject({ + recallMode: 'semantic', + semanticChunkTypes: ['summary', 'fact', 'topic', 'entity'], + applyChunkTypeBoosts: false, + }); + }); + + it('builds explicit entity-only options for semantic and hybrid recall', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'entity-only', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-entity'], + }) + ).toMatchObject({ + recallMode: 'semantic', + semanticChunkTypes: ['entity'], + applyChunkTypeBoosts: false, + }); + + expect( + buildBenchmarkRecallOptions({ + mode: 'hybrid', + variant: 'entity-only', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-entity'], + }) + ).toMatchObject({ + recallMode: 'hybrid', + semanticChunkTypes: ['entity'], + hybridChunkStrategy: 'derived-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + }); + + it('builds explicit content-plus-entity options for semantic and hybrid recall', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'content-plus-entity', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-content-entity'], + }) + ).toMatchObject({ + recallMode: 'semantic', + semanticChunkTypes: ['content', 'entity'], + applyChunkTypeBoosts: false, + }); + + expect( + buildBenchmarkRecallOptions({ + mode: 'hybrid', + variant: 'content-plus-entity', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-content-entity'], + }) + ).toMatchObject({ + recallMode: 'hybrid', + semanticChunkTypes: ['content', 'entity'], + hybridChunkStrategy: 'default', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + }); + + it('builds explicit content-plus-entity-parallel options for semantic and hybrid recall', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'content-plus-entity-parallel', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-content-entity-parallel'], + }) + ).toMatchObject({ + recallMode: 'semantic', + semanticChunkTypes: ['content', 'entity'], + semanticQueryStrategy: 'parallel-content-entity', + applyChunkTypeBoosts: false, + }); + + expect( + buildBenchmarkRecallOptions({ + mode: 'hybrid', + variant: 'content-plus-entity-parallel', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-content-entity-parallel'], + }) + ).toMatchObject({ + recallMode: 'hybrid', + semanticChunkTypes: ['content', 'entity'], + semanticQueryStrategy: 'parallel-content-entity', + hybridChunkStrategy: 'default', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + }); + + it('builds explicit content-plus-derived semantic options', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'content-plus-derived', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-content-derived'], + }) + ).toMatchObject({ + recallMode: 'semantic', + semanticChunkTypes: ['content', 'summary', 'fact', 'topic', 'entity', 'current_state'], + applyChunkTypeBoosts: false, + }); + }); + + it('describes the default variant explicitly', () => { + expect(describeBenchmarkRecallVariant('default')).toEqual({ + name: 'default', + semanticChunkTypes: 'default', + semanticQueryStrategy: undefined, + hybridChunkStrategy: 'default', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + }); + + it('keeps semantic defaults for multiview-no-chrono', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'multiview-no-chrono', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-3'], + }) + ).toMatchObject({ + recallMode: 'semantic', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-3'], + }); + expect( + buildBenchmarkRecallOptions({ + mode: 'semantic', + variant: 'multiview-no-chrono', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-3'], + }) + ).not.toHaveProperty('semanticChunkTypes'); + }); + + it('disables all boosts for multiview-no-boost', () => { + expect( + buildBenchmarkRecallOptions({ + mode: 'hybrid', + variant: 'multiview-no-boost', + limit: 5, + agentId: 'lumen', + topics: ['benchmark:memory-recall:case-4'], + }) + ).toMatchObject({ + recallMode: 'hybrid', + hybridChunkStrategy: 'multi-view', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }); + }); +}); diff --git a/packages/benchmarks/src/benchmark-memory-recall.variant.ts b/packages/benchmarks/src/benchmark-memory-recall.variant.ts new file mode 100644 index 00000000..0bb1dea6 --- /dev/null +++ b/packages/benchmarks/src/benchmark-memory-recall.variant.ts @@ -0,0 +1,297 @@ +import type { MemoryHybridChunkStrategy, MemorySearchOptions } from '@inklabs/api/benchmarks'; +import type { RecallMode } from './benchmark-memory-recall.types'; + +export type BenchmarkRecallVariant = + | 'default' + | 'content-only' + | 'content-plus-entity' + | 'content-plus-entity-parallel' + | 'content-plus-fact' + | 'content-plus-summary' + | 'content-plus-summary-fact' + | 'content-plus-entity-fact' + | 'content-plus-derived' + | 'entity-only' + | 'fact-only' + | 'summary-only' + | 'current-state-only' + | 'derived-only' + | 'multiview-no-boost' + | 'multiview-no-chrono'; + +const VARIANT_ALIASES: Record = { + default: 'default', + full: 'default', + 'content-only': 'content-only', + content: 'content-only', + raw: 'content-only', + 'content-plus-entity': 'content-plus-entity', + 'content+entity': 'content-plus-entity', + 'raw-plus-entity': 'content-plus-entity', + 'content-entity': 'content-plus-entity', + 'content-plus-entity-parallel': 'content-plus-entity-parallel', + 'content+entity-parallel': 'content-plus-entity-parallel', + 'parallel-content-entity': 'content-plus-entity-parallel', + 'content-plus-fact': 'content-plus-fact', + 'content+fact': 'content-plus-fact', + 'raw-plus-fact': 'content-plus-fact', + 'content-plus-summary': 'content-plus-summary', + 'content+summary': 'content-plus-summary', + 'raw-plus-summary': 'content-plus-summary', + 'content-plus-summary-fact': 'content-plus-summary-fact', + 'content+summary+fact': 'content-plus-summary-fact', + 'content-plus-entity-fact': 'content-plus-entity-fact', + 'content+entity+fact': 'content-plus-entity-fact', + 'content-plus-derived': 'content-plus-derived', + 'content+derived': 'content-plus-derived', + 'raw-plus-derived': 'content-plus-derived', + 'all-views': 'content-plus-derived', + 'entity-only': 'entity-only', + entity: 'entity-only', + entities: 'entity-only', + 'fact-only': 'fact-only', + fact: 'fact-only', + facts: 'fact-only', + 'durable-fact': 'fact-only', + 'durable-facts': 'fact-only', + 'durable-fact-only': 'fact-only', + 'durable-facts-only': 'fact-only', + 'summary-only': 'summary-only', + summary: 'summary-only', + summaries: 'summary-only', + 'current-state-only': 'current-state-only', + 'current-state': 'current-state-only', + state: 'current-state-only', + 'derived-only': 'derived-only', + derived: 'derived-only', + 'multiview-no-boost': 'multiview-no-boost', + noboost: 'multiview-no-boost', + 'multi-view-no-boost': 'multiview-no-boost', + 'multiview-no-chrono': 'multiview-no-chrono', + 'no-chrono': 'multiview-no-chrono', + 'multi-view-no-chrono': 'multiview-no-chrono', +}; + +export function parseBenchmarkRecallVariant(raw?: string): BenchmarkRecallVariant { + if (!raw) return 'default'; + const normalized = raw.trim().toLowerCase(); + const variant = VARIANT_ALIASES[normalized]; + if (!variant) { + console.warn(`[memory-benchmark] Unrecognized variant "${raw}", falling back to "default"`); + return 'default'; + } + return variant; +} + +function buildVariantSemanticOptions( + variant: BenchmarkRecallVariant +): Partial { + switch (variant) { + case 'content-only': + return { + semanticChunkTypes: ['content'], + applyChunkTypeBoosts: false, + }; + case 'entity-only': + return { + semanticChunkTypes: ['entity'], + applyChunkTypeBoosts: false, + }; + case 'content-plus-entity': + return { + semanticChunkTypes: ['content', 'entity'], + applyChunkTypeBoosts: false, + }; + case 'content-plus-entity-parallel': + return { + semanticChunkTypes: ['content', 'entity'], + semanticQueryStrategy: 'parallel-content-entity', + applyChunkTypeBoosts: false, + }; + case 'content-plus-fact': + return { + semanticChunkTypes: ['content', 'fact'], + applyChunkTypeBoosts: false, + }; + case 'content-plus-summary': + return { + semanticChunkTypes: ['content', 'summary'], + applyChunkTypeBoosts: false, + }; + case 'content-plus-summary-fact': + return { + semanticChunkTypes: ['content', 'summary', 'fact'], + applyChunkTypeBoosts: false, + }; + case 'content-plus-entity-fact': + return { + semanticChunkTypes: ['content', 'entity', 'fact'], + applyChunkTypeBoosts: false, + }; + case 'content-plus-derived': + return { + semanticChunkTypes: ['content', 'summary', 'fact', 'topic', 'entity', 'current_state'], + applyChunkTypeBoosts: false, + }; + case 'fact-only': + return { + semanticChunkTypes: ['fact'], + applyChunkTypeBoosts: false, + }; + case 'summary-only': + return { + semanticChunkTypes: ['summary'], + applyChunkTypeBoosts: false, + }; + case 'current-state-only': + return { + semanticChunkTypes: ['current_state'], + applyChunkTypeBoosts: false, + }; + case 'derived-only': + return { + semanticChunkTypes: ['summary', 'fact', 'topic', 'entity'], + applyChunkTypeBoosts: false, + }; + case 'multiview-no-boost': + return { + semanticChunkTypes: ['summary', 'fact', 'topic', 'entity', 'content'], + applyChunkTypeBoosts: false, + }; + case 'multiview-no-chrono': + return {}; + case 'default': + default: + return {}; + } +} + +function buildVariantHybridOptions( + variant: BenchmarkRecallVariant +): Pick< + MemorySearchOptions, + 'hybridChunkStrategy' | 'applyChunkTypeBoosts' | 'applyMultiViewBoost' | 'applyChronologyBoost' +> { + const base: Pick< + MemorySearchOptions, + 'hybridChunkStrategy' | 'applyChunkTypeBoosts' | 'applyMultiViewBoost' | 'applyChronologyBoost' + > = { + hybridChunkStrategy: 'default', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }; + + switch (variant) { + case 'content-only': + return { + hybridChunkStrategy: 'content-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }; + case 'content-plus-entity': + case 'content-plus-entity-parallel': + case 'content-plus-fact': + case 'content-plus-summary': + case 'content-plus-summary-fact': + case 'content-plus-entity-fact': + case 'content-plus-derived': + return { + hybridChunkStrategy: 'default', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }; + case 'entity-only': + case 'fact-only': + case 'summary-only': + case 'current-state-only': + return { + hybridChunkStrategy: 'derived-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }; + case 'derived-only': + return { + hybridChunkStrategy: 'derived-only', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }; + case 'multiview-no-boost': + return { + hybridChunkStrategy: 'multi-view', + applyChunkTypeBoosts: false, + applyMultiViewBoost: false, + applyChronologyBoost: false, + }; + case 'multiview-no-chrono': + return { + hybridChunkStrategy: 'multi-view', + applyChunkTypeBoosts: true, + applyMultiViewBoost: true, + applyChronologyBoost: false, + }; + case 'default': + default: + return base; + } +} + +export function buildBenchmarkRecallOptions(params: { + mode: RecallMode; + variant: BenchmarkRecallVariant; + limit: number; + agentId: string; + topics: string[]; +}): MemorySearchOptions { + const base: MemorySearchOptions = { + recallMode: params.mode, + limit: params.limit, + agentId: params.agentId, + includeShared: true, + topics: params.topics, + }; + + if (params.mode === 'semantic' || params.mode === 'auto') { + return { + ...base, + ...buildVariantSemanticOptions(params.variant), + }; + } + + if (params.mode === 'hybrid') { + return { + ...base, + ...buildVariantSemanticOptions(params.variant), + ...buildVariantHybridOptions(params.variant), + }; + } + + return base; +} + +export function describeBenchmarkRecallVariant(variant: BenchmarkRecallVariant): { + name: BenchmarkRecallVariant; + semanticChunkTypes: MemorySearchOptions['semanticChunkTypes'] | 'default'; + semanticQueryStrategy?: MemorySearchOptions['semanticQueryStrategy']; + hybridChunkStrategy: MemoryHybridChunkStrategy; + applyChunkTypeBoosts: boolean; + applyMultiViewBoost: boolean; + applyChronologyBoost: boolean; +} { + const semanticOptions = buildVariantSemanticOptions(variant); + const hybridOptions = buildVariantHybridOptions(variant); + + return { + name: variant, + semanticChunkTypes: semanticOptions.semanticChunkTypes || 'default', + semanticQueryStrategy: semanticOptions.semanticQueryStrategy, + hybridChunkStrategy: hybridOptions.hybridChunkStrategy || 'default', + applyChunkTypeBoosts: hybridOptions.applyChunkTypeBoosts !== false, + applyMultiViewBoost: hybridOptions.applyMultiViewBoost !== false, + applyChronologyBoost: hybridOptions.applyChronologyBoost !== false, + }; +} diff --git a/packages/benchmarks/vitest.config.ts b/packages/benchmarks/vitest.config.ts new file mode 100644 index 00000000..875155c4 --- /dev/null +++ b/packages/benchmarks/vitest.config.ts @@ -0,0 +1,8 @@ +import { defineConfig } from 'vitest/config'; + +export default defineConfig({ + test: { + include: ['src/**/*.test.ts'], + exclude: ['node_modules', 'dist', '**/*.integration.test.ts'], + }, +}); diff --git a/supabase/migrations/20260409215951_memory_chunk_type_filters_and_indexes.sql b/supabase/migrations/20260409215951_memory_chunk_type_filters_and_indexes.sql new file mode 100644 index 00000000..cd0626dc --- /dev/null +++ b/supabase/migrations/20260409215951_memory_chunk_type_filters_and_indexes.sql @@ -0,0 +1,137 @@ +-- Phase 2 multi-view retrieval support: +-- - add chunk-type filter indexes +-- - expose matched_chunk_type from match_memory_embedding_chunks RPC +-- - allow filtered retrieval by chunk view (summary/fact/topic/entity/content) + +CREATE INDEX IF NOT EXISTS idx_memory_embedding_chunks_user_chunk_type + ON public.memory_embedding_chunks(user_id, chunk_type); + +CREATE INDEX IF NOT EXISTS idx_memory_embedding_chunks_memory_chunk_type + ON public.memory_embedding_chunks(memory_id, chunk_type); + +DROP FUNCTION IF EXISTS public.match_memory_embedding_chunks( + vector, + double precision, + integer, + uuid, + text, + text, + text[], + text, + boolean, + boolean +); + +CREATE OR REPLACE FUNCTION public.match_memory_embedding_chunks( + query_embedding vector, + match_threshold double precision DEFAULT 0.2, + match_count integer DEFAULT 20, + p_user_id uuid DEFAULT NULL, + p_source text DEFAULT NULL, + p_salience text DEFAULT NULL, + p_topics text[] DEFAULT NULL, + p_agent_id text DEFAULT NULL, + p_include_shared boolean DEFAULT true, + p_include_expired boolean DEFAULT false, + p_chunk_types text[] DEFAULT NULL +) +RETURNS TABLE ( + id uuid, + user_id uuid, + content text, + summary text, + topic_key text, + source text, + salience text, + topics text[], + embedding vector, + metadata jsonb, + version integer, + created_at timestamptz, + expires_at timestamptz, + agent_id text, + identity_id uuid, + matched_chunk_text text, + matched_chunk_index integer, + matched_chunk_type text, + similarity double precision +) +LANGUAGE sql +STABLE +AS $$ + WITH ranked_matches AS ( + SELECT + m.id, + m.user_id, + m.content, + m.summary, + m.topic_key, + m.source, + m.salience, + m.topics, + m.embedding, + m.metadata, + m.version, + m.created_at, + m.expires_at, + m.agent_id, + m.identity_id, + c.chunk_text AS matched_chunk_text, + c.chunk_index AS matched_chunk_index, + c.chunk_type AS matched_chunk_type, + 1 - (c.embedding <=> query_embedding) AS similarity, + row_number() OVER ( + PARTITION BY m.id + ORDER BY c.embedding <=> query_embedding ASC, c.chunk_index ASC + ) AS rank_within_memory + FROM public.memory_embedding_chunks c + JOIN public.memories m ON m.id = c.memory_id + WHERE + (p_user_id IS NULL OR m.user_id = p_user_id) + AND (p_source IS NULL OR m.source = p_source) + AND (p_salience IS NULL OR m.salience = p_salience) + AND (p_topics IS NULL OR m.topics && p_topics) + AND (p_chunk_types IS NULL OR c.chunk_type = ANY(p_chunk_types)) + AND ( + p_agent_id IS NULL + OR ( + p_include_shared + AND (m.agent_id = p_agent_id OR m.agent_id IS NULL) + ) + OR ( + NOT p_include_shared + AND m.agent_id = p_agent_id + ) + ) + AND ( + p_include_expired + OR m.expires_at IS NULL + OR m.expires_at > now() + ) + AND 1 - (c.embedding <=> query_embedding) > match_threshold + ) + SELECT + id, + user_id, + content, + summary, + topic_key, + source, + salience, + topics, + embedding, + metadata, + version, + created_at, + expires_at, + agent_id, + identity_id, + matched_chunk_text, + matched_chunk_index, + matched_chunk_type, + similarity + FROM ranked_matches + WHERE rank_within_memory = 1 + ORDER BY similarity DESC, created_at DESC + LIMIT match_count; +$$; diff --git a/yarn.lock b/yarn.lock index 8eaf0c47..3b8c7674 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1833,7 +1833,7 @@ __metadata: languageName: node linkType: hard -"@inklabs/api@workspace:packages/api": +"@inklabs/api@workspace:*, @inklabs/api@workspace:packages/api": version: 0.0.0-use.local resolution: "@inklabs/api@workspace:packages/api" dependencies: @@ -1886,6 +1886,16 @@ __metadata: languageName: unknown linkType: soft +"@inklabs/benchmarks@workspace:packages/benchmarks": + version: 0.0.0-use.local + resolution: "@inklabs/benchmarks@workspace:packages/benchmarks" + dependencies: + "@inklabs/api": "workspace:*" + tsx: "npm:^4.20.6" + vitest: "npm:^4.0.18" + languageName: unknown + linkType: soft + "@inklabs/cli@workspace:packages/cli": version: 0.0.0-use.local resolution: "@inklabs/cli@workspace:packages/cli" @@ -14756,7 +14766,7 @@ __metadata: languageName: node linkType: hard -"tsx@npm:^4.19.0, tsx@npm:^4.7.0": +"tsx@npm:^4.19.0, tsx@npm:^4.20.6, tsx@npm:^4.7.0": version: 4.21.0 resolution: "tsx@npm:4.21.0" dependencies: