feat: SB-initiated context eviction — list_context + evict_context (by Wren) - #242
Merged
Conversation
…cal tools (by Wren) Give SBs agency over their own context window. Two new client-local tools that run in the CLI without a PCP server round-trip: - list_context: Introspect the context ledger — see all entries with IDs, token counts, sources, previews, and a per-source breakdown. Lets the SB decide what's relevant and what to drop. - evict_context: Surgically remove entries by ID, source, or role. Unlike /eject (positional) or /trim (oldest-first), this enables targeted removal of irrelevant memories, stale inbox messages, or old tool results while preserving everything else. Ledger additions: evictEntries(ids), evictBySource(source), evictByRole(role), summarizeEntries() for introspection. Tests for all new methods (13/13 pass). Co-Authored-By: Wren <noreply@anthropic.com>
…by Wren) Addresses Lumen's blocker on PR #242: list_context and evict_context results were being added to the ledger as local-tool entries, which pollutes the context being inspected and reintroduces evicted content. Context-management tools now pass their results to the continuation turn (so the backend can reason about them) but skip ledger persistence. Co-Authored-By: Wren <noreply@anthropic.com>
…on tests (by Wren) Covers: list_context (metadata, previews, bookmarks, empty state, truncation), evict_context (by ID, source, role, error cases, non-existent IDs, preview cap), evict→list consistency, sequential evictions, transcript exclusion after eviction. 32/32 tests pass across context-ledger + context-tools. Co-Authored-By: Wren <noreply@anthropic.com>
…l + budget monitor tests (by Wren) Hook registry for the sb chat REPL with: - Event-driven lifecycle hooks (on-turn-end, on-prompt, pre-tool, etc.) - Priority ordering, immediate injection/eviction, pre-* blocking - Error resilience (hooks never crash the REPL) - Fire log for debugging 22 tests covering: registration, priority ordering, event filtering, context injection (single + multi + cross-hook visibility), eviction via hooks, blocking semantics (pre-* only), error resilience, passive recall simulation (topic injection, cooldown, budget ceiling, dedup, re-injection after eviction), budget monitor simulation. 54/54 tests pass across the full context management suite. Co-Authored-By: Wren <noreply@anthropic.com>
… PCP server (by Wren) Integration tests hitting the live PCP server with real semantic search: - Topic signal extraction (keyword extraction, code/URL stripping) - Live recall: session routing query, context management query, nonsense query handling, limit parameter - End-to-end hook: topic → recall → inject → dedup across turns, budget ceiling suppression, full inject→evict→topic-shift→re-inject cycle - Latency benchmark: avg ~600ms (Ollama embedding pipeline on localhost) Uses vitest.integration.config.ts to bypass the *.integration.test.ts exclude. Tests skip gracefully when PCP server is unreachable. Co-Authored-By: Wren <noreply@anthropic.com>
…Wren) Team consensus (Myra +1, Conor +1): snake_case with noun first, modifier second. Drops on- prefix. Clean, greppable, valid TS identifiers. Renamed: session_start, session_end, prompt_build, turn_start, turn_end, tool_pre, tool_post, compact_pre, compact_post, evict_post, budget_warning, idle 66/66 tests pass (54 unit + 12 integration). Co-Authored-By: Wren <noreply@anthropic.com>
…% topic drift (by Wren) Curated benchmark testing whether passive recall surfaces the RIGHT memories: - 6 scenario relevance test: 96% avg keyword hit rate, 0% noise - Topic drift test: 4% Jaccard similarity across distinct topics (routing/auth/heartbeat) - Multi-turn session simulation: 5 turns across 3 topic phases, 8 unique memories injected, 55% avg relevance No secrets — uses generic development scenarios with keyword scoring. Co-Authored-By: Wren <noreply@anthropic.com>
…Wren)
Hooks are now live in sb chat:
- SbHookRegistry initialized alongside the context ledger
- Built-in hooks registered: passive-recall (turn_end) + budget-monitor (prompt_build)
- callRecall wraps pcp.callTool('recall') into the shape hooks expect
- turn_end fires after assistant response with lastTurn context + budget utilization
- prompt_build fires before prompt assembly for budget warnings
- Both fire-and-forget (never block the REPL)
The passive recall hook extracts topic signal from the last turn,
calls recall() with hybrid mode, deduplicates, respects budget ceiling
and cooldown, and injects relevant memories as evictable entries.
Ready for testing with: sb chat --backend claude --agent myra
Co-Authored-By: Wren <noreply@anthropic.com>
PcpToolCallResult is Record<string, unknown>, so accessing .content[0].text needs type assertions. Added parseResult() helper that casts through the content array shape. Used 'any' return type for test assertions since we're testing runtime behavior, not types. Co-Authored-By: Wren <noreply@anthropic.com>
…ion, full cycle (by Wren) 11 e2e tests against live PCP server validating the full sb chat pipeline: - Bootstrap verification: identity + constitution loads correctly - Hook lifecycle: turn_end fires passive recall, prompt_build fires budget monitor, priority ordering - Context tools: list_context accuracy, evict_context selective removal, routing - Full session cycle: bootstrap → turn → recall inject → evict → topic shift → re-inject (auth-related=true) - Topic signal extraction: distinct signals for distinct topics, code/URL filtering Total test count across PR: 80 (54 unit + 15 integration + 11 e2e) Co-Authored-By: Wren <noreply@anthropic.com>
…dler map (by Wren) The tool was returning getAgentGateway().getRegisteredAgents() which only tracks agents with active runtime trigger listeners — empty after restarts or when agents use polling. Now queries agent_identities table (the source of truth) and enriches each agent with hasRuntimeHandler from the gateway. Co-Authored-By: Wren <noreply@anthropic.com>
…(by Wren) budgetUtilization was calculated as ledger.totalTokens() / maxContextTokens, but maxContextTokens includes bootstrap budget. The ledger only holds transcript tokens, so utilization was artificially low — a 77% full ledger would report as 62% because bootstrap wasn't subtracted. Fix: effectiveBudget = maxContextTokens - bootstrapTokens, then utilization = ledger.totalTokens() / effectiveBudget. Applied to both prompt_build and turn_end hook fire sites. This ensures passive recall's 80% ceiling and budget-monitor warnings fire at the right time relative to actual available space. Co-Authored-By: Wren <noreply@anthropic.com>
…forget, inbox + recall (by Wren)
13 tests validating the real runtime wiring:
1. PcpClient.callTool('recall') shape: returns {success, memories} directly
(PcpClient parses JSON-RPC content[0].text internally). callRecall wrapper
feeds correctly into passive recall hook. CONFIRMED via live PCP server.
2. turn_end fire-and-forget: injections appear in ledger after current response
(visible in NEXT prompt, not current). Sequential hooks see earlier injections.
Errors swallowed silently — never propagate.
3. Inbox + passive recall: inbox messages influence topic signal extraction.
Recall fires on inbox-informed turns. Evicting inbox doesn't break recall.
Real PCP recall after inbox-triggered conversation returns relevant memories.
4. Budget with bootstrap: effective budget correctly excludes bootstrap tokens.
Budget monitor fires at right threshold. Passive recall suppressed above 80%.
Total test count: 93 (54 unit + 39 integration/e2e/benchmark)
Co-Authored-By: Wren <noreply@anthropic.com>
…ta (by Wren) Semi-automated integration test simulating a complete Myra heartbeat session: - Phase 1: Bootstrap as Myra (1934 tokens knowledge, 10 sessions) - Phase 2: Inbox check via PcpClient - Phase 3: Passive recall surfaces real Myra memories (3 injected) - Phase 4: Context management cycle — fill, evict tools + recall, verify clean - Phase 5: Optional Telegram send (SEND_TELEGRAM=true) - Phase 6: Full heartbeat: bootstrap → inbox → recall → process → evict → done 6 tests pass, 1 skipped (Telegram gate). Total across PR: 100 tests. Co-Authored-By: Wren <noreply@anthropic.com>
…onse (by Wren) send_response requires conversationId (the Telegram chat ID). Added the resolved ID and documented the gap: we need a user→platform→conversationId mapping so agents can initiate outbound messages without hardcoding. PCP task created for the mapping table. Co-Authored-By: Wren <noreply@anthropic.com>
…s (by Wren) Enables running N conversational turns in non-interactive mode: sb chat --message 'Heartbeat check' --max-turns 3 --agent myra After the initial message, subsequent turns use a continuation prompt that lets the SB decide whether to keep working or stop. If the SB says 'Heartbeat complete' it exits early. This enables automated multi-turn agent sessions for heartbeats, integration tests, and CI pipelines. Co-Authored-By: Wren <noreply@anthropic.com>
… (by Wren) Two fixes for session observability and continuity: 1. Session ID now printed in the startup banner so users can see it and attach from another terminal. 2. --max-turns no longer calls end_session. Instead it updates the phase to 'idle:awaiting-input' and prints the attach command. The session stays resumable — the user or another SB can follow up on what the agent did. Co-Authored-By: Wren <noreply@anthropic.com>
…y Wren)
Replace hardcoded 'heartbeat complete' text matching with structured
signal_status tool. The SB emits:
pcp-tool {"tool":"signal_status","args":{"status":"completed"}}
pcp-tool {"tool":"signal_status","args":{"status":"blocked","reason":"Need approval"}}
pcp-tool {"tool":"signal_status","args":{"status":"continuing"}}
The runtime reads the signal and:
- completed → pause session as idle:completed
- blocked → pause as blocked:needs-input, print reason
- continuing → give another turn
- no signal → treat as continuing (give next turn if under max-turns)
This is universal across heartbeats and coding tasks. The SB decides
its own state; the runtime responds to the signal.
Co-Authored-By: Wren <noreply@anthropic.com>
…text never denied (by Wren) Client-local tools (context management + signaling) were being run through the tool policy engine, which denied them in non-interactive sessions. These tools run in-process and never touch the PCP server — they should always be allowed. Fix: check isClientLocalTool() before the policy decision in executeOneToolCall(). Client-local tools go straight to execution. Discovered when Myra's signal_status calls were denied during a --max-turns heartbeat session, preventing the session from signaling completion. Co-Authored-By: Wren <noreply@anthropic.com>
…y Wren) The JSONL transcript is already the immutable record — every entry added and every eviction event is logged there. The in-memory ledger should stay lean: hard-delete frees RAM, the JSONL retains history. Soft-delete was overengineering — it would cause memory leaks in long sessions and duplicates the JSONL's job. Client-local tools (list_context, evict_context, signal_status) bypass policy because they operate on in-memory working state only. The SB must have full control over its own context window. Co-Authored-By: Wren <noreply@anthropic.com>
…ookup (by Wren) The SB emits tool names with the MCP namespace prefix (mcp__pcp__get_inbox) because that's what it sees in bootstrap. But PcpClient expects bare names (get_inbox) and the tool policy doesn't recognize the prefixed form. Fix: strip the prefix in both callTool (before PcpClient) and executeOneToolCall (before policy check). This allows --profile full to work correctly with MCP-prefixed tool names. Co-Authored-By: Wren <noreply@anthropic.com>
…, and signals (by Wren)
The user now sees what's happening with context management:
💡 memory surfaced: "Session routing gotcha: backendSessionId..." (47 tok)
🗑 evicted 3 entries (125 tok freed, 450 tok remaining)
📋 context: 12 entries, ~890 tok
bootstrap(2/300t) pcp-tool(3/125t) passive-recall(2/97t) (none)(5/368t)
✅ signal: completed — Heartbeat check done, inbox clear
🚫 signal: blocked — Need approval on auth approach
⚠ Context at 82% — 3,200 / 3,900 tok (bootstrap: 2,100 reserved)
All notifications are dim/subtle — visible but not interrupting.
Memory previews show first 120 chars; full content is in the ledger.
Co-Authored-By: Wren <noreply@anthropic.com>
…esponse) (by Wren) Memories are now surfaced BEFORE the backend responds, not just after. When you type a question, passive recall extracts topic signal from your input and injects relevant memories into the prompt. The backend sees them and can use them in its response. Two recall hooks share state (dedup set, cooldown, budget ceiling): - prompt_build: recall based on user input → memories in THIS turn - turn_end: recall based on response → memories in NEXT turn User sees 💡 notifications for both, before and after the response. Co-Authored-By: Wren <noreply@anthropic.com>
…/native MCP for PCP tools (by Wren) Claude Code as a backend has its own native tools (ToolSearch, Read, etc.) that remain available even in --tool-routing local mode. The SB was using ToolSearch to discover PCP tools and then trying native MCP calls, which don't go through our pcp-tool extraction pipeline. Fix: instruction now explicitly says 'Do NOT use ToolSearch, mcp__pcp__*, or native MCP tool calling for PCP tools — those will not work in this runtime. Only the fenced block format will execute PCP tools.' Co-Authored-By: Wren <noreply@anthropic.com>
Non-interactive mode (--message) sets approvalMode to 'auto-deny', which denies all tools that need prompting. But --profile full means the user explicitly chose to trust all tools — so it should auto-approve, not auto-deny. This was the root cause of Myra's get_inbox/list_tasks denials in automated heartbeat sessions with --profile full. Co-Authored-By: Wren <noreply@anthropic.com>
The non-interactive detection checked options.nonInteractive but not options.message. Using --message without --non-interactive fell through to 'interactive' approval mode, which can't prompt in background execution — silently denying all tools. Fix: check options.message alongside options.nonInteractive for approval mode selection. Co-Authored-By: Wren <noreply@anthropic.com>
…pping in post-approval (by Wren) Two bugs from Lumen's PR #242 review: 1. --max-turns called update_session_phase without sessionId, which could update the wrong active session. Now passes runtime.sessionId explicitly. 2. Post-approval policy re-check used the raw tool name (with mcp__pcp__ prefix) instead of the stripped name. A promptable prefixed tool would get blocked after user approval. Now uses policyToolName consistently. Blocker #2 (re-injection after eviction) deferred to next PR as a feature enhancement — tracked as PCP task. Co-Authored-By: Wren <noreply@anthropic.com>
…om main Co-Authored-By: Wren <noreply@anthropic.com>
7 tasks
conoremclaughlin
added a commit
that referenced
this pull request
May 8, 2026
## Summary Builds on PR #242 (context eviction + passive recall). Adds a real-scenario eval harness for measuring whether PCP recall surfaces the **right** memories during real vocational work — not synthetic keyword benchmarks. Spec: `ink://specs/memory-real-scenario-eval` (artifact). **Five capabilities defined, two implemented in v1:** | Capability | Status | What it proves | | ---------------- | --------------- | ------------------------------------------------------------------- | | **recall** | ✅ v1 | The right memories surface when working on a topic | | **correction** | ✅ v1 | Memory contradicts a stale premise rather than complying | | **eviction** | schema only | Irrelevant memories drop out when conversation shifts | | **re-hydration** | schema only | Previously-evicted memories return on re-entry | | **continuity** | schema only | Post-compaction, the SB knows what was being worked on and why | Eviction / re-hydration / continuity shapes validate in the loader but return a clear "not yet implemented" result from the runner — they need simulators in v2. **Scenario anatomy:** `context` + `impliedQuestion` + `expectedSurfaced` + `mustAssert` + `mustNotAssert` + `rubric`. `mustAssert` decouples "item X returned" from "fact Y derivable" — a claim can be derivable even when its canonical source isn't in context (e.g., "Conor replied to Dalton on Feb 3" derivable from a memory, without the full email chain in the prompt). **v1 scoring** is deterministic substring/phrase matching. The `claimDerivable(claim, surfaced)` interface is stable so v2 can swap an LLM judge without changing scenario files. ## What's shipped - **Schema** — `types.ts` (11 shapes, 5 capabilities, rubric, pre-state for continuity scenarios) - **Loader** — `loader.ts` YAML parsing + shape-specific validation (e.g., `stalePremise` required for `current-state-correction`) - **Scorer** — `scorer.ts` pure `scoreScenario(scenario, signal, surfaced)`. Failed high-criticality must-assert = hard fail. - **Runner** — `runner.ts` orchestration, takes `RecallFn` from caller (stub in unit tests, HTTP in integration tests). Builds topic signal from `stalePremise + context → userInput`, `impliedQuestion → assistantResponse`, mirroring passive recall at `turn_end`. - **Report** — `report.ts` markdown with per-scenario verdicts, surfaced items, must-assert passes. - **Fixtures** — 3 seed scenarios: `merge-strategy-rule`, `push-to-main-rule`, `restart-dev-server`. - **Tests** — 22 passing: loader (6), scorer (7), runner stubs (4), + live integration (1). ## First live-server run against current memory set ``` merge-strategy-rule: 0% precision, 0% recall — no canonical "NEVER squash" memory exists push-to-main-rule: 10% precision, 50% recall, must-assert 100% — found memory:never-push-to-main (the 90% "noise" is real signal: passive recall needs filtering) restart-dev-server: 0% precision, must-assert 67% — missing "never kill" phrase in surfaced set ``` These aren't test failures — they're the point. The integration test is a **reporting** test: it passes when the harness runs end-to-end and every supported scenario surfaces at least one memory. Rubric misses surface as findings we can act on (seed the memory, fix phrasing, tighten passive recall). ## Test plan - [x] Unit tests: `yarn workspace @inklabs/cli exec vitest run src/repl/real-scenarios/` — 22/22 passing - [x] Live integration against `http://localhost:3001` — harness works, report prints, pass rate surfaces correctly - [ ] Review by Lumen — especially the `claimDerivable` abstraction and the shape / capability matrix - [ ] Seed the "NEVER squash merge" memory so `merge-strategy-rule` rubric can pass - [ ] Add Dalton / person-centric fixture once Gmail API path is scoped - [ ] v2: compaction simulator for `post-compaction-continuity` - [ ] v2: eviction / re-hydration runners ## Non-goals for this PR - LLM judge (v2 — interface is ready) - Eviction / re-hydration / continuity runners (v2 — schemas defined) - Broader than 3 seed fixtures (easy to add once scaffold is blessed) — Wren
conoremclaughlin
added a commit
that referenced
this pull request
Jun 11, 2026
…ty (#403) ## Why Implements Layer 2 of `spec:sb-context-eviction` (v2 — consolidated tonight with Conor). PR #242 gave SBs `list_context`/`evict_context`, but evictions were in-memory only: hydration replays raw transcript events, so **evicted entries resurrected on reattach** — the SB's choices didn't stick. This is the persistence foundation everything else (interactive UI, `/evict`, retention policies) builds on. Storage decision (analyzed in the spec): **event-sourced exclusion in the JSONL** — append-only, local-first, same proven machinery as compaction events. Cloud mirror (Supabase projection) explicitly deferred. ## What - **`eid` identity**: every `appendTranscript` event gets a monotonic file-relative id; hydration seeds the counter from the file's max - **`context_evict` events**: `evict_context` results include `evictRefs` (eid + content hash per removed entry); the runtime persists them with actor/reason - **In-stream replay**: hydration applies evictions at their position in the file — an eviction only affects entries *before* it, so identical content appended later survives (ordering semantics for free, regression-tested) - **Ref matching**: eid when present (precise); content-hash fallback for legacy/live entries (identical duplicates evict together — documented semantics) - **Trims persist too**: `/trim` and the compaction hard-trim fallback write `context_evict` (actor: system) — closes the pre-existing "trims resurrect" gap - **Evicted ≠ erased**: skipped entries collect into a side list; Ctrl+O inspector gains an "Evicted from Context" section with attribution - **Safety**: evict events never touch entries that predate hydration (bootstrap); compaction `keptEntries` carry eids so kept-tail entries remain individually evictable ## Verification - 15 hydration tests (6 new eviction scenarios), 52/52 across hydration+ledger+tools, full CLI suite 788 passed (1 known gemini WIP failure) - **Live E2E**: Myra ran `list_context` → `evict_context source:old-heartbeat` (5 stale entries, 430 tok freed) → **reattach**: "Old-heartbeat entries: None present… they have not returned. Launch codename: BLUEBIRD — still intact." 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SB-initiated context management + passive recall + runtime hooks for the sb chat REPL. 26 commits, 100+ tests.
Context Eviction
list_context/evict_context/signal_status— client-local tools that bypass policy (SB owns its context window)evictEntries,evictBySource,evictByRole,summarizeEntriesHook Infrastructure
SbHookRegistrywith snake_case noun-first events:session_start,turn_end,prompt_build,tool_pre,compact_pre, etc.Passive Recall
prompt_build(user input → memories in THIS turn) +turn_end(response → memories in NEXT turn)Session Management
signal_statustool: SBs signalcompleted/blocked/continuing--max-turns Nfor multi-turn non-interactive sessions (heartbeats, tasks)idle:awaiting-input/blocked:needs-input/idle:completed)--profile full+--message= auto-approve all toolsRuntime Fixes
mcp__pcp__prefix stripped for PcpClient + policy lookupTested with Myra
Specs
pcp://specs/sb-context-eviction(v1)pcp://specs/sb-runtime-hooks(v3)Test plan
🤖 Generated with Claude Code
— Wren