feat(sessions): structured compaction summary with monotonic boundary

openspec/changes/compaction-rework/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-04-11

openspec/changes/compaction-rework/proposal.md

@@ -0,0 +1,149 @@
+# Change Proposal: compaction-rework
+
+## Why
+
+Netclaw's compaction pipeline exhibited post-compaction grounding failures in a
+live Slack session on 2026-04-11 — ArdyBot lost specificity around a `Rect`
+struct it had been inspecting, and failed to recall its own session ID because
+an earlier turn had referenced a foreign session ID. Root-cause analysis showed
+the observer phase strips the grounding signal it most needs (tool-call
+arguments, self-session-id disambiguation), produces a free-form bullet list
+that decays exponentially across successive compactions, and relies on an
+extractive reducer that can create orphan tool-result messages the provider
+will reject.
+
+Rather than another incremental prompt patch, this change is a structural
+rework informed by reading the actual source of four real LLM harnesses:
+**OpenCode** (SST, `2719063`), **Aider** (`f09d706`), **Cline** (`a0faf7c`),
+and **Claude Code** (Anthropic — docs at `code.claude.com/docs/en/*`). The
+research surfaced three ideas worth adopting: Cline's 9-section structured
+summary with explicit anti-drift rules, Cline's monotonic compaction boundary
+that prevents summary-over-summary decay, and OpenCode's
+truncate-only-at-user-message-boundaries rule that cleanly enforces tool
+pair integrity. The current netclaw-session spec already has a "Tool call/result
+pair integrity" scenario that the code does not honor — this change fixes that
+gap as well.
+
+Research write-up with file/line citations is in the plan file at
+`/home/petabridge/.claude/plans/proud-honking-goose.md`.
+
+## What Changes
+
+- **BREAKING**: `CompactionParameters` record gains a `SessionId` field. All
+  callers of `SessionCompactionPipeline.ExecuteAsync` must pass it.
+- **BREAKING**: `ObservationPromptBuilder.BuildObservationSystemPrompt()`,
+  `BuildObservationUserPrompt()`, and `WrapObservations()` gain `SessionId`
+  parameters and a new output contract (structured sections, not free-form
+  bullets).
+- **BREAKING**: The compacted session history format changes. Prior
+  observation messages stored as `User`-role messages with `[observations
+  from earlier in this session]` prefix now use a distinctive
+  `[session-summary session:{id}]` header. The header is the recognition
+  marker used by the observer on successive compactions and by the reducer
+  when walking backward to a user-message boundary. Journals written
+  before this change still replay via the existing `Apply(SessionCompacted)`
+  path — the old format stays readable — but new compactions produce the
+  new format.
+- **New**: Observer LLM system prompt rewritten to the 9-section structured
+  format, borrowed from Cline's `contextManagement.ts:10-110`:
+  Primary Request / Technical Concepts / Files+Code / Problem Solving /
+  Pending Tasks / Task Evolution (with direct user quotes to prevent drift) /
+  Current Work / Next Step / Required Files. Adapted to Netclaw vocabulary.
+- **New**: Observer receives the self `SessionId` in its system prompt so it
+  can disambiguate the running session from any foreign session IDs
+  referenced in the discarded window.
+- **New**: Observer is instructed: "If the input already contains a
+  `[session-summary]` block from a prior compaction, preserve its sections
+  verbatim and append/update — do not rewrite." This is the structural
+  second-compaction defense.
+- **Modified**: `ExtractiveSessionReducer` truncates only at user-message
+  boundaries. When the naive cutoff would land on a `Tool`-role message or
+  an `Assistant` message with `FunctionCallContent`, walk backward to the
+  nearest user-message boundary. Subsumes the latent pair-integrity bug in
+  the current reducer which slices by count and produces orphan tool
+  results.
+- **Modified**: `netclaw-session` "Conversation compaction" requirement —
+  updated to reflect the new structured summary format, the distinctive
+  `[session-summary session:{id}]` header, and user-boundary truncation.
+  The "Tool call/result pair integrity" scenario becomes enforced (was
+  aspirational).
+
+## Capabilities
+
+### New Capabilities
+
+_(none — this change modifies an existing capability)_
+
+### Modified Capabilities
+
+- `netclaw-session`: the "Conversation compaction" requirement is rewritten
+  to reflect structured 9-section summary output, the
+  `[session-summary session:{id}]` header that makes summaries
+  recognizable across successive compactions, and tool call/result pair
+  integrity via user-boundary truncation. The existing "Tool call/result
+  pair integrity during compaction" scenario is strengthened from
+  aspirational to strictly enforced.
+
+## Impact
+
+### Affected code
+
+- `src/Netclaw.Actors/Sessions/ObservationPromptBuilder.cs` — prompt rewrite,
+  new signatures, structured section output contract
+- `src/Netclaw.Actors/Sessions/Pipelines/SessionCompactionPipeline.cs` —
+  `CompactionParameters.SessionId`, emit summary as System-role with
+  boundary marker
+- `src/Netclaw.Actors/Sessions/SessionState.cs` — update
+  `Apply(SessionCompacted)` to preserve `WorkingContext` and rebuild
+  history with the structured summary at the head
+- `src/Netclaw.Actors/Sessions/ExtractiveSessionReducer.cs` — walk backward
+  to user-message boundary instead of slicing by count
+- `src/Netclaw.Actors/Sessions/LlmSessionActor.cs` — `CompactionParameters`
+  construction includes `_sessionId`
+
+### Affected tests
+
+- `src/Netclaw.Actors.Tests/Sessions/ObservationPromptBuilderTests.cs` —
+  assertions for structured sections, self-session-id embedding,
+  preserve-prior-summary rule
+- `src/Netclaw.Actors.Tests/Sessions/ExtractiveSessionReducerTests.cs` —
+  user-boundary truncation, orphan prevention for tool results and
+  assistant tool_calls
+- `src/Netclaw.Actors.Tests/Sessions/CompactionIntegrationTests.cs` —
+  monotonic boundary, second-compaction no-decay test, session-id
+  disambiguation test
+
+### Affected APIs / journals
+
+- **Journal compatibility**: existing `SessionCompacted` events continue to
+  deserialize and replay. New compactions write the extended event with the
+  boundary index. No migration required.
+- **IPC**: no public API change visible outside the actor package.
+- **Memory queue**: unchanged — compaction still emits its high-priority
+  memory checkpoint via `EnqueueCheckpointFireAndForget` per the existing
+  "Compaction boundary emits memory checkpoint" scenario.
+
+### Security & operational impact
+
+- **Security**: none. No new trust surfaces, no new grant categories, no
+  new tool capabilities. The observer LLM call continues to use the
+  existing `_compactionClient` with the same audience context.
+- **Operational**: slight increase in observer prompt size from the
+  structured template. Offset by the existing `KeepRecentToolResults` +
+  Phase 1 tool-result clearing which bounds the discarded window size. No
+  change to the compaction trigger threshold.
+- **Observability**: `SessionCompacted` event format extends — existing
+  eval replay tooling continues to work on old journals; new journals
+  carry additional metadata useful for regression analysis.
+
+### Dependencies / out of scope
+
+- Durable `WorkingContext` task state (`RecentFiles`, `OpenGoals`,
+  `ProgressMarkers`) is a separate OpenSpec change
+  (`working-context-grounding`) that stacks on this one.
+- Session CWD tracking and project identity file re-reading are tracked as
+  GitHub issues **#595** and **#596** against milestone 0.12, not this
+  change.
+- Aider's "files-as-source-of-truth" philosophy (stop persisting file
+  contents in history, re-read on demand) is called out as the deepest
+  architectural direction but is not in scope.

openspec/changes/compaction-rework/specs/netclaw-session/spec.md

@@ -0,0 +1,111 @@
+# netclaw-session Delta Spec — compaction-rework
+
+## MODIFIED Requirements
+
+### Requirement: Conversation compaction
+
+The system SHALL compact long session history using a tiered approach that
+produces a structured summary surviving successive compactions without
+grounding decay, enforces tool call/result pair integrity at the compaction
+boundary, and disambiguates the self session from any foreign session
+identifiers referenced in the discarded window. Before and after compaction
+boundaries, the session SHALL emit high-priority memory checkpoints into the
+durable memory queue instead of performing a synchronous one-off memory flush
+that depends on the turn path completing all curation work inline.
+
+The compaction observer LLM SHALL produce output in a fixed structured
+format with nine sections: Primary Request and Intent, Key Technical
+Concepts, Files and Code Sections, Problem Solving, Pending Tasks, Task
+Evolution, Current Work, Next Step, and Required Files. The Task Evolution
+section SHALL contain direct quotes from user messages that changed the
+task, to prevent drift across successive compactions.
+
+The compaction summary message SHALL be wrapped with a distinctive header
+of the form `[session-summary session:{id}]` so that consumers (the
+observer on successive compactions, the reducer, and the UI) can
+recognize it as a prior-compaction artifact and preserve it across
+successive compactions without relying on a separately-persisted index.
+
+The compaction observer SHALL receive the self `SessionId` in its system
+prompt and SHALL explicitly mark any foreign session identifiers in
+observations as `session:{id}` rather than conflating them with the self
+session.
+
+The compaction observer system prompt SHALL include a rule instructing the
+model to preserve any prior structured summary block verbatim and update
+in place, rather than re-summarizing or rewriting it.
+
+#### Scenario: Compaction threshold reached
+
+- **GIVEN** `UsageDetails.InputTokenCount` exceeds `SessionConfig.CompactionTokenLimit`
+- **WHEN** compaction runs
+- **THEN** the actor enters `Compacting` behavior state
+- **AND** incoming messages are buffered during compaction
+
+#### Scenario: Compaction boundary emits memory checkpoint
+
+- **GIVEN** compaction is about to run or has just completed a summary reduction
+- **WHEN** the compaction boundary is reached
+- **THEN** the session enqueues a high-priority memory checkpoint for durable
+  curation
+- **AND** the user-facing session does not wait for background curation to
+  finish
+
+#### Scenario: Tiered compaction — tool result clearing first
+
+- **GIVEN** compaction is triggered
+- **WHEN** phase 1 runs
+- **THEN** old tool results are replaced with placeholders
+- **AND** the N most recent tool interactions are preserved in full
+- **AND** if threshold is now satisfied, no summarization LLM call is made
+
+#### Scenario: Tiered compaction — structured summarization
+
+- **GIVEN** phase 1 (tool clearing) did not bring context under threshold
+- **WHEN** the observer LLM call runs
+- **THEN** the observer produces a summary containing the nine fixed sections
+  (Primary Request and Intent, Key Technical Concepts, Files and Code Sections,
+  Problem Solving, Pending Tasks, Task Evolution, Current Work, Next Step,
+  Required Files)
+- **AND** the Task Evolution section contains direct quotes from user
+  messages that changed the task
+- **AND** the summary is wrapped with a `[session-summary session:{id}]`
+  header and stored in the compacted history
+- **AND** a `SessionCompacted` event is persisted carrying the compacted
+  messages
+- **AND** a persistence snapshot is taken
+- **AND** compacted state remains usable for future turns
+
+#### Scenario: Successive compactions do not re-summarize prior summary
+
+- **GIVEN** a session that has been compacted, with a prior
+  `[session-summary session:{id}]` message in history
+- **WHEN** a subsequent compaction is triggered
+- **THEN** the observer system prompt instructs the model to preserve the
+  prior summary block verbatim and update its sections in place
+- **AND** the reducer's user-message-boundary walk-back preserves the
+  prior summary message in the kept window (the summary is a User-role
+  message with a distinctive header)
+
+#### Scenario: Self session disambiguation in observer
+
+- **GIVEN** the discarded window contains a reference to a session identifier
+  that is not the running session (e.g. the agent was investigating another
+  session via a tool call)
+- **WHEN** the observer LLM call runs
+- **THEN** the observer system prompt includes the self session id
+- **AND** the produced summary marks the foreign session as `session:{id}`
+- **AND** the produced summary does not conflate the foreign session with the
+  self session
+
+#### Scenario: Tool call/result pair integrity during compaction
+
+- **GIVEN** conversation history contains tool call/result pairs
+- **WHEN** the extractive reducer selects the kept window
+- **THEN** the kept window starts on a `User`-role message (not a
+  `Tool`-role message and not an `Assistant` message that contains
+  `FunctionCallContent` without a matching preceding user turn)
+- **AND** tool call/result pairs are never split across the compaction
+  boundary
+- **AND** older tool interactions remain representable in the journal for
+  checkpoint extraction and summarization

openspec/changes/compaction-rework/tasks.md

@@ -0,0 +1,86 @@
+# Tasks: compaction-rework
+
+## 1. ExtractiveSessionReducer — user-message boundary truncation
+
+- [x] 1.1 Replace the slice-by-count logic in
+  `ExtractiveSessionReducer.ReduceAsync` with a backward walk: starting
+  from `list.Count - keepCount`, walk backward until we hit a `User`-role
+  message that is not prefixed with `SessionState.SystemNudgePrefix`
+- [x] 1.2 Keep-zero edge case: skip the walk entirely and return just
+  the system prompt (or empty when no system prompt is present)
+- [x] 1.3 Unit tests in `ExtractiveSessionReducerTests.cs`:
+  - [x] 1.3.1 `Window_walks_backward_to_user_boundary_when_naive_cut_would_orphan_tool_result`
+  - [x] 1.3.2 `Window_walks_backward_past_assistant_tool_call_to_user_boundary`
+  - [x] 1.3.3 `Window_skips_system_nudges_when_finding_user_boundary`
+  - [x] 1.3.4 `Window_start_already_on_user_boundary_is_preserved`
+  - [x] 1.3.5 `Window_falls_back_to_keep_all_post_system_when_no_user_message_found`
+  - [x] 1.3.6 `Keep_zero_preserves_only_system_prompt` (and no-system variant)
+
+## 2. ObservationPromptBuilder — structured 9-section prompt
+
+- [x] 2.1 Rewrite `BuildObservationSystemPrompt` to accept `SessionId` and
+  emit the nine-section template (Primary Request and Intent, Key Technical
+  Concepts, Files and Code Sections, Problem Solving, Pending Tasks, Task
+  Evolution, Current Work, Next Step, Required Files)
+- [x] 2.2 Add explicit Task Evolution rule: include direct quotes from
+  user messages that changed the task (anti-drift rule)
+- [x] 2.3 Add explicit self-session-id disambiguation rule: "You are
+  summarizing session {id}. Mark foreign session IDs as `session:{id}`,
+  never conflate them with the self session."
+- [x] 2.4 Add explicit "preserve prior summary" rule: "If the input
+  already contains a `[session-summary ...]` block, preserve its sections
+  verbatim and update in place — do not rewrite."
+- [x] 2.5 `BuildObservationUserPrompt` — preserve tool-call arguments as
+  compact `{name}({short-args})` evidence for the observer. Raise tool
+  result truncation to 1500 chars.
+- [x] 2.6 `WrapObservations` takes `SessionId` and produces a canonical
+  `[session-summary session:{id}]` header block. Normalizes any
+  pre-existing header-like first line to the canonical form.
+- [x] 2.7 Update `ObservationPromptBuilderTests.cs`:
+  - [x] 2.7.1 `System_prompt_embeds_self_session_id_for_disambiguation`
+  - [x] 2.7.2 `System_prompt_lists_all_nine_structured_sections`
+  - [x] 2.7.3 `System_prompt_requires_direct_quotes_in_task_evolution`
+  - [x] 2.7.4 `System_prompt_instructs_preserve_prior_summary_verbatim`
+  - [x] 2.7.5 `User_prompt_preserves_tool_call_arguments_as_short_projection`
+  - [x] 2.7.6 `WrapObservations_uses_session_summary_marker_with_session_id`
+
+## 3. SessionCompactionPipeline — thread SessionId
+
+- [x] 3.1 Add `SessionId SessionId` field to `CompactionParameters` record
+- [x] 3.2 Thread `SessionId` through `ExecuteAsync` to
+  `GenerateObservationsAsync` and `WrapObservations` / `BuildObservationSystemPrompt`
+- [x] 3.3 Store the summary as a User-role `SerializableChatMessage` at
+  index 0 of the compacted messages list (the content begins with the
+  `[session-summary session:{id}]` header per task 2.6, which is how
+  consumers recognize it — no separate boundary index is persisted)
+- [x] 3.4 Update `LlmSessionActor` to construct `CompactionParameters`
+  with `_sessionId`
+
+## 4. CompactionIntegrationTests
+
+- [x] 4.1 `Compaction_observer_system_prompt_receives_self_session_id` —
+  inspect `_fakeChatClient.ReceivedMessages` for the observer sidecar
+  call and assert the self session id appears in the system text
+- [x] 4.2 `Compaction_observation_wrapper_embeds_session_id_in_header` —
+  after compaction, the next main-model call includes a User message
+  whose content starts with `[session-summary session:{id}]`
+- [x] 4.3 Existing scenarios continue to pass: buffer drain, session
+  recovery after compaction+kill, emergency compaction with buffered
+  message, summary format with context-summary tags
+
+## 5. Quality gates
+
+- [x] 5.1 `dotnet build src/Netclaw.Actors.Tests/Netclaw.Actors.Tests.csproj`
+  passes with zero warnings
+- [x] 5.2 `dotnet test src/Netclaw.Actors.Tests/Netclaw.Actors.Tests.csproj`
+  — all 909 tests pass
+- [x] 5.3 `dotnet slopwatch analyze` reports no new violations against
+  baseline
+- [x] 5.4 `openspec validate compaction-rework` passes
+
+## 6. PR + commit
+
+- [x] 6.1 Commit with a message referencing the Slack failure and the
+  research sources (Aider, OpenCode, Cline, Claude Code)
+- [x] 6.2 Push branch `compaction-rework`, open PR against `dev`
+- [x] 6.3 PR body references GH issues #595 and #596 as follow-ups