Reconcile stuck agent state against transcript ground truth - #156
Merged
Conversation
Pure helpers that read an agent's on-disk transcript (Claude/Codex JSONL) to report whether the current turn is complete vs in-progress — the ground truth for reconciling a stuck "working" state later. - claudeTurnState: scans back to the last `assistant` record and reads `message.stop_reason` (end_turn => complete, tool_use => in_progress), skipping the trailing bookkeeping records Claude appends after a turn. - codexTurnState: scans back to the last turn-boundary `event_msg` (task_complete/turn_complete/turn_aborted), mirroring cmux's signal set. - readFileTail (256KB positional read), probeTranscript (stat+tail+parse), findCodexTranscriptPath (UUID-guarded recursive lookup). Fail-open by design: truncation/partial/malformed input yields "unknown", never a false "complete". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The agent state machine is event-driven: derived.activity only moves when a hook arrives, with no fallback. A dropped `stop` hook (compact, interrupt, daemon down, resume) strands an agent at "running" so it reads "working" forever, and a daemon restart loses the in-memory interaction registry while the persisted "needs_response" attention lingers. Add a periodic TranscriptReconciler (project-service-scoped) that corrects both against ground truth: - Stuck working: for an agent at activity running/waiting + attention normal, probe its transcript; if the turn is complete AND the file has gone quiescent across a tick (a working agent is still appending), settle it to idle so it derives "ready". Uses setActivity, not task_done — a correction must not bump unseen counts or fire a completion alert. - Stranded needs_response: clear it once no live interaction remains. Started/stopped with the project service (singleton-guarded, not gated on the HTTP endpoint since it talks to the metadata server in-process). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A resumed agent is reattached at its prompt, not mid-generation, but resumeOfflineSession restored the persisted derived state verbatim. A stale activity:"running" (from a turn whose stop hook was dropped before the session was shelved) therefore came back reading "working" forever. On backend resume, settle a "running" activity to "idle" so it derives "ready". A real prompt-submit hook re-marks it running the moment work resumes. Genuine needs_input/blocked (activity "waiting") is preserved; "running" always pairs with attention "normal", so nothing actionable is lost. This is the eager, deterministic counterpart to the periodic transcript reconciler. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Owner
Author
|
@coderabbitai review |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📝 WalkthroughWalkthroughAdds a ChangesTranscript Reconciliation Feature
Sequence Diagram(s)sequenceDiagram
participant Timer
participant TranscriptReconciler
participant probeTranscript
participant MetadataServer
rect rgba(70, 130, 180, 0.5)
note over Timer,MetadataServer: Periodic reconciliation scan
Timer->>TranscriptReconciler: interval tick → scan()
TranscriptReconciler->>MetadataServer: hasPendingInteraction(sessionId)
alt attention=needs_response, no pending interaction
TranscriptReconciler->>MetadataServer: reconcileClearResponse(sessionId)
end
TranscriptReconciler->>probeTranscript: toolConfigKey, resolvedPath
probeTranscript-->>TranscriptReconciler: {turn, size, mtimeMs} | null
end
rect rgba(34, 139, 34, 0.5)
note over TranscriptReconciler,MetadataServer: Quiescence check (two ticks)
alt turn=complete AND size+mtime unchanged from prior tick
TranscriptReconciler->>MetadataServer: reconcileSettleActivity(sessionId)
else transcript grew or still in_progress
TranscriptReconciler->>TranscriptReconciler: store probe snapshot for next tick
end
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/multiplexer/transcript-reconciler.ts`:
- Around line 70-75: The current caching strategy only caches successful lookups
in the codexPathCache, but when findCodexTranscriptPath returns a falsy value
(cache miss), the code does not cache this miss. This causes the same expensive
lookup to re-execute on every tick for unresolved paths. Modify the cache logic
to store both successful paths and cache misses (using a sentinel value such as
null or a special marker) so that repeated lookups for the same session.id that
previously failed do not trigger expensive filesystem scans on every interval.
This pattern should be applied consistently wherever similar Codex path
resolution logic exists in the codebase.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: ed34d5d1-de6e-4cbe-b492-3c48472e6ab4
📒 Files selected for processing (8)
src/metadata-server.tssrc/multiplexer/dashboard-model.tssrc/multiplexer/runtime-state.test.tssrc/multiplexer/runtime-state.tssrc/multiplexer/transcript-reconciler.test.tssrc/multiplexer/transcript-reconciler.tssrc/transcript-turn-state.test.tssrc/transcript-turn-state.ts
…clears - claudeTurnState: treat any terminal stop_reason (max_tokens, refusal, …) as complete, not just end_turn/stop_sequence — a context-maxed agent with a dropped stop hook was never reconciled. Also report in_progress when a new user prompt already follows the last end_turn, closing a false-settle window while the model is thinking before its first token. - Codex path lookups now back off after a miss (8 ticks) instead of walking the ~/.codex/sessions tree every tick; recursion is depth-capped. (CodeRabbit) - Dedupe the Claude transcript-path encoding into a shared claudeTranscriptPath() in backend-session-discovery so it can't drift. - needs_response is only cleared after staying unbacked for a second tick, so a fast daemon restart mid-re-registration can't clear a live interaction. - probeTranscript stats once and passes the size to readFileTail. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
Sub-agent reviewer (orphan finding, not posted inline) — fixed in d1d7ec9: |
Owner
Author
|
@coderabbitai review |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/multiplexer/transcript-reconciler.ts`:
- Around line 39-42: The cache maps codexPathCache and codexMissUntil are keyed
by session.id, but the actual Codex lookups use session.backendSessionId. If
backendSessionId changes for a live session (around lines 73-83), the old cache
entries keyed by session.id will be stale and point to the wrong transcript.
Detect when backendSessionId has changed for a session and clear the
corresponding entries from both codexPathCache and codexMissUntil maps to
prevent stale cache hits/misses from being used with the new backend ID.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: d1e19d81-3e67-4c1f-b701-45ead1a81c19
📒 Files selected for processing (5)
src/backend-session-discovery.tssrc/multiplexer/transcript-reconciler.test.tssrc/multiplexer/transcript-reconciler.tssrc/transcript-turn-state.test.tssrc/transcript-turn-state.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/transcript-turn-state.ts
The codex path cache and miss-backoff were keyed only by session id, but the lookup identity is backendSessionId. If a live session's backend id is rewritten (e.g. a fresh relaunch that supersedes the old id), the stale cached path/miss would be reused. Tag both maps with the backendSessionId they were resolved for and re-resolve on mismatch. (CodeRabbit) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The agent state machine is purely event-driven with no reconciliation:
derived.activityonly moves when a Claude/Codex hook arrives. When the clearingstophook is dropped —/compact, interrupt, daemon down during the 10s hook window, or resume (nostopre-fires; stale state restored from disk) — the agent is stranded atrunningand reads WORKING forever, with nothing to recover it. A daemon restart similarly loses the in-memory interaction registry while a persistedneeds_responselingers.This is the root cause behind agents showing
WORKINGwith "output 1h ago" and never settling toready.Fix — reconcile against ground truth (no decay, deterministic)
The agent's real turn-state is recorded deterministically in its transcript. Three phases:
1. Transcript turn-state readers (
transcript-turn-state.ts, new) — pure helpers. Claude: scan back to the lastassistantrecord,message.stop_reason(end_turn→complete,tool_use→in_progress), skipping the trailing bookkeeping records. Codex: last turn-boundaryevent_msg(task_complete/turn_complete/turn_aborted), matching cmux's signal set. Fail-open tounknown— never a falsecomplete.2. TranscriptReconciler (
transcript-reconciler.ts, new) — 4s, project-service-scoped. Settles a stuckrunning/waiting+normalagent toidle→readyonly when the transcript says the turn is complete and the file is quiescent across a tick (a working agent is still appending). UsessetActivity, nottask_done, so it never bumps unseen counts or fires a "done" alert. Also clearsneeds_responsestranded by a daemon restart.3. Resume settle (
runtime-state.ts) — eager, deterministic counterpart: a backend-resumed agent's stalerunningsettles toidleimmediately. Genuineneeds_input(waiting) is preserved.Covers Claude and Codex. cmux validated the Codex signals + tail-read approach; its own robustness is architectural (it pipes stream-json, which aimux's interactive-tmux model can't adopt), so a reconciler is the right adaptation.
Process
Built via plan-execute: 3 phases, each plan-audited and impl-audited by an independent agent (2 blockers caught + fixed in Phase 2: Codex path-cache pruning, reconciler gated on HTTP endpoint).
Tests
33 new tests across
transcript-turn-state.test.ts,transcript-reconciler.test.ts,runtime-state.test.ts. Full suite 1370 passing, typecheck/lint/build clean.🤖 Generated with Claude Code
Summary by CodeRabbit
Release Notes
New Features
Bug Fixes
Tests