Harden Think and AIChat recovery foundations#1611
Merged
Merged
Conversation
🦋 Changeset detectedLatest commit: 93d3d55 The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
Add characterization coverage for the production recovery gaps before changing runtime behavior. These tests capture the current failure surfaces so the follow-up fixes can prove that they improve behavior rather than only reshaping APIs. The Think recovery e2e harness now covers repeated restart churn around an interrupted turn, the post-persist/pre-turn request failure path, and parent agent-tool recovery interruption after restart. The test worker persists recovery observations across restarts and exposes test-only controls for forced turn failures and retained agent-tool rows. The focused regression tests also document two non-restart reliability issues: poisoned transcripts with orphan tool calls fail later turns, and createCompactFunction can skip summarization when tool-heavy histories are under-counted by the heuristic tail budget. This commit intentionally contains only tests and test harness changes. Runtime recovery behavior is left unchanged for the subsequent fix commits. Co-authored-by: Cursor <cursoragent@cursor.com>
Mirror the Think restart-churn characterization coverage in the AIChatAgent recovery e2e harness before changing shared recovery behavior. Both chat packages use the same underlying fiber recovery pattern, so the shared fix work needs coverage that proves AIChatAgent receives the same protection rather than only improving Think. The e2e worker now persists recovery observations in Durable Object storage so restart churn does not lose the evidence we need to assert on. The new e2e test starts a slow recovered chat turn, repeatedly kills and restarts wrangler around the interrupted fiber, and verifies recovery still fires and stale fiber rows are cleaned up. This commit is intentionally limited to characterization coverage. It excludes Think-only concerns such as Session compaction and durable submissions, which do not apply to AIChatAgent. Co-authored-by: Cursor <cursoragent@cursor.com>
Introduce dedicated observability event types and diagnostics channels for the recovery work that follows. Fiber recovery, chat recovery, transcript repair, and agent-tool reconciliation now have stable event names instead of being mixed into unrelated message or lifecycle streams. The channel routing tests lock in the public names for the new channels and ensure chat:transcript:* events land on the transcript channel while chat recovery events stay on the chat channel. Co-authored-by: Cursor <cursoragent@cursor.com>
Make the generic fiber substrate safer before adding chat-specific retry policy. Recovered fibers now carry a recoveryReason, interrupted run rows emit fiber recovery lifecycle events, and internal framework recovery hooks are bounded by a default timeout so startup cannot wedge forever on a broken internal recovery path. Managed fibers whose recovery hook throws are now marked terminal error instead of staying indefinitely interrupted with only an error message attached. Unmanaged run rows continue to be pruned after recovery handling so the same broken stale row does not re-trigger forever across boots. This also exposes the agentTool observability channel under the camelCase subscribe key while preserving the diagnostics channel name agents:agent_tool. Co-authored-by: Cursor <cursoragent@cursor.com>
Add shared chat recovery configuration and incident context so Think and AIChatAgent recovery are bounded by framework-owned attempt state. Recovery hooks now receive incidentId, attempt, maxAttempts, and recoveryKind, with attempt === 1 representing the first handling of an incident. Both chat runtimes persist recovery incidents in Durable Object storage, emit chat recovery lifecycle events, use configurable stable-state timeouts, and stop scheduling more recovery work once maxAttempts is exceeded. Exhaustion emits a terminal chat recovery event and sends a user-visible terminal chat error frame; Think also marks any matching running submission as error. The default behavior is enabled for existing chatRecovery=true users with maxAttempts 6 and stableTimeoutMs 10000, while custom policy can be supplied via chatRecovery object configuration. Co-authored-by: Cursor <cursoragent@cursor.com>
Extend Think's existing onChatError hook with optional stage context and route post-persist request failures through it. The request path now emits chat:request:failed with request id, stage, persistence state, and sanitized error text before sending the terminal chat error frame. This gives applications a server-side hook for failures that occur after user messages have been accepted but before or during the model turn, without routing those chat failures through the generic Agent onError hook. Co-authored-by: Cursor <cursoragent@cursor.com>
Add a total recovery deadline for parent-side agent-tool reconciliation so a restart cannot spend unbounded time inspecting stale child rows. Per-child inspection remains bounded, and any rows reached after the total deadline are terminalized as interrupted with an explicit recovery-deadline error. The recovery loop now emits structured agent_tool recovery lifecycle events for begin, per-row outcome, deadline, completion, and unexpected failure, giving operators visibility into exactly which child run was finalized and why. Co-authored-by: Cursor <cursoragent@cursor.com>
Repair incomplete tool-call transcript shapes before Think converts UI messages for provider calls. Persisted orphan tool calls are stripped from the active Session history, clients are rebroadcast with the repaired transcript, and a chat:transcript:repaired event records the repair counts and tool call ids. Also let createCompactFunction use an optional tokenCounter for tail-budget selection. Callers with tokenizer or model-reported accounting can avoid the heuristic under-count that otherwise protects too much tool-heavy history and skips summarization. Co-authored-by: Cursor <cursoragent@cursor.com>
Record the public recovery API, observability, and behavior changes across the published packages so the release notes explain the new default recovery bounds and transcript/compaction fixes. Co-authored-by: Cursor <cursoragent@cursor.com>
Expose the new chat recovery config and exhaustion context types from agents/chat so downstream packages can typecheck against the shared lifecycle surface. Update the Think test fixture to construct the enriched fiber recovery context. Co-authored-by: Cursor <cursoragent@cursor.com>
Stabilize chat recovery incident accounting across retry fibers, preserve failed internal recovery rows for later scans, and avoid stale continuation targets after orphan persistence. Tighten transcript repair behavior and document the new recovery and observability surfaces so users can configure and debug bounded recovery. Co-authored-by: Cursor <cursoragent@cursor.com>
Document the distinction between stream resumption and durable chat recovery so users can configure and observe recovery behavior correctly. Co-authored-by: Cursor <cursoragent@cursor.com>
Track length-preserving part replacements so repaired tool inputs are written back before provider calls. Co-authored-by: Cursor <cursoragent@cursor.com>
Align transcript repair observability with AI SDK v6 message parts by dropping the unused removedToolResults field. Co-authored-by: Cursor <cursoragent@cursor.com>
Follow-up hardening from a full review of the recovery foundation. These close edge cases where bounded recovery could still leak storage, exceed its attempt budget, or fail to surface a stuck turn. Chat recovery incidents (Think + AIChatAgent): - Drop `recoveryKind` from the incident identity so a single interrupted turn that flips between `retry` (no chunks persisted) and `continue` (partial chunks exist) across restarts shares one attempt budget. The kind is still tracked as a mutable field on the incident record. - Delete the incident record on `completed` (success) and add a TTL sweep (1h inactivity) on each new incident so durable storage no longer grows without bound. Exhausted/failed/skipped records are retained for inspection until they age out. - Guard a throwing `onExhausted` hook so the terminal error frame (and Think's running-submission interruption) is always delivered. - Wrap the post-begin recovery dispatch (onChatRecovery, orphan persist, scheduling) so any throw flips the incident to a terminal `failed` state and emits `chat:recovery:failed` instead of leaking in `attempting`. Generic fiber recovery (Agent): - Add `fiberRecoveryMaxAgeMs` (default 24h). A repeatedly-throwing unmanaged `onFiberRecovered()` row is still retried while fresh but is evicted with a `fiber:recovery:skipped` / `max_age_exceeded` event once it ages out, so a poison row cannot re-trigger forever across boots. - Note that the fiber recovery hook timeout bounds the wait but does not cancel the underlying internal operation. Observability: - Replace the hand-coded `agentTool` subscribe special-case with a `CHANNEL_DIAGNOSTIC_NAME_OVERRIDES` lookup to prevent future drift between camelCase keys and snake_case diagnostics channel names. Tests: - Think + AIChatAgent: shared attempt budget across retry/continue flip, incident deletion on completion, stale incident sweep, `failed` transition when onChatRecovery throws, and terminal UX delivery when onExhausted throws. - Agent fibers: a fresh throwing unmanaged row is retained (retryable), an aged throwing row is evicted with `max_age_exceeded`. - Update existing incident-id assertions to the kind-less format. All unit and e2e suites pass (Think, AIChatAgent, Agent fiber recovery, observability routing). Co-authored-by: Cursor <cursoragent@cursor.com>
threepointone
force-pushed
the
think-recovery-foundation
branch
from
May 29, 2026 12:28
d9343c2 to
0716c1c
Compare
Clarify that an `onFiberRecovered()` hook which always throws is retried only until the row exceeds `fiberRecoveryMaxAgeMs` (default 24h), after which it is discarded with a `fiber:recovery:skipped` / `max_age_exceeded` event. Previously the bullet implied a thrown hook kept the row indefinitely. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR builds a recovery foundation for production chat reliability across
@cloudflare/think,@cloudflare/ai-chat, genericAgentfibers, and retained agent-tool runs.The work started from production reports that Think recovery was unreliable under Durable Object restarts and process churn. Instead of patching individual symptoms, this branch first adds characterization and e2e coverage for the failure modes, then introduces bounded recovery state machines, durable incident tracking, observability, transcript repair, and documentation that explains the difference between client stream resumption and Durable Object eviction recovery.
Problem
Production reports pointed at several related failure modes:
onChatErroror observability.What Changed
1. Characterization and e2e coverage
Adds focused coverage before the fixes:
These tests exercise the branch through realistic
wrangler devrestarts and direct recovery status inspection instead of only unit-level mocks.2. Recovery observability taxonomy
Adds granular recovery lifecycle events and channels:
fiber:run:*andfiber:recovery:*chat:request:failedchat:recovery:*chat:transcript:repairedagent_tool:recovery:*Also adds dedicated diagnostics channels:
agents:chatagents:transcriptagents:fiberagents:agent_toolThe typed
subscribe()helper exposesagentToolas the public camelCase key while raw diagnostics channel subscribers useagents:agent_tool.3. Bounded generic fiber recovery
Generic
runFiber()recovery now has clearer lifecycle behavior:recoveryReason: "interrupted"toFiberRecoveryContext.fiberRecoveryHookTimeoutMs.fiberRecoveryScanDeadlineMs.onFiberRecovered()throws, allowing a later scan to retry the recovery hook.errorif framework recovery throws, rather than silently deleting failed internal recovery state.This keeps application recovery hooks retryable while letting framework-managed fibers fail terminally when the framework knows the row cannot safely be recovered.
4. Bounded chat recovery incidents
Both
ThinkandAIChatAgentnow track durable chat recovery incidents with:incidentIdattemptmaxAttemptsrecoveryKind(retryorcontinue)chatRecovery = { maxAttempts, stableTimeoutMs, terminalMessage, onExhausted }The recovery state machine now distinguishes:
Think keeps
chatRecoveryenabled by default. PlainAIChatAgentremains opt-in withchatRecovery = falseby default to preserve existing behavior.5. More robust recovery identity and continuation handling
The branch hardens edge cases found during review:
recoveryRootRequestIdto chat fiber snapshots so retries and continuations remain tied to the original interrupted request.conversation_changedskips.finallyblocks so nested or failed scheduled recovery paths do not leak state.6.
onChatErrorrequest failure contextThink's
onChatErrornow accepts an optional context object:The context includes:
requestIdstagemessagesPersistedThink now emits
chat:request:failedfor request and stream failures, giving users and operators a consistent hook for distinguishing pre-persist failures from failures after the user message is already durable.7. Bounded parent agent-tool recovery
Parent agent-tool recovery now has a total scan deadline and emits detailed recovery events:
If retained child runs cannot be reconciled in time, they are marked
interruptedinstead of leaving parent tool calls in active states indefinitely.8. Transcript repair before provider calls
Think now repairs provider-breaking transcript damage before calling the model:
chat:transcript:repairedThe repair pass is intentionally narrow: it avoids deleting otherwise-valid empty or step-start assistant messages and only drops messages made empty by orphan tool-call removal.
9. Compaction token counter support
createCompactFunction()now accepts a customtokenCounterso applications with model-reported usage or tokenizer access can make better tail-budget decisions for tool-heavy histories.CompactTokenCounteris exported from the memory utilities package.10. Docs, README, examples, and design notes
Updates user-facing and contributor-facing surfaces so the new recovery model is discoverable:
AIChatAgentopt-in recovery and Think's default recovery.onChatRecoveryguidance.onChatError(error, ctx)docs.tokenCounterusage.Commit Stack
The branch is intentionally split for review:
test: cover Think recovery failure surfacestest: cover AIChatAgent recovery failure surfacesfeat: add recovery observability channelsfeat: bound and observe fiber recoveryfeat: add bounded chat recovery incidentsfix: surface Think chat request failuresfix: bound agent-tool recovery scansfix: repair poisoned chat history before model callschore: add recovery foundation changesetfix: export chat recovery lifecycle typesfix: harden recovery edge cases and docsdocs: clarify recovery UX surfacesAPI and Behavior Notes
Think.chatRecoveryremains enabled by default, but can now be configured with an object.AIChatAgent.chatRecoveryremains disabled by default and is opt-in.onChatRecovery(ctx)receives incident metadata:incidentId,attempt,maxAttempts, andrecoveryKind.onChatError(error, ctx)is backward-compatible for subclasses that only accept one parameter.FiberRecoveryContextnow includesrecoveryReason.onFiberRecovered()failures preserve the fiber row for retry instead of deleting it.interruptedinstead of hanging.Test Plan
Already run locally:
npm run formatnpm run checknpm run checkcompleted successfully, including:sherifoxfmt --check .oxlint .Focused tests and e2e suites were also run during the implementation/review cycle for the touched areas, including Think recovery, AIChatAgent recovery, observability routing, fiber recovery, transcript repair, and session compaction regressions.
Note:
sherifstill reports the repository's existing warnings for example folders withoutpackage.json, but the command exits successfully.Review Guide
Suggested review order:
packages/agents/src/index.ts.Think/AIChatAgentrecovery state machines together.Risk and Rollout
The main runtime risk is recovery behavior changing from implicit/unbounded attempts to explicit bounded incident tracking. That is intentional: repeated failures now become observable and terminal instead of leaving users with a stuck turn.
The highest-risk code paths are:
The PR includes focused tests for these areas and e2e restart coverage for both Think and AIChatAgent.
Made with Cursor