Fix chat recovery being permanently abandoned under repeated deploys#1615
Merged
Conversation
A mid-turn deploy resets the Durable Object ("Durable Object reset because
its code was updated"); the interrupted chat continuation is re-detected on
the next wake. Previously every such interruption consumed one of the bounded
recovery attempts, so a deploy every few minutes drove the incident to
`max_attempts_exceeded` and terminally abandoned a turn that was actually
making progress — even though each fresh isolate was healthy and served new
messages fine.
Root cause: `_beginChatRecoveryIncident` incremented `attempt` on every
eviction-detection and terminalized at `attempt > maxAttempts`, with no notion
of whether recovery was advancing. An environmental interruption (deploy) was
indistinguishable from a genuinely failing turn.
Fix (in both @cloudflare/think and @cloudflare/ai-chat, which share this
logic): reset the attempt budget when recovery has made forward progress since
the last attempt (count of persisted assistant messages — `_persistOrphanedStream`
persists each interrupted partial and does not prune it, so the count is
monotonic within an incident). A turn that never advances still exhausts at
`maxAttempts`. A 15-minute wall-clock ceiling per incident bounds the worst
case so a continuously churning environment cannot retry forever.
Validation:
- New unit tests in both packages cover the progress-reset and wall-clock
branches (the previous exhaustion tests only covered the no-progress path).
- All suites pass: agents (1662), ai-chat (543), think (429), plus the
kill/restart e2e recovery suites (think 9, ai-chat 2).
- Reproduced and verified against real deploys with the new
examples/deploy-churn harness: the same 10-deploy churn that previously
exhausted recovery at deploy 5 now self-recovers and stays healthy.
Also adds examples/deploy-churn: a harness that drives a durable chat turn
against a deployed Worker while firing repeated real `wrangler deploy`s
mid-turn, records a millisecond timeline (deploys, script versions, WS
reconnects, recovery-incident transitions), and captures errors via
`onChatError`/`onError`. Used to find and verify this fix.
Co-authored-by: Cursor <cursoragent@cursor.com>
🦋 Changeset detectedLatest commit: 67941df The changes in this PR will be included in the next version bump. This PR includes changesets to release 2 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Fixes a typecheck error: the test helper was added to the agent worker but not to the typed RPC stub interface used by the durable-chat-recovery tests. Co-authored-by: Cursor <cursoragent@cursor.com>
Format README table for consistent column alignment and switch a word to italic emphasis. Reorder dependencies in examples/deploy-churn/package.json (moved `@cloudflare/think` below `@cloudflare/kumo`) — purely stylistic change with no functional impact.
agents
@cloudflare/ai-chat
@cloudflare/codemode
hono-agents
@cloudflare/shell
@cloudflare/think
@cloudflare/voice
@cloudflare/worker-bundler
commit: |
Merged
7 tasks
threepointone
added a commit
that referenced
this pull request
May 29, 2026
…s on a code-update DO reset (#1617) * test(ai-chat): reproduce recovery abandonment on a superseded-isolate alarm Adds a deterministic reproduction (currently RED) for a second deploy-churn abandonment path that #1615 cannot reach. After an interrupted turn is detected, recovery schedules `_chatRecoveryContinue` and deletes the orphaned `cf_agents_runs` row — so the turn rides solely on the one-shot schedule row. If that alarm then fires on a SUPERSEDED isolate, the first storage op throws the catchable `Durable Object reset because its code was updated.` for the whole invocation; `_executeScheduleCallback` burns its in-process retries, swallows the error, and `alarm()` deletes the one-shot row. Both recovery vehicles are now gone and `_beginChatRecoveryIncident` never runs again — the turn is permanently abandoned. The test drives the REAL dispatch path (`alarm()` -> `_executeScheduleCallback` -> swallow -> one-shot DELETE); the only injected element is the (production-observed) reset error itself, thrown from the recovery callback via a test-only `setSimulateSupersededIsolateForTest` flag. It asserts the desired post-fix behavior (a recovery vehicle survives), so it fails on current main. The fix follows in the next commit. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(agents): defer one-shot recovery callbacks on a code-update DO reset When a scheduled one-shot callback fires on an isolate a deploy has just superseded, the first `ctx.storage` op throws `Durable Object reset because its code was updated.` for the whole invocation. `_executeScheduleCallback` previously burned its in-process retries (all doomed — code never reloads mid-invocation), swallowed the error, and let `alarm()` delete the one-shot row — permanently abandoning the work. For chat recovery this is a second deploy-churn abandonment path, upstream of and unreachable by the progress-aware budget added in #1615: the orphaned fiber row is deleted as soon as recovery is "handled", so the turn rides solely on the `_chatRecoveryContinue` / `_chatRecoveryRetry` schedule row, and deleting it leaves nothing for the boot scan to re-detect. Fix: for a one-shot row failing with this transient, skip the doomed in-process retries (`shouldRetry`) and re-throw instead of swallowing, so `alarm()` rejects, the one-shot row survives, and the platform re-runs the alarm on a fresh isolate (= new code) under the at-least-once guarantee. The work auto-resumes once the deploy settles. Every other callback and error class keeps the existing swallow-and-exhaust behavior. Turns the reproduction test added in the previous commit green; verified it fails without this change. All suites pass (agents 1662, ai-chat 544, think 431, e2e think 9 + ai-chat 2). Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced May 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Under repeated deploys, durable chat recovery could permanently abandon an in-flight turn — even though the Durable Object itself stays healthy. A mid-turn deploy resets the DO (
Durable Object reset because its code was updated); the interrupted continuation is re-detected on the next wake, and every such interruption consumed one of the bounded recovery attempts. So a deploy every few minutes drove the incident tomax_attempts_exceededand terminalized a turn that was actually making progress. The turn then never retried until the user sent a new message.This was reproduced empirically against real deploys (see the new harness below): a 10-deploy churn over a single turn exhausted recovery at deploy 5 (attempt 8/6), the orphaned fiber row never cleared, and the turn was abandoned — while a fresh message afterward worked fine, proving only the turn was orphaned, not the DO.
Root cause
_beginChatRecoveryIncident(duplicated in@cloudflare/thinkand@cloudflare/ai-chat) incrementedattempton every eviction-detection and terminalized atattempt > maxAttempts, with no notion of whether recovery was advancing. A deploy-driven interruption was indistinguishable from a genuinely failing turn, so deploy churn burned the whole budget.Notably, the reset is a hard isolate kill — not a catchable JS throw — so
onChatError/onErrornever fire for it; the only server-side symptom is the orphaned fiber + the attempt counter, which is exactly what this logic governs.The fix
_persistOrphanedStreampersists each interrupted partial and does not prune it, so the count is monotonic within an incident), reset the attempt counter. An environmental interruption that followed progress no longer counts against the budget; a turn that never advances still exhausts atmaxAttempts.CHAT_RECOVERY_MAX_WINDOW_MS) bounds the worst case so a continuously churning environment cannot retry forever.Applied identically to both packages (they share this recovery foundation).
Edge cases considered
maxAttempts(marker stays flat) — preserved, and confirmed by the existing exhaustion tests still passing.progress→ defaults to0→ at most one lenient reset on first detection (errs toward recovering).progress: max(prev, current)) so message deletion /clearHistorycannot lower the bar or trigger a spurious reset.Validation
resets the attempt budget when recovery makes forward progressexhausts via the wall-clock window even while making progressattemptat 1–2 throughout, never exhausts, and self-recovers (continuation completes) once churn settles —SELF-RECOVERED · DO HEALTHY.Also included:
examples/deploy-churnA reliability harness that drives a durable chat turn against a deployed Worker while firing repeated real
wrangler deploys mid-turn. It records a millisecond timeline (deploys + script versions, WS reconnects, recovery-incident transitions), captures errors viaonChatError/onError, and prints a verdict distinguishing self-recovery vs. exhaustion vs. a wedged DO. This is the tool used to find and verify the fix; it also serves as a repro others can run. Flags include--deploys,--duration,--no-monitor(proves the failure is deploy-driven, not the client reconnect storm), and--client partysocket.Follow-ups (not in this PR)
ChatRecoveryConfig(maxRecoveryWindowMs).agents/chatso think + ai-chat can't drift.Test plan
npm run test -w @cloudflare/thinknpm run test -w @cloudflare/ai-chatnpx nx run-many -t test --projects=agentsnpm run test:e2e -w @cloudflare/thinknpm run test:e2e -w @cloudflare/ai-chatexamples/deploy-churn(before: exhausts; after: self-recovers)Made with Cursor