[Bug] buzz-acp: dead Claude Code session never recycled — events retried, dead-lettered, then bound channels go permanently silent

[cristiansotogarciaxatech]

Summary

When the Claude Code process behind an ACP session crashes, buzz-acp keeps routing events to the dead session instead of recycling it. Every delivery fails with -32603, batches are retried with backoff and then dead-lettered (events discarded). After the final dead-letter the agent slot stops logging entirely, and every channel bound to it becomes a black hole: the agent still shows an online presence heartbeat but never responds. A message from a brand-new channel forced the harness to spawn a fresh session, which worked immediately — that is how we eventually recovered, ~5.5 hours later.

Environment

• Buzz Desktop on Windows 11 Pro (10.0.26200), app id xyz.block.buzz.app
• buzz-acp harness, agent_cmd=claude-agent-acp.cmd (@agentclientprotocol/claude-agent-acp 0.64.0, bundled Node v24.18.0)
• Harness config from startup line: idle_timeout=900s max_turn=7200s agents=10 subscribe=Mentions dedup=Queue meh=Steer permission_mode=bypassPermissions
• Agent model: claude-fable-5

Timeline (agent log agents/logs/<pubkey>__<relay>.log, 2026-08-01, UTC)

1. 07:05:35 — first failure while a turn was in flight:

   WARN buzz_acp: agent_returned (application error — pipe intact) agent=0 outcome="error" pid=29144
   error=Agent reported error (code -32603): Internal error: [ede_diagnostic] result_type=user last_content_type=n/a stop_reason=null
   

2. Immediately after, the adapter reports the child process is gone:

   Session 02f491a6-ab1f-45d0-a184-2ebadb9141f7: Claude Agent process died: Claude Code process exited with code 3221226505
   

   (3221226505 = 0xC0000409 STATUS_STACK_BUFFER_OVERRUN, a fail-fast abort.)
3. 07:13:18 — next delivery attempt returns:

   Agent reported error (code -32603): Internal error: The Claude Agent process exited unexpectedly. Please start a new session.
   

4. 07:13 → 07:38 — the same batch is requeued with exponential backoff, attempts 1..10, every attempt hitting the same dead session. Then:

   ERROR buzz_acp::queue: dead-lettering batch after 10 retries — discarding 3 events channel_id=<channel-A> attempt=11 events=3
   

5. 07:44 → 09:11 — the cycle repeats for newly arriving events; at 09:11:34 a second dead-letter discards 2 more events.
6. 09:11:34 → 12:47:00 — no queue/agent log lines at all. During this window the workspace owner sent multiple DMs and mentions to the agent (four messages over ~3 hours); none were delivered, nothing was logged, and no error was surfaced anywhere user-visible.
7. 12:47:00 — INFO buzz_acp: membership notification: subscribing to new channel channel_id=<channel-B> (a different user opened a brand-new DM with the agent) → the harness spawned a fresh session for it, which worked immediately and has been healthy since. The originally bound channels (owner DM, team channel) remained dead until the harness was restarted.

Problems

1. Dead session is never recycled. The harness logs Claude Agent process died, and the adapter's own error text says "Please start a new session" — but deliveries keep being classified as application error — pipe intact and retried into the pinned dead session. The pipe to the adapter is intact; the session behind it is gone.
2. Dead-lettering silently discards user events. Real messages and mentions are dropped after 10 retries with no owner-visible notification and no redelivery once a healthy session exists.
3. After the final dead-letter the slot goes fully silent. Subsequent events for channels bound to that agent slot produce no queue activity at all — not even retry logs — so the failure also stops being observable.
4. Net effect: the agent looks online (presence heartbeat keeps publishing) while its owner's DM channel is a black hole for hours. The owner had to reach the agent through a second user's DM, which happened to force a fresh session.

Expected behavior

• On -32603 with a session-dead signal (process died / "Please start a new session"): tear down and respawn the ACP session (or open a new session) before retrying, instead of retrying into a dead session.
• Dead-lettered events should surface to the owner, or be redelivered once a healthy session for that channel exists.
• A dead/wedged agent slot should not keep channels permanently bound to it while presence continues to report the agent as online.

The underlying Claude Code crash (exit 0xC0000409 following the ede_diagnostic internal error) is an upstream agent issue; this report is about the harness's handling — a single agent-process crash should not take an agent's channels offline for hours.

Possibly related: #2961 (per-channel ACP session lifecycle on Windows — sessions never released).

[pat-dubois]

Second platform (Linux), and a different trigger with the same destructive ending — which I think argues the fix belongs one level lower than session recycling.

We hit dead-lettered user messages on 2026-07-30, but our ACP session never died. The child process was healthy the whole time. What failed was the model provider returning a transient quota error, and the harness treated it as a normal retryable failure, exhausted the retry budget, and destroyed the events.

Environment

• Ubuntu 24.04.4 LTS, kernel 6.8.0-134
• Self-hosted relay + buzz-acp built from 6300a6b1
• @agentclientprotocol/claude-agent-acp 0.63.0, model Fable 5
• idle_timeout=900s max_turn=7200s agents=1 subscribe=All dedup=Queue meh=Steer permission_mode=bypassPermissions respond_to=owner-only

Incident A — 4 events discarded, 45 minutes before the limit was due to lift

11:30:51Z WARN  buzz_acp::queue: requeueing failed batch with backoff attempt=1 max=10 delay_secs=4.39 events=1
11:30:51Z WARN  buzz_acp: agent_returned (application error — pipe intact) agent=0 outcome="error"
          error=Agent reported error (code -32603): Internal error: You've hit your session limit · resets 6:40am (America/Regina)
...  attempts 2..10, delays 8s → 17s → 33s → 65s → 141s → 241s → 250s → 249s → 243s ...
11:55:04Z ERROR buzz_acp::queue: dead-lettering batch after 10 retries — discarding 4 events attempt=11

The whole retry budget burned in 24 minutes. The error string names its own reset time: 6:40am America/Regina is 12:40Z — the harness gave up and discarded the messages 45 minutes before the condition would have cleared on its own.

Incident B — same day, same channel, 1 more event discarded

16:42:14Z ERROR buzz_acp::queue: dead-lettering batch after 10 retries — discarding 1 events attempt=11
16:42:14Z WARN  buzz_acp: agent_returned (application error — pipe intact) agent=0 outcome="error"
          error=Agent reported error (code -32603): Internal error: You're out of usage credits.
                Run /usage-credits to keep using Fable 5 or /model to switch models.

Five of the workspace owner's messages destroyed across the two incidents. 11 quota errors that day. Nothing was surfaced to the owner and presence stayed online throughout — the same silent-black-hole symptom described in the original report.

Three things this case adds

1. -32603 is overloaded, and the fix in this issue may not catch quota. The proposal here keys on session-dead signals (process died / "Please start a new session"). Ours reports application error — pipe intact and is genuinely accurate about that: the pipe was intact and the session was alive. A session-recycle fix would correctly skip our case and we would keep losing messages. The common factor across both reports is not the session state — it is that the dead-letter path discards user data.

2. The retry budget is the wrong shape for quota. Backoff caps out around 250s, so 10 attempts is always ~24 minutes regardless of the failure. Provider quota windows are hours. A budget that can never outlast the failure class is guaranteed to discard. The provider even hands us the answer in plain text (resets 6:40am (America/Regina)) and we throw it away with the batch.

3. Late-arriving messages get almost no grace. Watch the batch grow across attempts: events=1 at attempt 1, events=3 at attempt 7, events=4 at attempt 8. Messages that arrived after the failure were merged into a batch already 7-8 retries deep and were destroyed within minutes of being sent. Effective retry budget depends on when you happened to speak.

What would have prevented all five losses

In rough order of value to us:

• Never discard. Park a failed batch and redeliver when a healthy session exists. Dead-lettering is fine for poison events; it is the wrong default for "the model was briefly unavailable."
• Classify quota/rate-limit separately and back off on the provider's own stated reset time instead of a fixed attempt count.
• Tell the owner. A single "I dropped 4 of your messages" beats silence — right now the only trace is a server log line the owner never sees.
• Don't merge new events into a batch that is already deep into its retry budget.

Happy to split this into its own issue if you would rather keep this one scoped to the crashed-session case — say the word and I will move it. Also glad to test a patch on Linux.

[cristiansotogarciaxatech]

Reproduced today (2026-08-05) with a clean evidence chain, and it isolates the exact mechanism: the harness applies its transient-error retry policy to a permanent session failure.

Sequence (all times UTC, from agents/logs/<agent>__<host>.log):

1. A long-lived Claude Code session bound to one channel accumulated ~820K cached input tokens over ~36h of continuous use (final turn usage: cache_read_input_tokens: 819583). Its transcript stops at 07:03:14Z — the session is dead from that point.
2. The one time the SDK's real error text surfaces, it is explicit:
   11:17:58Z WARN agent_returned ... error=Agent reported error (code -32603): Internal error: Prompt is too long
3. From 11:40:45Z through at least 15:19:17Z, every delivery to that channel cycles attempts 1→10 with exponential backoff (~4s→255s), then:
   ERROR dead-lettering batch after 10 retries — discarding N events ... attempt=11
   …and the next batch restarts at attempt 1. 100% failure for ~3.5 hours; the cascade cannot ever succeed because the bound session is permanently over the context limit.
4. Meanwhile the same agent pid serves other channels normally (steer success: renewed hard deadline during the failure window) — so presence shows online and the operator has no signal that one channel is a black hole. Users saw four -32603 Internal error bubbles (12:05, 12:23, 14:01, 14:26) with no detail.

Two concrete asks this evidence supports:

1. Terminal-error classification. Prompt is too long (and context-overflow generally) is not retryable. On seeing it, the harness should retire the session and spawn a fresh one for that channel binding instead of retry→dead-letter→discard forever. That single change converts a permanent channel outage into a one-turn blip.
2. Propagate error detail. Of the many -32603s in this window, only one carried the underlying reason; the rest surfaced as bare Internal error. The SDK detail (Prompt is too long, [ede_diagnostic] ...) exists in at least some responses — always forwarding it would have made this diagnosable from chat instead of requiring log forensics.

Workaround confirmed again: restarting the agent from Buzz Desktop clears the dead binding (same as the 08-01 incident in the original report).