buzz-acp: hung in-flight turn has no watchdog — steer renews deadline, withheld mentions invisible until manual restart

[Tony-ooo]

Summary

When a buzz-acp in-flight turn's agent read loop dies (agent process alive but wedged), there is no effective watchdog: the turn stays "in-flight" for up to the renewed 2h deadline, a single successful steer renews that deadline, subsequent mentions get no steer ack and sit in the withheld table, and deadline expiry only recovers queue events — it never cancels the hung turn. The channel is effectively blocked until the agent instance is manually restarted, with no error surfaced anywhere.

Observed incident (2026-08-05, Codex agent, Windows)

• 07:56:48 UTC — agent ACKs a task, starts investigating; research notes complete on disk at 08:02:15 UTC; final reply never sent.
• 08:03:54 — last buzz-acp log line; process tree (buzz-acp → codex-acp adapter → codex app-server → session kernel) all alive, CPU ≈ 0, no child processes, no agent_returned, no error, no restart (managed-agents.json last_error empty, last_started_at unchanged).
• 08:20:55 — a mention for the channel is steered: non-cancelling steer ack received ... ack=Ok(Success) release_withheld=false drop_withheld=true + extending in-flight deadline by 7200s + 100s buffer. The agent "received" the steer but produced nothing in the next 10+ minutes.
• 08:27:41 — next mention: no steer ack at all (read loop no longer selecting the steer arm). Message parked in withheld_native_steer.
• Only recovery: manual restart of the agent instance in Desktop.

Mechanism (buzz-repo @ 7bee84d)

• crates/buzz-acp/src/lib.rs:2522-2627 — on SteerAck::Success the withheld event is dropped and extend_in_flight_deadline(channel_id, max_turn_duration) renews the in-flight deadline (7200s).
• crates/buzz-acp/src/lib.rs:1147-1164, 2256 + crates/buzz-acp/src/pool.rs:300-421 — steer is a capacity-1 per-turn channel consumed by the prompt read loop (acp.rs:1280-1347). If the read loop is wedged (blocked in a model read or synchronous call), the steer request never gets an ack; there is no timeout on the ack itself.
• crates/buzz-acp/src/queue.rs:155-166, 280-287, 576-580, 758-771 — withheld steer events are only recovered on in-flight-deadline expiry; expiry does not cancel the turn or signal the agent. A Err(_) steer ack would fire the cancel+merge fallback (signal_fallback), but a missing ack is never treated as failure.
• No heartbeat exists between buzz-acp and the agent while a turn is running; agent_returned never fires for the wedged turn.

Impact

• Channel blocked for up to 2h per renewed deadline; further mentions are silently invisible (no ack, no error).
• Operator-facing symptom: agent process "alive" (green), zero output, no logs — indistinguishable from idle without deep inspection.
• Same symptom class as the Hermes hang on Windows (process alive, CPU ≈ 0, no timeout fired) — likely a family of "wedged tool/model call inside a turn" cases.

Expected behavior

• A watchdog should detect a non-responsive in-flight turn (no steer ack and/or no progress for a bounded window — do not let steer success renew the deadline indefinitely) and:
  1. cancel the turn via the existing cancel+merge fallback path (ControlSignal::Steer, signal_fallback semantics in lib.rs:2522-2627),
  2. surface an error/status to the Desktop UI (the agent currently shows healthy),
  3. recover the withheld events to the queue for redelivery.
• Consider a per-turn progress heartbeat between buzz-acp and the agent, with a hard cap that cannot be extended by steers alone.

Related

• buzz-agent/buzz-acp: transient network outage paralyzes agent for 12+ min; retries noisy and recovery passive #4477 (transient network outage paralyzes agent; passive recovery)
• [Bug] Hermes ACP initializes but every managed prompt hangs until 60s timeout on Windows #4098 (Hermes ACP prompts hang until 60s timeout on Windows)
• fix(acp): retry channel turns that leave a mention unanswered #3648 (retry channel turns that leave a mention unanswered — closed)
• fix(acp): name the hung RPC method in request timeout errors #2554 (name the hung RPC method in request-timeout errors)

[aflanagan]

Independent repro on macOS (same signature, cross-platform)

Seeing this repeatedly on macOS (Apple Silicon, headless) running a Codex agent via buzz-acp (codex-acp 1.1.9, codex-cli 0.145.0). Same signature as the report:

• Codex agent ACKs a mention (posts its 👀/💬 reactions) then never posts the text reply.
• Whole process tree stays alive — buzz-acp → codex-acp → codex app-server — CPU ~0, no agent_returned, no error, no restart, last_error empty.
• The agent's channel is blocked until a manual restart; nothing surfaced anywhere. Later mentions get the 👀 ack but no reply (matches the withheld-steer path in the writeup above).
• Recurs every few hours, unattended.

So it's not Windows-specific.

Not idle-related (worth noting, since it's tempting to assume): the longest idle gap we saw — ~8.7h — replied fine in ~32s, while turns wedged after only a few-second gap. So it's tied to the state of a long-lived session, not how long it sat.

What codex's logs show around the hangs (offered as a lead, not a confirmed cause): the model-response stream drops mid-turn — stream disconnected before completion: Connection reset without closing handshake — and separately an idle timeout waiting for SSE line appears but did not recover the turn within an ~18-min window. Our reading is that when the model stream half-drops, codex's disconnect-triggered retry doesn't fire and the ACP session/prompt never returns, so the buzz in-flight turn hangs exactly as described here. We're not certain of the transport-level specifics, so treating that as a hypothesis.

Notably only long-lived automated consumers seem to hit it: codex exec is one-shot (fresh stream per call, ~3s, exits — nothing accumulates) and interactive users interrupt a stall in seconds. A bridge holding a persistent per-channel session for hours with no human is the consumer that just sits in the stall.

Two buzz-acp-side suggestions regardless of the upstream piece:

1. A silent in-flight turn should be hard-cancellable by max_turn_duration / an idle-output timeout. Per the mechanism in the writeup, a successful steer renews the deadline and a fully-silent turn never trips it, so recovery is manual-only. A "no streamed output for N s → cancel + reset session" watchdog would catch this whole class.
2. Force-restarting a wedged agent by killing only the buzz-acp PID can orphan the codex subprocess tree (adapter + app-server keep running) — a process-group kill is cleaner.

Possibly the same half-open-stream class as #3224 (desktop has relayStallWatchdog.ts; the headless path doesn't). We're mitigating locally for now with an external watchdog that detects an unanswered mention and restarts the wedged agent. Happy to share the full log excerpt + a spindump of the wedged process tree if useful.

[jdrolls]

Third platform (Linux), plus a trigger path that sits outside the read loop

Adding a Linux data point and two mechanism details that I think widen this past the steer-renewal path in the writeup.

Incident. Headless buzz-acp on Linux, --agents 1, wedged for 46 hours — 23x the configured --max-turn-duration. The 900s idle timeout never fired either. Process tree alive, CPU ~0, no agent_returned, no error, systemctl reporting the unit healthy throughout. Same signature as both reports above.

The wedge was not inside the read loop. The mechanism section here locates the hang in the prompt read loop no longer selecting the steer arm. In our case the future was parked in an await that runs between selects — relay/REST posting and agent stdin writes in run_prompt_task, outside read_until_response_with_idle_timeout entirely.

That distinction matters because it makes the deadline structurally unreachable rather than merely un-acked. Both the idle and hard deadlines are only ever evaluated at the top of that loop (the pre-select check and the select! arms in acp.rs). If the future never re-enters the loop, neither deadline can fire regardless of steer state. So "a successful steer renews the deadline" is one way in, and "the loop is never re-entered" is another — a cap keyed off steer/ack behaviour alone would not catch the second.

Slot starvation, not only channel blocking. In-flight expiry in queue.rs removes the channel from in_flight_channels / in_flight_deadlines and recovers withheld steer events, but it never touches the JoinSet task or the pool slot. With --agents 1 the wedged turn therefore holds the entire harness capacity indefinitely: the queue's view of the channel unblocks while no agent can ever be dispatched again.

Why the agent renders healthy — turn_liveness actively sustains it. run_turn_liveness (pool.rs) is a fixed-interval tokio::time::interval ticker with no progress input. It emits a turn_liveness observer frame every turn_liveness_secs (default 10) for as long as the task is alive, stopping only when LivenessGuard drops. Its documented purpose is detecting a host that died without unwinding (kill -9 / crash) — the opposite failure mode. During a wedge the task is very much alive, so liveness keeps ticking and the turn keeps reporting healthy. I believe that is the concrete reason for the "green, zero output, indistinguishable from idle" symptom in both reports.

Fix — offering a PR. I implemented the "hard cap that cannot be extended by steers alone" from the Expected behavior list, and have been running it in production:

• An independent per-turn watchdog task, armed at slot checkout for max_turn_duration + 35s grace. It is absolute — computed at turn start and never renewed, so extend_in_flight_deadline cannot push it out — and trigger-agnostic, so it fires regardless of where the future is stuck.
• On expiry it aborts the prompt task via its AbortHandle. The abort drops OwnedAgent, whose AcpClient::Drop kills the dedicated child process group — which also covers the orphaned codex-acp / app-server tree noted in the comment above.
• The JoinSet cancellation is reconciled through the existing recover_panicked_agent path, which releases the queue in-flight marker and schedules a replacement for the slot. Withheld-event recovery reuses the existing machinery rather than adding a second path.
• The 35s grace sits deliberately above the existing control-cancel drain, so the watchdog only fires when the normal HardTimeout -> cancel_with_cleanup path is itself wedged.

Three tests, including one that drives abort -> JoinError::is_cancelled -> recover_panicked_agent and asserts both the in-flight marker release and the slot respawn. cargo test -p buzz-acp: 664 passed, 0 failed. clippy clean. I searched open PRs and found none covering this — happy to be pointed at one if I missed it.

Two questions before I send it, since both are maintainer calls:

1. Should the grace interval be configurable rather than a constant?
2. An actively-steered long turn is now killed at the cap. That is what the Expected behavior asks for, but it is a behaviour change for anyone currently relying on steer renewal to extend a legitimate turn — worth gating behind a flag?

To be clear on scope: this is the buzz-acp-side backstop only. It does not address the upstream codex stream half-drop hypothesis, it just stops that class from wedging a slot indefinitely.

[devdaalper]

Same failure on macOS with the Claude Code harness — reproduced twice across two versions

Adding a data point from a different platform and harness than the reports above, in case it helps narrow the cause.

Environment

• Buzz Desktop 0.5.5, then reproduced on 0.5.7
• macOS 26.5.2, Apple Silicon (M4 Max)
• Agent: built-in Honey persona, harness claude-agent-acp, default model
• Claude Code CLI 2.1.109, authenticated, subscriptionType: max
• turn_timeout_seconds: 320, parallelism: 10, respond_to: owner-only

What happened

Both times the trigger was an @mention placed inside a thread rather than at channel level.

1. 2026-08-08T00:40:19Z (Buzz 0.5.5) — agent started, reacted to the mention with 👀 and 💬, then produced nothing. Still running 8 minutes later.
2. 2026-08-08T00:55:58Z (Buzz 0.5.7) — identical behaviour after a clean retry with a fresh thread and a fresh prompt. No reaction this time, no reply, 12 minutes.

In both cases the turn ran well past turn_timeout_seconds: 320 without aborting. Matching the report above, managed-agents.json showed no signal of trouble: last_error: null, last_error_code: null, last_exit_code: null, last_stopped_at: null, and last_started_at correctly updated to the moment of the mention. Ten claude-agent-acp node processes plus one live claude CLI process, all alive, no CPU activity.

Channel-level blocking confirmed

While the turn was hung I sent a deliberately trivial control mention to the same agent at channel level, outside any thread (@Honey control: reply with the single word VIVO). It also never got a response, which is consistent with the withheld-mention behaviour described in this issue: the hung turn blocks the agent for the whole channel, not just the thread where it stalled.

Worth stressing for anyone triaging: from the UI this is indistinguishable from a slow agent. Nothing anywhere indicates the message will never be answered.

Recovery

"Stop running agents" in the Desktop Agents panel recovered cleanly — last_exit_code: 0, all claude-agent-acp and claude processes gone. Consistent with manual restart being the only exit.

Possibly related, possibly separate

During the same session the client repeatedly showed Some message context could not be loaded on the thread panel, and one channel message appeared to fail to send, then showed up in the channel several minutes later. I have not established whether that shares a root cause with the hung turn or is an independent relay-sync issue, so flagging it only as an observation.

[matthewdonsemail-lab]

Watchdog scope addition: upstream 400 — reasoning_content not passed back in thinking mode (field report, 2026-08-08, InferenceSaver deployment)

Observed intermittently: a turn through the Buzz ACP harness driving a thinking-mode model fails upstream with:

status_code=400, The reasoning_content in the thinking mode must be passed back to the API. (request id: 01KZGP8DSH6JG7G84S5F4094QR)

• The 400 is emitted by the upstream model router (NewAPI) — i.e. the harness did not pass the prior reasoning_content back on the follow-up request, so the thinking-mode model rejects the call. Reported as "occasionally the reasoning isn't being passed through via buzz", trigger unknown.
• Fits this issue's family: agent process stays healthy, no useful error is surfaced to the operator/UI, and the turn is effectively lost — the exact "no watchdog" gap described above. Related symptom class seen the same week: agent turns failing with a generic AI-gateway 422 that the harness just retries, with the underlying upstream error swallowed (InferenceSaver normalizes NewAPI errors to a generic gateway message in lib/newapi/errors.ts, repo matthewdonsemail-lab/inferencesaver @ 2049c843).

Requirement for the watchdog:

• Watch for this error signature — a 400/422 whose body matches reasoning_content ... must be passed back (or similar thinking-mode state error) — and treat it as a turn-failure signal, not a silent no-op.
• On detection, at minimum surface the raw upstream error + request id to the Desktop UI (the agent currently shows healthy). Ideally recover: retry without thinking mode, or pass the stored reasoning_content back on the follow-up request.

Root cause is a hypothesis until traced in buzz-acp: the harness/provider adapter may be dropping reasoning_content between assistant turns, and there is no error path that propagates this upstream 400 to the operator.