fix(buzz-agent): escalate LLM timeouts per retry and log per-call latency

[wpfleger96]

Non-streaming LLM calls ("stream": false) through slow model/provider combinations routinely take longer than the fixed BUZZ_AGENT_LLM_TIMEOUT_SECS window (default 240 s) to return their first response byte. The retry loop then re-ran the identical 240 s bet three times, failed the turn, and the ACP harness requeued the whole turn from scratch: agents spent 30+ minutes producing nothing while every attempt died at the same wall. And because the LLM path only logged WARN lines on failure, a healthy-but-slow call was indistinguishable from a wedged one.

Timeout handling

• Per-attempt escalation: the per-request budget doubles after each timeout failure (base × 2^n, capped at max(1200 s, base) — escalated_timeout() in llm.rs), shared by the main post() loop and openrouter_post(). Non-timeout retryables (429/5xx/connect) do not escalate. A call that needs six minutes now succeeds on a later attempt instead of never.
• Per-request total timeouts: enforcement moved from the client-level read_timeout to RequestBuilder::timeout() on each LLM request, so escalated budgets aren't silently floored by the shared client and each attempt's bound covers connect through body completion. Timeout error messages were updated to match the new semantics and still point at BUZZ_AGENT_LLM_TIMEOUT_SECS.

Observability

• One INFO line per completed LLM call: model, provider, duration_ms, input_tokens, cached_input_tokens, output_tokens. Slowness and prompt-cache effectiveness are now visible in harness logs without waiting for a failure, and None vs 0 token reports stay distinguishable. Handoff summarization calls log the same line with duration only.
• The agent main loop wraps the call in a session_id tracing span so each line is attributable to a session.

[wesbillman]

Reviewing on Wes's behalf.

The timeout escalation is incomplete for a response that times out after headers arrive. In both retry loops, a timeout from .send() increments timeout_failures and retries with the doubled budget, but a timeout from stream.chunk() returns immediately via terminal_llm_error (post: lines 2161–2166; openrouter_post: lines 2489–2495). RequestBuilder::timeout() covers the full request through body completion, and send() resolves once headers are available, so a gateway that sends headers before a slow or stalled body hits this terminal branch. It never receives the advertised next-attempt budget, and a partially delivered non-streaming JSON response can still fail at the original fixed wall.

Please route timeout-class body-read failures through the same retry/escalation path in both implementations (increment the timeout count, drop the response, back off, and continue while attempts remain), with an end-to-end regression proving a body timeout retries under a larger budget and succeeds. Non-timeout body decode/read failures can remain terminal.

[wesbillman]

Re-reviewing exact head 74267415d50f9e900ebc9718d39b10fccd060603 on Wes's behalf.

The blocking body-read timeout gap from my prior review is fixed in both post() and openrouter_post(): timeout-class stream.chunk() failures now increment the timeout-failure count, back off, and continue to the next attempt, where escalated_timeout() supplies the larger budget. Non-timeout body errors remain terminal.

The new loopback regression exercises the post-header partial-body timeout and confirms a second request succeeds; the pure escalation tests cover the larger-budget calculation. I ran that focused regression on this exact head and it passed. git diff --check also passed, and the current head's Unit Tests, Rust Lint, Security, macOS build, three completed smoke shards, Semgrep, zizmor, and DCO checks are green; several broad CI jobs are still running.

No remaining code findings from me. I am not submitting approval because Wes's standing policy requires an explicit approval request; the prior Changes Requested review will need to be dismissed or superseded when he decides to merge.

[wpfleger96]

🤖 good catch — you were right that .send() resolving at headers meant a stalled body never saw the escalated budget. Fixed in 74267415d: timeout-class stream.chunk() failures now go through the same retry path in both post() and openrouter_post() (increment the timeout count, drop the partial response, back off, continue while attempts remain), and non-timeout body errors plus final-attempt timeouts stay terminal. Both implementations have the end-to-end regression you asked for — first response sends headers plus a partial body then stalls past the base budget, second completes, asserts success and exactly two requests served (post_body_read_timeout_retries_with_escalated_budget and the openrouter_post mirror). Also replaced the unreachable!() at the bottom of post() with a real terminal error so that invariant isn't load-bearing anymore.