Skip to content

fix(agent): retry LLM completion on malformed 2xx JSON body - #5351

Merged
tlongwell-block merged 1 commit into
mainfrom
eva/retry-malformed-llm-json
Aug 8, 2026
Merged

fix(agent): retry LLM completion on malformed 2xx JSON body#5351
tlongwell-block merged 1 commit into
mainfrom
eva/retry-malformed-llm-json

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Problem

A provider can return HTTP 200 with a truncated JSON body — cleanly closed connection, correct framing, content cut off mid-value. Both LLM HTTP loops treated this as a terminal error on the first attempt: AgentError::Llm("json: EOF while parsing a value"), surfaced as code -32000 at the ACP boundary, killing the agent turn before it produced anything.

Observed live in a tb2.1 bench trial (write-compressor, tb21-twins-1): deepseek via OpenRouter returned a truncated body, the agent died mid-prompt with 0 turns completed, and the trial scored 0 on a provider hiccup.

Meanwhile the same loops already retry timeouts, 429s, 5xxs, 499s, and mid-body stream stalls — a truncated-but-complete body was the one transient upstream fault that fell through to terminal.

Fix

In both post() and openrouter_post() (crates/buzz-agent/src/llm.rs): when the fully-received success body fails serde_json::from_slice, continue the existing retry loop instead of returning terminal — same MAX_RETRIES (3) bound, same backoff_with_jitter. On exhaustion, the error goes through terminal_llm_error so it carries cumulative duration + attempt count like every other retried failure (previously the json: error carried neither).

post_anthropic routes through post(), so Anthropic/OpenAI/Databricks/mesh and OpenRouter are all covered.

Why this cannot re-run a tool call

Hard requirement: tool calls are not idempotent, and this change must not introduce any possibility of replaying one.

  1. The retry lives inside the HTTP POST helper, below the parse boundary. Tool calls are only ever extracted from a successfully parsed response value (parse_openai/parse_anthropic/parse_responses, all downstream of these helpers' Ok return). A malformed body never parses, therefore no tool call was ever extracted from it, therefore nothing downstream of it ever dispatched.
  2. What is re-sent is the completion request itself — the identical body_bytes captured once at function entry. Sending a completion request executes no tools; it asks the model for the next message.
  3. Same safety class as existing behavior. The loop already re-sends this identical request on 429/5xx/timeout/stream-stall; this adds one more transient-fault arm to the same loop with the same bytes.

Tests

Three new tests mirroring the existing 499/dropped-connection fixtures (raw TcpListener stubs):

  • post_retries_malformed_json_body_and_succeeds — truncated 200 body on attempt 1, valid JSON on attempt 2; asserts success and exactly 2 server-side requests
  • post_exhausts_retries_on_persistent_malformed_json — always-truncated body; asserts exactly MAX_RETRIES attempts and a terminal error carrying json: + cumulative/attempt context
  • openrouter_post_retries_malformed_json_body_and_succeeds — same recovery through OpenRouter's separate loop

Full cargo test -p buzz-agent green at e7a5d7b (430 lib + all integration targets, 0 failures); cargo fmt + clippy --all-targets clean.

Originating conversation: buzz-benchmarking channel, thread 397a992d.

A provider that returns HTTP 200 with a truncated JSON document (cleanly
closed, framing intact, content cut off mid-value) previously died on the
first attempt with a terminal AgentError::Llm("json: EOF while parsing a
value"), killing the agent turn before it produced anything. Observed
live: deepseek via OpenRouter returned a truncated body in a tb21 bench
trial and the agent exited with 0 turns (code -32000 at the ACP surface).

Treat an unparseable success body like a 5xx in both HTTP loops (shared
post() and openrouter_post()): retry under the existing MAX_RETRIES bound
with the same backoff_with_jitter, and surface terminal_llm_error with
the json parse detail when retries are exhausted.

Tool-safety: the retry lives entirely inside the HTTP POST helpers,
before any parsed response value exists. Tool calls are only ever
extracted from a successfully parsed response; a malformed body produced
none, so nothing downstream of it could have dispatched. The retry
re-sends the identical request bytes captured at function entry — an LLM
completion request, which executes no tools. No tool call can be
re-run by this change.

Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 8, 2026 21:14
@tlongwell-block
tlongwell-block merged commit 5bf7867 into main Aug 8, 2026
27 checks passed
@tlongwell-block
tlongwell-block deleted the eva/retry-malformed-llm-json branch August 8, 2026 21:29
tlongwell-block pushed a commit that referenced this pull request Aug 9, 2026
…ounting-and-solo

* origin/main:
  fix(agent): retry LLM completion on malformed 2xx JSON body (#5351)
  fix(desktop): welcome banner overlap and missing dismiss control (#5330)
  fix(desktop): prevent horizontal clipping in Prompt Context modal (#5324)
  chore(release): release Buzz Relay version 0.2.1 (#2856)

Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant