fix(agent): retry LLM completion on malformed 2xx JSON body - #5351
Merged
Conversation
A provider that returns HTTP 200 with a truncated JSON document (cleanly
closed, framing intact, content cut off mid-value) previously died on the
first attempt with a terminal AgentError::Llm("json: EOF while parsing a
value"), killing the agent turn before it produced anything. Observed
live: deepseek via OpenRouter returned a truncated body in a tb21 bench
trial and the agent exited with 0 turns (code -32000 at the ACP surface).
Treat an unparseable success body like a 5xx in both HTTP loops (shared
post() and openrouter_post()): retry under the existing MAX_RETRIES bound
with the same backoff_with_jitter, and surface terminal_llm_error with
the json parse detail when retries are exhausted.
Tool-safety: the retry lives entirely inside the HTTP POST helpers,
before any parsed response value exists. Tool calls are only ever
extracted from a successfully parsed response; a malformed body produced
none, so nothing downstream of it could have dispatched. The retry
re-sends the identical request bytes captured at function entry — an LLM
completion request, which executes no tools. No tool call can be
re-run by this change.
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
tlongwell-block
pushed a commit
that referenced
this pull request
Aug 9, 2026
…ounting-and-solo * origin/main: fix(agent): retry LLM completion on malformed 2xx JSON body (#5351) fix(desktop): welcome banner overlap and missing dismiss control (#5330) fix(desktop): prevent horizontal clipping in Prompt Context modal (#5324) chore(release): release Buzz Relay version 0.2.1 (#2856) Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A provider can return HTTP 200 with a truncated JSON body — cleanly closed connection, correct framing, content cut off mid-value. Both LLM HTTP loops treated this as a terminal error on the first attempt:
AgentError::Llm("json: EOF while parsing a value"), surfaced as code -32000 at the ACP boundary, killing the agent turn before it produced anything.Observed live in a tb2.1 bench trial (write-compressor, tb21-twins-1): deepseek via OpenRouter returned a truncated body, the agent died mid-prompt with 0 turns completed, and the trial scored 0 on a provider hiccup.
Meanwhile the same loops already retry timeouts, 429s, 5xxs, 499s, and mid-body stream stalls — a truncated-but-complete body was the one transient upstream fault that fell through to terminal.
Fix
In both
post()andopenrouter_post()(crates/buzz-agent/src/llm.rs): when the fully-received success body failsserde_json::from_slice,continuethe existing retry loop instead of returning terminal — sameMAX_RETRIES(3) bound, samebackoff_with_jitter. On exhaustion, the error goes throughterminal_llm_errorso it carries cumulative duration + attempt count like every other retried failure (previously thejson:error carried neither).post_anthropicroutes throughpost(), so Anthropic/OpenAI/Databricks/mesh and OpenRouter are all covered.Why this cannot re-run a tool call
Hard requirement: tool calls are not idempotent, and this change must not introduce any possibility of replaying one.
parse_openai/parse_anthropic/parse_responses, all downstream of these helpers'Okreturn). A malformed body never parses, therefore no tool call was ever extracted from it, therefore nothing downstream of it ever dispatched.body_bytescaptured once at function entry. Sending a completion request executes no tools; it asks the model for the next message.Tests
Three new tests mirroring the existing 499/dropped-connection fixtures (raw
TcpListenerstubs):post_retries_malformed_json_body_and_succeeds— truncated 200 body on attempt 1, valid JSON on attempt 2; asserts success and exactly 2 server-side requestspost_exhausts_retries_on_persistent_malformed_json— always-truncated body; asserts exactlyMAX_RETRIESattempts and a terminal error carryingjson:+ cumulative/attempt contextopenrouter_post_retries_malformed_json_body_and_succeeds— same recovery through OpenRouter's separate loopFull
cargo test -p buzz-agentgreen at e7a5d7b (430 lib + all integration targets, 0 failures);cargo fmt+clippy --all-targetsclean.Originating conversation: buzz-benchmarking channel, thread 397a992d.