Skip to content

fix(buzz-agent): recover from 400-shaped image rejections; unbound benchmark agent rounds - #5318

Merged
tlongwell-block merged 1 commit into
mainfrom
eva/vision-400-context-poison
Aug 8, 2026
Merged

fix(buzz-agent): recover from 400-shaped image rejections; unbound benchmark agent rounds#5318
tlongwell-block merged 1 commit into
mainfrom
eva/vision-400-context-poison

Conversation

@tlongwell-block

Copy link
Copy Markdown
Collaborator

Problem

Two failure modes from the tb21-glm52-crusoe-1 benchmark run (GLM-5.2 solo, TB2.1) wedged or killed 13 of 89 trials without the model being at fault:

  1. Conversation poisoning on text-only endpoints. Crusoe's serverless crusoeai/GLM-5.2-NVFP4 rejects any request whose history contains an image with 400: ... is not a multimodal model. The recovery machinery for exactly this case already exists — AgentError::UnsupportedImageInputreplace_unsupported_images() strips the image blocks, marks the tool result as an error, substitutes a text placeholder, and continues the turn. But classification only matched OpenRouter's 404 body (no endpoints found that support image input) and was only consulted on the 404 arms. The Crusoe 400 fell through to terminal AgentError::Llm: the image stayed in history, every subsequent call failed identically, buzz-acp rode its 10-retry ladder (~40 min), and the trial idled to budget death. Measured blast radius: 8 trials wedged, 12.7h aggregate idle-after-poison.

  2. Bounded agent rounds in benchmark trials. The harness default DEFAULT_MAX_AGENT_ROUNDS = 32 ended solo trials mid-work when turns rotated (thinking-heavy models hit max_tokens rotation fast; 4 trials died this way). Benchmark trials already have a wall-clock budget as the real limit — the round cap only converts recoverable rotation into trial death.

Fix

  • is_unsupported_image_input_error() also matches the verbatim is not a multimodal model body. Matcher stays deliberately tight (same doctrine as is_context_length_error): misclassifying a generic 400 as recoverable would mutate history for an error that removing images cannot fix.
  • Both status ladders — shared post() and openrouter_post() — consult it on their 400 arms and return the typed UnsupportedImageInput (OpenAI-compatible providers report this as 400; a BYOK/passthrough upstream can surface the provider's own 400 through OpenRouter).
  • Harness DEFAULT_MAX_AGENT_ROUNDS0 (unbounded — BUZZ_AGENT_MAX_ROUNDS=0 is the agent config's documented unbounded value). Per-agent budget.max_calls in manifests still overrides.

Acceptance

  • A 400 with the image-rejection body reaches the existing image-strip recovery path instead of wedging the session — asserted through complete() (covers the return path into the convergence mapper) and at the openrouter_post terminal, both proving single-attempt (a deterministic capability rejection must never be retried).
  • Ordinary 400s stay terminal AgentError::Llm (existing negative tests unchanged).
  • Benchmark trials run unbounded rounds by default; python tests updated for 0-is-legal with a negative arm at -1.

Verification

  • cargo test -p buzz-agent: 427 + 18 + 20 + 15 + 8 + 1 + 48 passed, 0 failed (full package, 3 consecutive clean runs)
  • cargo clippy -p buzz-agent --all-targets, cargo fmt --check: clean
  • uv run --extra dev pytest tests/ in harbor-buzz-orchestra: 35 passed
  • Pre-push hooks (full workspace rust-tests + desktop-tauri-checks) green on rustc 1.95.0 at head b043860

…nchmark agent rounds

Two failure modes from the tb21-glm52-crusoe-1 benchmark run wedged or
killed 13 of 89 trials without the model being at fault.

**Conversation poisoning on text-only endpoints.** Crusoe's serverless
`crusoeai/GLM-5.2-NVFP4` rejects any request whose history contains an
image with `400: ... is not a multimodal model`. The recovery path for
exactly this situation already exists (`AgentError::UnsupportedImageInput`
→ `replace_unsupported_images` strips the image blocks and continues the
turn with a text placeholder), but classification only matched
OpenRouter's 404 body ("no endpoints found that support image input") and
was only consulted on the 404 arms. The Crusoe 400 fell through to
terminal `AgentError::Llm`; the image stayed in history, every subsequent
call failed identically, buzz-acp rode its 10-retry ladder (~40 min), and
the trial idled to budget death. Measured: 8 trials, 12.7h aggregate idle.

Fix: extend `is_unsupported_image_input_error()` to also match the
verbatim "is not a multimodal model" body, and consult it on the 400 arms
of both the shared `post()` ladder and `openrouter_post()` (a BYOK/
passthrough upstream can surface the provider's own 400). The matcher
stays deliberately tight, same doctrine as `is_context_length_error`:
misclassifying a generic 400 as recoverable would mutate history for an
error that removing images cannot fix.

**Bounded agent rounds in benchmark trials.** The harness default
`DEFAULT_MAX_AGENT_ROUNDS = 32` ended solo trials mid-work when a turn
rotated (max_tokens rotation on thinking-heavy models burned rounds
fast). Benchmark trials already have a wall-clock budget as the real
limit, so the round cap only converts recoverable rotation into trial
death. Default is now 0 (unbounded — `BUZZ_AGENT_MAX_ROUNDS=0` is the
agent's own documented unbounded value); per-agent `budget.max_calls`
in manifests still overrides.

Tests: 400-body classification asserted through `complete()` on the
shared-post path and at the `openrouter_post` terminal, both proving
single-attempt (no retry of a deterministic capability rejection);
python tests updated for 0-is-legal with a negative arm at -1.

Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
@tlongwell-block
tlongwell-block requested a review from a team as a code owner August 8, 2026 13:18
@tlongwell-block
tlongwell-block merged commit 261c460 into main Aug 8, 2026
34 checks passed
@tlongwell-block
tlongwell-block deleted the eva/vision-400-context-poison branch August 8, 2026 16:17
tlongwell-block pushed a commit that referenced this pull request Aug 8, 2026
…ounting-and-solo

* origin/main:
  chore(release): release Buzz Desktop version 0.5.8 (#5326)
  fix(buzz-agent): recover from 400-shaped image rejections; unbound benchmark agent rounds (#5318)
  Revert "fix(acp): reject unattended permission requests" (#5323)
  feat(desktop): unify add agent flows (#5015)

Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
wpfleger96 pushed a commit that referenced this pull request Aug 8, 2026
* origin/main:
  chore(release): release Buzz Desktop version 0.5.8 (#5326)
  fix(buzz-agent): recover from 400-shaped image rejections; unbound benchmark agent rounds (#5318)
  Revert "fix(acp): reject unattended permission requests" (#5323)
  feat(desktop): unify add agent flows (#5015)
  fix(buzz-agent): budget summarizer reasoning separately so it cannot starve the handoff summary (#5248)
  infra: bind development services to loopback (#4871)

Signed-off-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>

# Conflicts:
#	CHANGELOG.md
wpfleger96 added a commit that referenced this pull request Aug 8, 2026
* origin/main:
  fix(desktop): welcome banner overlap and missing dismiss control (#5330)
  fix(desktop): prevent horizontal clipping in Prompt Context modal (#5324)
  chore(release): release Buzz Relay version 0.2.1 (#2856)
  chore(release): release Buzz Desktop version 0.5.8 (#5326)
  fix(buzz-agent): recover from 400-shaped image rejections; unbound benchmark agent rounds (#5318)
  Revert "fix(acp): reject unattended permission requests" (#5323)
  feat(desktop): unify add agent flows (#5015)
  fix(buzz-agent): budget summarizer reasoning separately so it cannot starve the handoff summary (#5248)
  infra: bind development services to loopback (#4871)
  chore(release): release Buzz Desktop version 0.5.7 (#5252)
  fix(desktop): isolate relay admission tests (#5221)
  fix(desktop): externalize boot <style> to prevent Tauri CSP nonce override (#5242)
  fix(desktop): let imported and recovered identities finish onboarding (#5228)

Co-authored-by: Will Pfleger <pfleger.will@gmail.com>
Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
tlongwell-block pushed a commit that referenced this pull request Aug 8, 2026
Two-seat twin roster smoked green on hello-world (reward 1.0) and now
running the full TB2.1 89-task bench as tb21-twins-1. Mia holds the conn
(orchestrator seat — the harness watches that pubkey for DONE:), Mie is
the delegable worker twin. Both personas are the Meli solo prompt plus
twin team sections, byte-identical to each other except the identity
line and the conn/worker paragraph.

Contract: thread-of-record comms (--reply-to the task post), both keep
the todo tool loaded, Mia never ends her turn while the task is
incomplete (sleep-30 poll loop; requires unbounded agent rounds from
#5318), Mie ends every turn with a report mentioning Mia, only Mia sends
DONE:. Model: deepseek-v4-flash via OpenRouter pinned cloudflare/fp8 —
the existing openrouter-cloudflare.json endpoint config, unchanged.

Manifest persona sha256 pins match the committed files.

Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant