Flaky E2E: multimodal + approval-flow tests intermittently get empty responses (`chatStream fatal`) from aimock

[tombeckenham]

Summary

The E2E suite is intermittently red on unrelated PRs. The failures cluster in the multimodal and approval-flow specs and are non-deterministic — the same commit both passes and fails across re-runs, so this is CI flakiness, not a code regression.

Evidence it's flaky (not a regression)

On branch 705-… the identical commit 1720516 produced 3 passing E2E runs and 1 failing run. The parent commit 2d85506 passed all 4 of its runs. Same code, different outcomes.

It's not branch-specific — recent E2E failures span multiple unrelated branches interleaved with successes:

failure | 705-…grok-video…
success | 707-…openrouter-video…
success | 705-…grok-video…        ← same branch, passes
failure | gemini-upgrade
success | gemini-upgrade           ← same branch, passes
failure | renovate/all-minor-patch

Affected tests

• tests/multimodal-structured.spec.ts — <provider> — multimodal-structured › analyzes an image and returns structured output (seen on openai, anthropic)
• tests/multimodal-image.spec.ts — <provider> — multimodal-image › describes an uploaded image (seen on gemini, openrouter)
• tests/tools-test/approval-flow.spec.ts — approval then client tool
• Occasionally tests/tts.spec.ts — grok -- tts › http-stream (one-off, recovered on retry)

Most failures self-recover on Playwright retry (anthropic / gemini / openrouter passed retry #1 in the failing run); occasionally one provider exhausts all retries in a given run (e.g. openai — multimodal-structured).

Symptom

The web server logs a fatal during the streamed chat call and the assistant message comes back empty:

[WebServer] ❌ [tanstack-ai:errors] ❌ anthropic.chatStream fatal
[WebServer] ❌ [tanstack-ai:errors] ❌ gemini.chatStream fatal

Error: expect(received).toContain(expected)
  Expected substring: "guitar"
  Received string:    ""

The empty "" body across several different providers points at the shared layer — the aimock server's streamed response for the image-upload (multimodal) request path — rather than any one adapter. The multimodal specs sendMessageWithImage(...) then assert the model echoes content from test-assets/guitar-shop.png; intermittently aimock returns nothing and the stream errors.

Suspected cause

Non-deterministic aimock behavior for multipart/multimodal requests under CI load (intermittent empty/aborted stream), surfacing as chatStream fatal. The approval-flow approval then client tool flake is likely the same streaming nondeterminism on a multi-step run.

Possible remediations (for discussion)

• Reproduce locally with repeats (--repeat-each=20) on the multimodal specs to characterize the empty-stream condition.
• Check whether aimock emits a usable response for the image-upload request path deterministically; if it's an aimock bug, file upstream (CopilotKit/aimock) with a minimal repro.
• If it's a race in our harness (response read before the stream settles), harden waitForResponse / the empty-body assertion to fail with the captured chatStream fatal detail instead of a bare empty string, to make future triage faster.
• As an interim, these specs already rely on Playwright retries — confirm retry counts are adequate, but treat retries as a stopgap, not the fix.

Acceptance

• Root cause identified (aimock vs harness race).
• Multimodal + approval-flow specs run green deterministically (e.g. 20× repeat with no failures), or the flaky source is fixed upstream and pinned.