Symptom
server/tests/unit/addie-router.test.ts > AddieRouter.route (LLM) > prod: channel messages Addie should IGNORE > should ignore opinion requests failed once during local CI rehearsal for #3095, passed on retry.
Expected: \"ignore\"
Received: \"respond\"
❯ server/tests/unit/addie-router.test.ts:787:27
Why it's flaky
The test calls Claude Haiku live (claude-haiku-4-5, ~1.6s latency, ~95 output tokens per the run logs). LLM responses on the boundary cases ("opinion requests" vs "ignore") aren't deterministic — the model can land on either side depending on sampling.
Impact
Probabilistic CI failures on PRs that don't touch this code path. Each failure costs a rerun-the-job loop or, worse, a real bug investigation that lands on "oh, it's the LLM test again."
Possible mitigations (pick one)
- Tighten the prompt. If the test is asserting on a true edge case, the prompt that drives the routing may be ambiguous — fix the prompt so both humans and Haiku agree, then assert determinism.
- Move to a recorded fixture. Snapshot a known-good Haiku response for this input and replay it. Loses live model coverage but eliminates flakes. Same pattern other LLM tests in the repo may already use.
- Tag and quarantine. Mark the LLM-driven "prod: channel messages" describe block as
test.concurrent.skipIf(!process.env.RUN_LLM_TESTS) so it only runs in a nightly job, not on every PR.
- Retry once on failure. Cheap but hides real regressions. Last resort.
Repro: re-run the test enough times. The same input returned both "ignore" and "respond" within minutes during the #3095 rehearsal — passing on the second attempt I made.
Not blocking
Filing for visibility. Anyone who hits this on their own PR can rerun and ignore.
Symptom
server/tests/unit/addie-router.test.ts > AddieRouter.route (LLM) > prod: channel messages Addie should IGNORE > should ignore opinion requestsfailed once during local CI rehearsal for #3095, passed on retry.Why it's flaky
The test calls Claude Haiku live (
claude-haiku-4-5, ~1.6s latency, ~95 output tokens per the run logs). LLM responses on the boundary cases ("opinion requests" vs "ignore") aren't deterministic — the model can land on either side depending on sampling.Impact
Probabilistic CI failures on PRs that don't touch this code path. Each failure costs a rerun-the-job loop or, worse, a real bug investigation that lands on "oh, it's the LLM test again."
Possible mitigations (pick one)
test.concurrent.skipIf(!process.env.RUN_LLM_TESTS)so it only runs in a nightly job, not on every PR.Repro: re-run the test enough times. The same input returned both "ignore" and "respond" within minutes during the #3095 rehearsal — passing on the second attempt I made.
Not blocking
Filing for visibility. Anyone who hits this on their own PR can rerun and ignore.