fix(addie): stabilise opinion-poll channel-router test#3103
Merged
Conversation
Two changes together eliminate the 'should ignore opinion requests' flake:
1. Replace the live-API assertion with a vi.spyOn fixture — the input
('What do you all think about the new IAB guidelines for CTV
measurement?') sits on a genuine semantic boundary where Haiku is
non-deterministic at default sampling temperature. The fixture asserts
the routing logic (parseRouterResponse + channel policy filter) without
depending on model output.
2. Add an opinion-poll example to the channel-ignore guidelines in the
routing prompt, with a carve-out preserving responses to AdCP-specific
protocol questions, so production routing is more consistent on this
class of input.
Non-breaking: test-only change + prompt clarification; no schema or API impact.
https://claude.ai/code/session_017BFqhixgiLh2kjYMAzs4d2
bokelley
force-pushed
the
claude/issue-3101-fix-opinion-poll-flaky-test
branch
from
ff565af to
0e9ce0c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3101
Summary
Two changes together eliminate the
'should ignore opinion requests'flake inserver/tests/unit/addie-router.test.ts:Recorded fixture — replaces the live-API assertion with a
vi.spyOnstub. The input ('What do you all think about the new IAB guidelines for CTV measurement?') sits on a semantic boundary (opinion poll, but IAB/CTV nouns are ad-tech adjacent) where Haiku is non-deterministic at default sampling temperature. Settingtemperature: 0was explored but the Anthropic SDK deprecates it for all models released after Opus 4.6 (including Haiku 4.5) — any non-1.0 value returns a 400. The fixture stubsclient.messages.createonce-per-test and asserts the routing logic (parseRouterResponse+ channel-policy filter) without depending on live model output.Prompt clarification — adds an opinion-poll example to the
## Messages to Ignorelist and theCRITICAL — CHANNEL SOURCEblock, with an explicit carve-out preserving responses to AdCP-specific protocol questions. This makes the production routing more consistent on this class of input without over-suppressing legitimate "what does everyone think about the 3.0 schema?" questions.Non-breaking justification: test-only change (
vi.spyOnfixture) + prompt clarification with no semantic change to the ignore/respond binary for well-formed inputs; no schema, API, or protocol impact.Pre-PR review
temperature: 0as a blocker — correctly identified the Anthropic SDK deprecation; also flagged missing changeset — both resolved)Nits noted but not fixed
thirdPersonAddieregex misses some third-person phrasings (pre-existing, unrelated to this fix)Session: https://claude.ai/code/session_017BFqhixgiLh2kjYMAzs4d2
Generated by Claude Code