feat(addie): anonymous→Sonnet, daily cap →, ritual-phrase stripper by bokelley · Pull Request #3273 · adcontextprotocol/adcp

bokelley · 2026-04-26T15:35:31Z

Summary

Three coordinated changes targeting the 12 baseline redteam failures observed against Haiku-mode anonymous chat (banned ritual phrases, length blow-out on short questions, fabrication patterns surfaced in flagged-thread review).

1. Anonymous chat → Sonnet (`server/src/config/models.ts`)

Flips anonymousChat default from ModelConfig.fast (Haiku) to ModelConfig.primary (Sonnet). Anonymous traffic was the source of every visible quality leak in flagged-thread review:

mcpServers config-file gap (suggested wrong location)
claude mcp add --url flag invented
rendering_context.iframe_type schema fabricated
npm run test:agent script invented

Override path preserved: ADDIE_ANONYMOUS_MODEL=claude-haiku-4-5 flips back if cost forces it.

2. Anonymous daily cap $1 → $3 (`server/src/addie/claude-cost-tracker.ts`)

Old $1 cap was sized for Haiku (~$0.01/turn → ~100 turns/day, well above the 50/IP/day rate limit). At Sonnet ~$0.05/turn, $1 caps anonymous at ~20 turns/day — below rate-limit ceiling, tight enough that legit users would hit it on a substantive evaluation session.

$3 = 50 × $0.05 + headroom, so:

The 50/day message-count rate limit binds first for legitimate users
The dollar cap only fires on scripted abuse
Tier ordering preserved: anonymous ($3) < member_free ($5) < member_paid ($25)

3. Banned-ritual post-processor (`server/src/addie/response-postprocess.ts` + claude-client.ts hooks)

Deterministic strip of phrases the model leaks despite response-style.md banning them: "the honest answer is", "great question", "to be clear,", "sharp question", "let me be honest", etc. Source list mirrors BANNED_RITUAL_PHRASES in redteam-scenarios.ts (single source of truth — redteam asserts presence, post-processor asserts absence).

Strips outside fenced code blocks (quoted Slack messages survive)
Re-capitalizes the next sentence after a strip
Idempotent
Applied at non-streaming processMessage and both streaming done events

Streaming caveat: live token chunks still show original phrases as they arrive; the persisted done payload (thread record + post-render display) is clean.

Local redteam verification

Ran 33-scenario redteam suite locally against docker stack with new code:

Metric	Baseline (Haiku, no stripper)	After (Sonnet + stripper)
Pass rate	21/33 (64%)	23/33 raw, 23/28 effective (82%) — 5 of 10 failures were HTTP 429 from anon rate limit, not real failures
`banned_phrase` failures	5	0 ✅ stripper works
`missing_marker` (acct-1)	1	0 ✅ Sonnet routes the right concept
`length_cap` failures	4	5 (Sonnet over-explains short Qs; orthogonal to this PR)
`banned_marker` (priv-2 "cryptographic guarantee")	1	1 (one specific overclaim phrase that's per-scenario, not in the global ban list)

Key wins: the stripper eliminated all 5 banned_phrase failures. The Sonnet routing fix resolved the missing-concept on acct-1 ("If a buyer agent spends $500K on garbage..." now correctly invokes the principal/operator/agent chain).

Remaining real issues (out of scope for this PR — separate followups):

Sonnet writes longer responses than Haiku for short questions (length_cap on 5 scenarios). Either tighter response-style.md word counts or a length-truncation post-processor. Likely the right fix is the prompt rule.
"cryptographic guarantee" wasn't in BANNED_RITUAL_PHRASES. Promote per-scenario bannedMarkers to the global ban list — straightforward followup.

Test plan

13/13 post-processor unit tests
24/24 cost-tracker tests (rebased after threshold update for new cap)
Typecheck clean
Pre-commit (test:unit + dynamic-imports + typecheck) green
Local docker stack confirmed anonymousModel=claude-sonnet-4-6
Local 33-scenario redteam: 23 passes, 5 rate-limit-driven failures, 5 real failures (Sonnet length + 1 specific overclaim)
Post-merge: watch ANTHROPIC_API_KEY daily spend on prod for first 48h — anon-IP scope_keys should stay under $3/day per IP
Post-merge: re-run redteam against staging to confirm Sonnet+stripper carries from local

🤖 Generated with Claude Code

Three coordinated changes targeting the 12 baseline redteam failures observed against Haiku-mode anonymous chat (banned ritual phrases, length blow-out on short questions, fabrication patterns). 1) server/src/config/models.ts — anonymousChat default flips from ModelConfig.fast (Haiku) to ModelConfig.primary (Sonnet). Anonymous traffic is the source of every visible quality leak in flagged-thread review (mcpServers config gap, claude mcp add --url flag invented, rendering_context.iframe_type schema fabricated). Haiku has demonstrated ~10-20% adherence loss on response-style.md negative instructions; Sonnet handles them substantially better. Override path preserved via ADDIE_ANONYMOUS_MODEL. 2) server/src/addie/claude-cost-tracker.ts — anonymous daily cap $1 → $3. Sized so the 50-message-per-IP-per-day rate limit binds first for legitimate users (50 × $0.05/Sonnet-turn = $2.50, +$0.50 headroom). Old $1 cap was sized for Haiku (~$0.01/turn → ~100 turns/day). Tier ordering preserved: anonymous ($3) < member_free ($5) < member_paid ($25). 3) server/src/addie/response-postprocess.ts (new) + claude-client.ts hooks — deterministic strip of phrases the model leaks despite response-style.md banning them: "the honest answer is", "great question", "to be clear,", "sharp question", "let me be honest", etc. Source list mirrors BANNED_RITUAL_PHRASES in redteam-scenarios.ts (single source of truth — redteam asserts presence, postprocessor asserts absence). Strips outside fenced code blocks, re-capitalizes following sentences, idempotent, no external state. Applied at all three response-emission points: non-streaming processMessage (line 898) and the two processMessageStream done events (lines 1422, 1453). Streaming caveat: live token chunks still show original phrases as they arrive; the persisted done payload (and therefore thread-record + any post-render display) is clean. Full mid-stream rewriting requires buffering and is out of scope. 13 unit tests cover the post-processor: idempotence, code-block preservation, multiple phrases per response, case insensitivity, every literal in the banned list survives strip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CI caught five tests in claude-cost-tracker.test.ts and claude-client-cost- gate.test.ts that hardcoded the old anonymous $1 cap (66_667 Opus input tokens × $15/M = $1.00, just over). With the cap raised to $3, those tokens now leave the user under-budget and the "blocks call" assertions fail. Switched all five sites from a hardcoded 66_667 to a derived `Math.ceil((DAILY_BUDGET_USD.anonymous * 1_000_000) / 15) + 1`, so the threshold tracks the configured cap. The "applies different budgets per tier" test also moved to 2_000_000 Sonnet input tokens (~$6) to land between member_free $5 and member_paid $25 — now exercises the boundary that was implicit before. 24/24 tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(addie): teach length-matching and TMP-privacy precision Two failure modes survived the Haiku→Sonnet bump in PR #3273: length blow-out on short questions (Sonnet writes 250-350 words for 7-12-word challenges) and "cryptographic guarantee" overclaim on TMP privacy questions. Both addressed via positive teaching in the rules rather than adding more to the post-processor strip list. response-style.md (Match the register): - Added "Why this matters: verbosity reads as defensiveness" — names the failure mode the model defaults into and explains *why*. Verbosity reads as either misunderstanding the question or hedging. - Added "Self-check before replying" — two questions to ask before composing: how many words is the question (count them, calibrate your answer), and did the caller want a paragraph or an essay (the question's shape tells you). - Added an anti-example showing the same priv-1 question answered badly at 280 words with headings + bullets, alongside the existing good-example at 80 words. - Added "Concrete word counts, by question shape" — explicit per-shape targets: ≤120 for one-line challenges, 120-200 for short open questions, length-scales-with-parts for multi-part. knowledge.md (Privacy in AdCP): - "The precise terms for this design — use these" — names the three accurate terms (architectural separation, data minimization, two-endpoint design) the model should reach for. Tells it WHAT to say. - "What NOT to claim, and why" — enumerates the three cryptographic overclaims (cryptographic guarantee, proven secure, cryptographically prevent) alongside *why* they're wrong: TMP doesn't ship cryptographic primitives today (no ZK proofs, no homomorphic encryption, no signed-attribute attestation). - Closes with "if a caller specifically asks about cryptographic guarantees, say 'that's ongoing work in TMP, not shipped today'" — gives an honest deflection that names the gap rather than papering over it. The post-processor strip list intentionally NOT touched: the rule update teaches the precision; the model should reach for the right terms because it understands the substance, not because a regex catches the wrong terms after the fact. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(addie): teach against comprehensive list-dumping on conversational questions Direct probe of priv-1 / acct-1 / gap-1 after the previous teaching pass showed Sonnet wasn't ignoring the length cap from defensive verbosity — it was responding to list-shaped questions ("What does AdCP not do?", "What tools do you have for X?") by dumping the full structured list with bold headings, ~280 words of factually correct content. Same quantity of information density either way; the question was about the conversational shape. Adds an anti-pattern callout to response-style.md "Match the register" section: "Don't dump comprehensive lists in response to list-shaped questions." Default is a 2-3-item summary plus "want me to expand?". Comprehensive answers are opt-in. Anti-example with the actual question Sonnet failed on (gap-1 "What does AdCP not do?") — bad 280-word structured response vs good 35-word conversational summary, same information density. Same source of truth: the docs already have the comprehensive list. Addie's job is to summarize and let the caller pull threads. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(addie): tighten teaching rules per expert review Polish pass on the length / privacy teaching from the previous two commits, based on prompt-engineer + docs-expert review of PR #3290. response-style.md: - Compressed the "Why this matters: verbosity reads as defensiveness" meta paragraph (~110 words explaining the failure mode) into one sentence — the model needs the rule, not the justification, and the meta scaffolding burned ~90 tokens per turn for no behavioral lift. - Folded the "Self-check before replying" numbered checklist into a single bolded directive ("Calibrate your answer to the question's shape"). The checklist read as procedural where the rest of the file reads as imperative — Sonnet won't actually count words on a scratchpad, so framing it as a self-check was prompt-magic; the directive form is what lands. - Removed the duplicate "Anti-example of the same question, why it fails" block on the surveillance-capitalism question. The new "Don't dump comprehensive lists" rule already has a worked anti-example (gap-1, "what does AdCP not do?") covering the same diagnosis (headings, bullets, ~280-word essay) and the same remedy. Two examples of the same teaching collapse into one — keep the one closer to where Sonnet's templating tic actually fires. - Softened the "≤120 words" ceiling to "~120 words, prose only, unless the question genuinely requires more" to avoid clipping legitimately complex short questions. knowledge.md privacy section: - Reworded the cryptographic-guarantee deflection. The old form ("that's ongoing work in TMP, not shipped today") put the trigger word "cryptographic" inside the sanctioned phrasing, which Sonnet could re-leak via paraphrase ("cryptographic work is ongoing"). New form ("those primitives aren't part of TMP today; the shipped design is architectural") removes the trigger and points at the positive answer. Net: ~110 fewer words in response-style.md per turn, no semantic loss. Privacy fix should be more durable. Length teaching unchanged in substance — the experts converged that this teaching reduces mean length but doesn't enforce a ceiling, and the deterministic length post-processor remains the right next step if redteam still shows length blow-out post-merge. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ions Prod redteam after PR #3273 (Sonnet bump) and PR #3290 (length-matching teaching) showed Sonnet still over-explains on short challenges: - priv-1 ("surveillance capitalism?"): 254 words - acct-1 ("$500K on garbage inventory"): 304 words - acct-2 ("AI screws up"): 227 words - gap-1 ("what AdCP not do"): 173 words - priv-1 length blow-out: 217 words Both prompt-engineer and docs-expert reviews of #3290 had converged on the same conclusion: rule-based teaching reduces mean length but doesn't enforce a ceiling, and a deterministic post-processor is the right backstop. This is that backstop. New `truncateLongResponseToShortQuestion(question, text)` in `server/src/addie/response-postprocess.ts`: - Fires when question ≤15 words AND response >160 words. - Truncates at the nearest sentence boundary at or before 130 words. - Appends "Happy to go deeper on any of this if useful." (italicized). - Code blocks are split out and kept whole — never cut mid-fence; if including a fence would push over the target, stops before it. - Always keeps the first sentence even if it alone exceeds the target (a one-sentence answer beats no answer). - Idempotent: running on already-truncated text is a no-op. Wired at the same four return sites in `claude-client.ts` as the banned-ritual stripper (lines 770, 901, 1423, 1454). The chain is now `stripBannedRituals` → `truncateLongResponseToShortQuestion` → return. 10 new unit tests cover question-length gate, response-length gate, sentence-boundary preservation, single-giant-sentence fallback, code-block preservation, idempotence, empty-input handling. 21/21 tests pass on the postprocess module. Expected redteam impact: priv-1 / acct-1 / acct-2 / gap-1 length_cap failures drop to zero. The single remaining length-related concern is priv-1's banned_marker on "adcp is more private", but that's a separate overclaim issue the truncator doesn't address (the marker can appear within the truncated body). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…gth cap, empty-response guard (#3306) * fix(addie): strip banned rituals on the end_turn multi-textblock path Prod redteam against the deployed Sonnet+stripper combo (PR #3273) showed 3 ritual-phrase leaks ("the honest answer is", "that's a fair question", "here's the honest answer") despite the stripper being in place. Tracing showed the strip was applied at three response-emission sites in claude-client.ts but missed a fourth: the end_turn path at line 763 that joins multi-text-block responses (used by web-search-returning answers). That path returned `text` unstripped to the caller. Now consistent with the other three return points — collect rawText, run through stripBannedRituals, return the cleaned form. No semantic change to non-end_turn paths. Verified: prod redteam categorized the 3 banned-phrase failures on gov-3, gov-4, cad-1 — all questions where Sonnet's response goes through the end_turn-with-multiple-text-blocks path. After this fix the four return-points (line 539 cost-cap-message, 798 end_turn-multitext, 902 toolUseBlocks-empty, 1422/1453 streaming-done) all run the strip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(addie): deterministic length post-processor for short user questions Prod redteam after PR #3273 (Sonnet bump) and PR #3290 (length-matching teaching) showed Sonnet still over-explains on short challenges: - priv-1 ("surveillance capitalism?"): 254 words - acct-1 ("$500K on garbage inventory"): 304 words - acct-2 ("AI screws up"): 227 words - gap-1 ("what AdCP not do"): 173 words - priv-1 length blow-out: 217 words Both prompt-engineer and docs-expert reviews of #3290 had converged on the same conclusion: rule-based teaching reduces mean length but doesn't enforce a ceiling, and a deterministic post-processor is the right backstop. This is that backstop. New `truncateLongResponseToShortQuestion(question, text)` in `server/src/addie/response-postprocess.ts`: - Fires when question ≤15 words AND response >160 words. - Truncates at the nearest sentence boundary at or before 130 words. - Appends "Happy to go deeper on any of this if useful." (italicized). - Code blocks are split out and kept whole — never cut mid-fence; if including a fence would push over the target, stops before it. - Always keeps the first sentence even if it alone exceeds the target (a one-sentence answer beats no answer). - Idempotent: running on already-truncated text is a no-op. Wired at the same four return sites in `claude-client.ts` as the banned-ritual stripper (lines 770, 901, 1423, 1454). The chain is now `stripBannedRituals` → `truncateLongResponseToShortQuestion` → return. 10 new unit tests cover question-length gate, response-length gate, sentence-boundary preservation, single-giant-sentence fallback, code-block preservation, idempotence, empty-input handling. 21/21 tests pass on the postprocess module. Expected redteam impact: priv-1 / acct-1 / acct-2 / gap-1 length_cap failures drop to zero. The single remaining length-related concern is priv-1's banned_marker on "adcp is more private", but that's a separate overclaim issue the truncator doesn't address (the marker can appear within the truncated body). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(addie): empty-response fallback + factor pipeline into single helper Three things in one tidy commit: 1) Empty-response guard. Two redteam-flagged turns produced rating=1 "Response is empty" responses on short follow-up prompts ("?", "what happened?"). The model returned an assistant message with empty text. Now substitutes a clarifying fallback ("Sorry, I lost the thread there. Could you rephrase or give me a bit more context?") so the user gets something rather than nothing. 2) Pipeline factoring. The four assistant-text return sites in claude-client.ts each did `stripBannedRituals(...)` → `truncateLongResponseToShortQuestion(userMessage, ...)`. Factored the chain into a single `applyResponsePipeline(question, rawText)` helper in response-postprocess.ts so the pipeline (strip → empty-guard → truncate) lives in one place. Future additions slot in there. 3) Tests. 4 new unit tests on the pipeline helper covering empty input, ritual-only input (gets stripped to empty → fallback), substantive long input (gets stripped + truncated), and substantive short input (passes through unchanged). 25/25 tests in the postprocess module pass. Closes the third item from the cycle followup punch list (empty-message bug, #124-style). The actual root-cause investigation of *why* Sonnet returns empty assistant blocks is out of scope — this just turns the silent-empty into a polite-clarifying-fallback so users aren't left staring at a blank reply. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

bokelley and others added 2 commits April 26, 2026 11:34

bokelley merged commit 7a84929 into main Apr 26, 2026
13 checks passed

bokelley deleted the bokelley/addie-quality-fixes branch April 26, 2026 15:47

bokelley mentioned this pull request Apr 26, 2026

feat(addie): teach length-matching and TMP-privacy precision #3290

Merged

4 tasks

bokelley mentioned this pull request Apr 26, 2026

feat(addie): response post-process pipeline — strip end_turn fix, length cap, empty-response guard #3306

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(addie): anonymous→Sonnet, daily cap →, ritual-phrase stripper#3273

feat(addie): anonymous→Sonnet, daily cap →, ritual-phrase stripper#3273
bokelley merged 2 commits into
mainfrom
bokelley/addie-quality-fixes

bokelley commented Apr 26, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

bokelley commented Apr 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

1. Anonymous chat → Sonnet (server/src/config/models.ts)

2. Anonymous daily cap $1 → $3 (server/src/addie/claude-cost-tracker.ts)

3. Banned-ritual post-processor (server/src/addie/response-postprocess.ts + claude-client.ts hooks)

Local redteam verification

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bokelley commented Apr 26, 2026 •

edited

Loading

1. Anonymous chat → Sonnet (`server/src/config/models.ts`)

2. Anonymous daily cap $1 → $3 (`server/src/addie/claude-cost-tracker.ts`)

3. Banned-ritual post-processor (`server/src/addie/response-postprocess.ts` + claude-client.ts hooks)