feat(addie): response post-process pipeline — strip end_turn fix, length cap, empty-response guard - #3306
Merged
Merged
Conversation
Prod redteam against the deployed Sonnet+stripper combo (PR #3273) showed 3 ritual-phrase leaks ("the honest answer is", "that's a fair question", "here's the honest answer") despite the stripper being in place. Tracing showed the strip was applied at three response-emission sites in claude-client.ts but missed a fourth: the end_turn path at line 763 that joins multi-text-block responses (used by web-search-returning answers). That path returned `text` unstripped to the caller. Now consistent with the other three return points — collect rawText, run through stripBannedRituals, return the cleaned form. No semantic change to non-end_turn paths. Verified: prod redteam categorized the 3 banned-phrase failures on gov-3, gov-4, cad-1 — all questions where Sonnet's response goes through the end_turn-with-multiple-text-blocks path. After this fix the four return-points (line 539 cost-cap-message, 798 end_turn-multitext, 902 toolUseBlocks-empty, 1422/1453 streaming-done) all run the strip. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ions Prod redteam after PR #3273 (Sonnet bump) and PR #3290 (length-matching teaching) showed Sonnet still over-explains on short challenges: - priv-1 ("surveillance capitalism?"): 254 words - acct-1 ("$500K on garbage inventory"): 304 words - acct-2 ("AI screws up"): 227 words - gap-1 ("what AdCP not do"): 173 words - priv-1 length blow-out: 217 words Both prompt-engineer and docs-expert reviews of #3290 had converged on the same conclusion: rule-based teaching reduces mean length but doesn't enforce a ceiling, and a deterministic post-processor is the right backstop. This is that backstop. New `truncateLongResponseToShortQuestion(question, text)` in `server/src/addie/response-postprocess.ts`: - Fires when question ≤15 words AND response >160 words. - Truncates at the nearest sentence boundary at or before 130 words. - Appends "Happy to go deeper on any of this if useful." (italicized). - Code blocks are split out and kept whole — never cut mid-fence; if including a fence would push over the target, stops before it. - Always keeps the first sentence even if it alone exceeds the target (a one-sentence answer beats no answer). - Idempotent: running on already-truncated text is a no-op. Wired at the same four return sites in `claude-client.ts` as the banned-ritual stripper (lines 770, 901, 1423, 1454). The chain is now `stripBannedRituals` → `truncateLongResponseToShortQuestion` → return. 10 new unit tests cover question-length gate, response-length gate, sentence-boundary preservation, single-giant-sentence fallback, code-block preservation, idempotence, empty-input handling. 21/21 tests pass on the postprocess module. Expected redteam impact: priv-1 / acct-1 / acct-2 / gap-1 length_cap failures drop to zero. The single remaining length-related concern is priv-1's banned_marker on "adcp is more private", but that's a separate overclaim issue the truncator doesn't address (the marker can appear within the truncated body). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
changed the title
Three things in one tidy commit:
1) Empty-response guard. Two redteam-flagged turns produced rating=1
"Response is empty" responses on short follow-up prompts ("?", "what
happened?"). The model returned an assistant message with empty text.
Now substitutes a clarifying fallback ("Sorry, I lost the thread
there. Could you rephrase or give me a bit more context?") so the
user gets something rather than nothing.
2) Pipeline factoring. The four assistant-text return sites in
claude-client.ts each did `stripBannedRituals(...)` →
`truncateLongResponseToShortQuestion(userMessage, ...)`. Factored the
chain into a single `applyResponsePipeline(question, rawText)` helper
in response-postprocess.ts so the pipeline (strip → empty-guard →
truncate) lives in one place. Future additions slot in there.
3) Tests. 4 new unit tests on the pipeline helper covering empty input,
ritual-only input (gets stripped to empty → fallback), substantive
long input (gets stripped + truncated), and substantive short input
(passes through unchanged). 25/25 tests in the postprocess module pass.
Closes the third item from the cycle followup punch list (empty-message
bug, #124-style). The actual root-cause investigation of *why* Sonnet
returns empty assistant blocks is out of scope — this just turns the
silent-empty into a polite-clarifying-fallback so users aren't left
staring at a blank reply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three response post-processor changes from the prod redteam follow-up, factored into a single
applyResponsePipeline(question, rawText)helper used at all four assistant-text return sites inclaude-client.ts.1. Bug fix: missed code path in the banned-ritual stripper
Prod redteam against the deployed Sonnet+stripper combo (PR #3273) showed 3 ritual-phrase leaks ("the honest answer is", "that's a fair question", "here's the honest answer") despite the stripper being in place. The strip was applied at three of four return sites in
claude-client.tsbut missed theend_turnpath that joins multi-text-block responses (used by web-search-returning answers).2. Feature: deterministic length post-processor
Both prompt-engineer and docs-expert reviews of PR #3290 converged: rule-based teaching reduces mean length but doesn't enforce a ceiling on Sonnet's verbosity. Prod redteam confirmed:
New
truncateLongResponseToShortQuestion(question, text):3. Empty-response guard
Two redteam-flagged turns produced rating=1 "Response is empty" responses on short follow-up prompts ("?", "what happened?"). The model returned an assistant message with empty text. Now substitutes a clarifying fallback rather than ship the void.
Single pipeline
All three live behind
applyResponsePipeline(question, rawText):Used at the four assistant-text return sites in
claude-client.ts(lines 770, 901, 1423, 1454). The chain is the same everywhere; future additions slot into one helper.Verification
npm run typecheckcleanExpected redteam impact
ritual-phrase hitslength_capRemaining failures (out of scope, separate followups):
"adcp is more private"banned_marker"proven secure"banned_markerBoth are overclaim phrases the truncator/stripper don't catch (they appear within otherwise valid body text). Right fix: promote per-scenario
bannedMarkersto a global ban list. Small followup PR.🤖 Generated with Claude Code