fix(rag): behavior-neutral top-5 safety net for clozapine withhold-threshold evidence#236
fix(rag): behavior-neutral top-5 safety net for clozapine withhold-threshold evidence#236BigSimmo wants to merge 2 commits into
Conversation
The golden case clozapine-wcc-abbreviation-threshold currently passes on main (content_recall@5=1, full eval:retrieval:quality suite green), but the pass is corpus-state dependent: the collapsed lexical query "clozapine monitoring" pulls many sibling clozapine chunks, and a small ranking jitter can float them above the single structured threshold table that pairs the blood parameter (WBC/ANC/neutrophil) with the withhold/cease/red action, dropping that one answer-bearing chunk past rank 5. That is the regression the case was nearly deferred for (claude/defer-clozapine-wcc-golden-case). Add a narrowly scoped, safety-critical top-5 guarantee: for clozapine blood-action withhold-threshold queries only, when the top-5 does not already carry both the blood term and the action, promote the best already-retrieved threshold+action chunk from below the cutoff into the last top-5 slot. Pure reordering of the existing candidate set; a no-op whenever the evidence is already ranked, so it changes nothing on the current corpus. The coverage gate now shares the same query-shape predicate and term/action patterns (behaviour-identical refactor). Validated: - unit: promotes under simulated jitter, no-op when already covered, no-op for non-clozapine queries (tests/retrieval-query-variants.test.ts) - live: eval:retrieval:quality --fail-on-threshold green before and after (25 cases, content_recall@5=1, 0 failures, identical clozapine top-5); anc/cbc/wcc cases deterministic across runs - typecheck, lint, prettier clean Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f381f4c217
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const topText = topEvidenceText(results, limit); | ||
| if (clozapineBloodTermPattern.test(topText) && clozapineWithholdActionPattern.test(topText)) return results; |
There was a problem hiding this comment.
This early no-op checks the concatenated top-five text, so unrelated top-five chunks can satisfy the blood term and action separately (for example one generic clozapine/ANC monitoring chunk plus another chunk mentioning a red/stop action) while the actual structured threshold+action table remains at rank 6. In that scenario this new safety net returns without promotion even though no top-five result carries the answer-bearing evidence, leaving the clozapine withhold threshold fragile in exactly the jitter case this change is meant to guard against; the check should use the same per-result structured threshold predicate used for promoted candidates.
Useful? React with 👍 / 👎.
| // they must agree on what counts as blood evidence and a withhold/cease action, or the gate could | ||
| // accept a top-5 the net believes is uncovered (or vice versa). None carry the /g flag, so `.test` | ||
| // is stateless and safe to reuse. | ||
| const clozapineBloodTermPattern = /\b(?:anc|fbc|wbc|wcc|neutrophil|neutrophils|full blood|white cell)\b/i; |
There was a problem hiding this comment.
When the user asks this same clozapine withhold question using only CBC / complete blood count or white blood cell count rather than ANC/FBC/WBC/WCC, this predicate returns false because those blood terms are not in the shared pattern. The added CBC test is masked because it also contains neutrophil, so those common phrasings still skip the new promotion and the clozapine-specific coverage gate, leaving the answer vulnerable to the same rank-jitter failure.
Useful? React with 👍 / 👎.
| // Final step, after all reranking: guarantee the clozapine withhold-threshold table is in the | ||
| // top-5 the coverage gate evaluates and the answer layer sees. Runs last so no later sort can | ||
| // undo it; a no-op for every other query and whenever the evidence is already ranked. | ||
| results = ensureClozapineBloodActionEvidenceRanked(args.query, results); |
There was a problem hiding this comment.
This call runs after selectRankedRetrievalResults has already truncated to the caller's topK, but the API accepts topK values down to 1 and a caller asking for 5 results is exactly asking for the top five. In that case a threshold table at rank 6 is discarded before the safety net runs, so results.length <= limit makes the helper no-op and the rank-jitter case this change is intended to fix still reaches the coverage gate without the answer-bearing table; fetch/select at least one candidate beyond the cutoff for this query shape before applying the promotion.
Useful? React with 👍 / 👎.
BigSimmo
changed the title
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 188f7a947a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (!hasStructuredThresholdEvidence(result)) return false; | ||
| const text = evidenceTextForGate(result); | ||
| return clozapineBloodTermPattern.test(text) && clozapineWithholdActionPattern.test(text); |
There was a problem hiding this comment.
For clozapine withhold-threshold queries, this treats hasStructuredThresholdEvidence as if it only means table facts / threshold units / table crops, but that helper also returns true from a broad keyword fallback such as anc, withhold, or stop. When the top 5 lacks the answer table and a below-cutoff prose chunk says something like “monitor ANC” and “stop clozapine” without a numeric cutoff/table row, the new safety net can promote that non-answer chunk, after which the coverage gate accepts and skips vector retrieval even though the actual threshold is still absent.
Useful? React with 👍 / 👎.
| // accept a top-5 the net believes is uncovered (or vice versa). None carry the /g flag, so `.test` | ||
| // is stateless and safe to reuse. | ||
| const clozapineBloodTermPattern = /\b(?:anc|fbc|wbc|wcc|neutrophil|neutrophils|full blood|white cell)\b/i; | ||
| const clozapineWithholdQueryPattern = /\b(?:withhold|withheld|withholding|cease|ceased|stop|stopped)\b/i; |
There was a problem hiding this comment.
Because this new query-shape pattern now includes ceased, a query like “when should clozapine be ceased based on WBC?” is supposed to receive the clozapine safety net, but decideTextFastPath still only blocks early table-threshold fast paths for cease/stop/withhold forms and not ceased. If the initial top results contain any strong structured-threshold-looking chunk, that phrasing can return before prepareCoverageGateResults runs, so the new promotion and stricter clozapine coverage gate are bypassed for one of the phrasings added here.
Useful? React with 👍 / 👎.
| // clozapine?" (any phrasing/abbreviation). Shared by the coverage gate and the top-5 safety net. | ||
| export function isClozapineBloodActionThresholdQuery(query: string) { | ||
| return ( | ||
| /\bclozapine\b/i.test(query) && clozapineBloodTermPattern.test(query) && clozapineWithholdQueryPattern.test(query) |
There was a problem hiding this comment.
This predicate checks only the raw query text for clozapine, but the search pipeline already typo-corrects clozapin to clozapine when building/ranking candidates, and the eval corpus includes the withhold-threshold phrasing “What ANC or FBC cut off means clozapin should be withheld?”. For that realistic typo, the same rank-jitter scenario still skips the new promotion entirely even though the candidate set is otherwise the clozapine monitoring set, leaving the safety net absent for an existing supported query form.
Useful? React with 👍 / 👎.
|
Closing in favour of the deferral that landed on Why closing: this PR existed to make the golden case
Preserved for future revival (branch
Key finding worth keeping: the reported 0.67 for this case was corpus-state dependent, not a code gap — the defer branch ran on identical ranking code yet the case passed deterministically on the live corpus at the time I measured it; a live re-index had moved the answer-bearing structured-threshold table into the top-5. Root fragility: the collapsed lexical query If a real, reproducible clozapine withhold-threshold miss ever appears in production, revive this deliberately with that failure as the gate. Note the token-append lever (adding wbc/neutrophil to the collapsed query) is unsafe: the RPC uses |
1 participant
Summary
What this PR adds is a standalone, behavior-neutral retrieval safety net, independent of the golden fixture:
"clozapine monitoring") pulls many sibling clozapine chunks, and the single structured threshold table that pairs the blood parameter (WBC/ANC/neutrophil) with the withhold/cease/red action is the only top-5 chunk carrying the action, so a small ranking jitter can push it past rank 5 — leaving a top-5 that names the drug but omits the actual threshold.ensureClozapineBloodActionEvidenceRanked: for clozapine blood-action withhold-threshold queries only, when the top-5 doesn't already carry both the blood term and the action, it promotes the best already-retrieved threshold+action chunk from below the cutoff into the last top-5 slot. Pure reordering of the existing candidate set; wired as the final step ofprepareCoverageGateResultsso no later sort can undo it.Reviewers: this is offered on its own merits as defence-in-depth for a safety-critical answer path. It's independent of the deferred golden case; accept or decline without affecting the defer.
Verification
npm run verify:cheap— 106 files, 984 tests pass (re-run after merging currentmain)npm run verify:ui— N/A (no UI, routing, styling, or browser behaviour change)npm run verify:release— N/Anpm run format:check— cleannpm run check:production-readiness— READY (Supabase target confirmedClinical KB Database)npm run check:deployment-readiness— N/AAdditional (retrieval-quality gate, live Supabase + OpenAI, embedding-skip ~1.0 so negligible cost):
npm run eval:retrieval:quality -- --fail-on-thresholdgreen before and after the guard — identical metrics, deterministic clozapine top-5 across runs.tests/retrieval-query-variants.test.ts).Clinical Governance Preflight
Change touches search/ranking and clinical output.
Clinical KB Database(sjrfecxgysukkwxsowpy) — verified bycheck:production-readiness; no env/config changeNotes
main(inherits test(eval): defer clozapine-wcc-abbreviation-threshold golden case #227).eval:retrieval:qualityreads the live shared corpus, so a golden case can flip pass/fail with no code change — treat single-run passes as a corpus snapshot.🤖 Generated with Claude Code