From d50b465ad4aaf31a38e351b1d0d5cffca549b733 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 02:46:40 +0000 Subject: [PATCH 1/4] docs(ledger): E-3b + PR-B live pair verdict (run #59 green vs #57/#58) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- docs/branch-review-ledger.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index cb5a950b..2cc5d24f 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -670,3 +670,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3b budget-aware generation deadlines) | 314d03f (+eb08ec5 review row) | ADDENDUM 5 wave E-3b implemented per the design-agent plan: generation attempts clamped to route budget minus a measured 2s recovery reserve (single call-site, all four attempt kinds); truncation self-heal gated on retry viability (reserve+5s floor) with observable truncation_retry_skipped_budget_reserve marker; additive route_budget_exhausted_by_retrieval runtime flag; eval route-ceiling gains the triple-condition cross-region carve-out (context + runtime flag + zero generation) with retrieval-exhausted audit cells — local/release gates provably strict. Fixes I3 (54ms budget overrun after 22.6s provider timeout), I5 (82s truncation waste class), resolves I2 (clozapine 13.3s retrieval vs 12s runtime budget = geography, now suppressed ONLY in the sanctioned cross-region context with full auditability). Reviewer verdicts: rag-retrieval-reviewer APPROVE-WITH-NITS (2 P3: prod-dead requestTimeoutMs retained for symmetry; report-cell coupling cosmetic; cached-replay invariant PROVEN — budget-exhausted answers never cached, carve-out unreachable via replay; marker isolation proven — SLO counters key on fallback_reason not answer_retry_reasons); clinical-governance-reviewer APPROVE-WITH-NITS (prior row) — internal-timeout→PublicApiError→existing-fallback path verified, all safety gates still applied to recovery answers. ALSO BANKED — E-2 targeting baseline (canary run #58, 29788404357, all-green incl. first execution of the !cancelled()-fixed instrument, ~$1-2): metric_rates relevance 0.6 / readability 1.0 / artifact_leaks 1.0 / intent_coverage 0.9333 / fail_closed 0.9; targeting_rate 0.5909 (13/22); by intent: document_lookup 5/5, red_result_action 3/3, contraindication 2/2, dose 1/5, monitoring_schedule 1/5, pathway_referral 1/2; all 9 misses = missing dose figure/schedule-interval (answer lengths 73-232 chars) → E-3c co-primary target alongside the wasted-generation class. Phase E spend ≈$3-6 of ≤$20. | Red-proofs: reserve pinned 3 independent ways (exact 23000ms grant, deadline flag clear, total under budget); self-heal skip pins exact marker + single provider call; offline flag pinned true/false. Focused: route-budget 9/9, eval-quality 27/27, fallback+offline 52/52, parser/abort regressions 15/15. Full suite 3043 passed / 1 known container pdf artifact. typecheck+lint+prettier clean. No provider calls; live proof = E-4 paired run | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from e02ba3d; PR: E-3c PR-A instrument) | see PR head | ADDENDUM 5 wave E-3c design accepted (Plan agent, full report in session record) and PR-A delivered: eval-answer-quality gains --dump-answers (per-case answer TEXT + sections + targeting verdict JSON for the canary artifact — answers are not retained at rest by privacy design, so miss diagnosis needs eval-time capture), parseArgs/buildAnswerDumpRecord exported behind an import.meta main-guard (eval-quality precedent), canary targeting step wired with the flag. Design highlights for the record: DEFECT 1 root confirmed as fast-attempt-doomed-then-discarded on strong_routine_retrieval procedural shapes — PR-B generalizes the EXISTING validated-extractive short-circuit pattern (LAI + blocked-recovery precedents, hasValidatedExtractiveCandidate) to the measured shape via a new rag-extractive-first.ts module (net ~-110 rag.ts lines against the 5030 budget); DEFECT 2 rank-1 root = extractive lead-slot selection prefers shortest sentence and admits figure-less leads (sort at rag-extractive-answer.ts:917, 1-slot monitoring leads) — PR-C adds intent-figure-aware lead promotion with a claim-support atom-corpus nuke-proofing guard + dose/threshold fallback candidate preference (find(safe && figure) ?? find(safe)); answer-verification CLEARED as direct cause (whole-answer gap or unbold only, never per-figure deletion). H2 (strong-route comparison/complex residual) explicitly deferred as the named E-3d candidate. E-4 metrics set: discarded-generation <20% from ~47%, dose ≥3/5, monitoring ≥3/5, no intent below #58, recalls pinned 1.0, relevance/fail_closed/readability/artifact_leaks ≥ #58. | PR-A gates: new tests/eval-answer-quality.test.ts 4/4; typecheck, check:github-actions, check:ci-scope, eval-canary-workflow test, prettier all clean; no provider calls | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3c PR-B short-circuit) | f7e6cbb + hardening commit | E-3c PR-B: pre-generation validated-extractive short-circuit for gate-passed routine procedural "What...process/include/required" queries (marker validated_routine_extractive_first), generalizing the LAI + blocked-recovery precedents via new rag-extractive-first.ts (3 predicates moved byte-verbatim, machine-verified; rag.ts 5029→4908 vs 5030 budget). Kills the run-#57 6x wasted-generation class. REVIEWS (both pre-push): rag-retrieval-reviewer APPROVE-WITH-NITS — move fidelity brace-diff verified byte-identical; confidence-gate skip PROVEN safe (passed-gate is a no-op in applyConfidenceGate; markers pairwise mutually exclusive); comparison false-positives blocked by unchanged classifier precedence; offline/source-only idempotent; zero retrieval/ranking/selection/threshold change; P2 = eval-only assertions (intent_coverage/artifact_leaks/expected-file) unverifiable offline for flip candidates (quality-nocc-document-support, quality-form-required-documentation, quality-discharge-documentation, quality-duress-pathway + rag-set siblings) → pre-merge BRANCH canary recommended and ADOPTED (offline-green + review-approved proven insufficient for this surface, 2026-07-20). clinical-governance-reviewer APPROVE-WITH-NITS — full gate-stack trace: nothing bypassed (same finalizeRagAnswerQuality, same citation scoping, numeric verification not fail-open, ungrounded-finalize defense at rag.ts:3694); P2 = pre-existing bare-cross-reference-with-overlap gap, NOT materially widened (new trigger anti-correlates), hardening recommended → APPLIED this PR: !isBareCrossReferenceAnswer screen in hasValidatedExtractiveCandidate (closes all three short-circuit paths; discriminating test added; disclosed post-review delta, strictly narrows shipping). MERGE GATE: draft until the branch canary pair (baseline #57/#58 vs branch run with answer_case_limit=44 + answer_quality_eval=true, est $3-6 of authorized envelope) is green — zero per-case regressions, recalls 1.0, quality/targeting rates >= baseline. | Red-proof + 3 negative guards; focused 61/61 + fallback/offline/contract suites; full suite 3061 passed / 1 known container artifact (pre-hardening tree; hardening re-verified focused); typecheck+lint+prettier clean; maintainability budget passed (4908/5030) | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (pair verdict — no code change; main stays at 22b6a2e) | canary run 29794759627 (#59, branch head 7310cb3 = merged PR-B content) | E-3b + E-3c PR-B LIVE-VALIDATED (pair vs banked #57/#58): route_ceiling_failures 2→0 (E-3b proven: agitation timeout now fits inside budget; clozapine retrieval-exhausted ceiling honestly excused via the triple-condition cross-region carve-out — exactly one "retrieval-exhausted" audit cell in the report); p95 17.4s→15.47s (-11%); golden retrieval SUCCESS 36/36 (stop-ship criterion held); validated_routine_extractive_first fired on 4 of the 6 target cases (patient-safety-plan 3.3s, treatment-team-process 2.6s, ect-procedure 2.2s, illegal-substances 2.2s — all pure extractive, zero generation, was 6-9s each with a discarded attempt), the other 2 (community-home-visits, best-practice-prescribing) stayed on generation+fallback because their extractive candidates legitimately fail validation gates = the designed-conservative outcome; ZERO new failing cases (list 5→2, both known residuals: neuroleptic citation red = Option A territory, admission-comparison expected-doc = non-blocking labeling residual); grounded 1.0 + unsupported_correct 1.0 held; targeting 0.5909→0.619, fail_closed 0.9→0.9333, readability/artifact_leaks/intent_coverage unchanged; relevance 0.6→0.5667 = single-case wobble on n=30, WATCH in E-4, not a gate. Discarded-generation rate materially down (4 conversions; residual = the H2 strong-route slice named as E-3d candidate, per design's 20-33% expectation band). MERGE STANDS (user had armed auto-merge pre-verdict; revert drill not triggered). Spend +~$3-6 → Phase E total ~$6-12 of ≤$20. | Pair evidence: run #59 job log (Blocking failures = citation only; Answer Case Diagnostics markers; metric_rates + targeting blocks); dump artifact populated for PR-C diagnosis | From 043b030d233e79f25a7528e5fcf9194f72349b24 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 03:15:29 +0000 Subject: [PATCH 2/4] feat(rag): figure-aware extractive selection for dose/monitoring answers (E-3c PR-C) Dose and monitoring answers must carry the asked-for figure/schedule when a cited chunk verbatim supports it (run-#58 baseline: targeting 1/5 on both intents, every miss = missing figure). Lead-slot promotion swaps/appends the first later figure-bearing fact ONLY when no lead carries one, guarded by the claim-support atom corpus (sourceEvidenceText, now exported) so a promoted figure can never convert a grounded answer into an evidence gap - the guard proven load-bearing by red-testing it disabled. The dose/threshold generation fallback now prefers the safe candidate that carries the intent figure over the first safe one; the safety gate itself is unchanged. Fallback helpers extracted to rag-extractive-answer (cycle-check verified); rag.ts 4908->4901. Six discriminating tests, each verified failing on prior code. Full suite 3068 passed / 1 known container artifact. RAG impact: behaviour change (extractive answer composition + fallback candidate preference only) - E-4 pair vs banked #57/#58/#59 baselines post-merge; retrieval, ranking, selection ordering untouched. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- src/lib/rag/rag-claim-support.ts | 7 +- src/lib/rag/rag-extractive-answer.ts | 136 ++++++++++++++++++++- src/lib/rag/rag.ts | 37 +++--- tests/extractive-answer-formatting.test.ts | 129 ++++++++++++++++++- tests/rag-answer-fallback.test.ts | 42 +++++++ 5 files changed, 324 insertions(+), 27 deletions(-) diff --git a/src/lib/rag/rag-claim-support.ts b/src/lib/rag/rag-claim-support.ts index 845dd7aa..3c4fe9dd 100644 --- a/src/lib/rag/rag-claim-support.ts +++ b/src/lib/rag/rag-claim-support.ts @@ -212,7 +212,12 @@ function compatibleSafetyDimensions(claim: string, evidence: string) { return [...safetyDimensions(claim)].every((dimension) => evidenceDimensions.has(dimension)); } -function sourceEvidenceText(source: SearchResult) { +// Exported so the extractive figure-promotion guard (rag-extractive-answer.ts) can check a +// candidate fact's value atoms against the EXACT corpus this module assesses claims with. +// Keep the two in lockstep: a promoted figure verified against a wider corpus (e.g. one that +// includes adjacent_context) would pass numeric verification and then trip +// claim_support_high_risk_gap here. +export function sourceEvidenceText(source: SearchResult) { return [ source.section_heading, source.content, diff --git a/src/lib/rag/rag-extractive-answer.ts b/src/lib/rag/rag-extractive-answer.ts index f641a910..ca658e29 100644 --- a/src/lib/rag/rag-extractive-answer.ts +++ b/src/lib/rag/rag-extractive-answer.ts @@ -1,5 +1,5 @@ import { boldHighYieldClinicalText } from "@/lib/answer-ranking"; -import { applyNumericVerification } from "@/lib/answer-verification"; +import { applyNumericVerification, extractClinicalValueAtoms, type ClinicalValueAtom } from "@/lib/answer-verification"; import { citationFromResult as resultCitation, compactCitations } from "@/lib/citations"; import { classifyRagQuery } from "@/lib/clinical-search"; import { ragDeepMemoryVersion } from "@/lib/deep-memory"; @@ -23,6 +23,7 @@ import { sanitizeAnswerText, splitBalancedWords, } from "@/lib/rag/rag-answer-text"; +import { cloneAnswer } from "@/lib/rag/rag-cache"; import { ragProviderMode } from "@/lib/rag/rag-provider"; import { isLowYieldClinicalText, @@ -39,7 +40,7 @@ import type { RagQueryClass, SearchResult, } from "@/lib/types"; -import { assessAndEnforceClaimSupport } from "@/lib/rag/rag-claim-support"; +import { assessAndEnforceClaimSupport, sourceEvidenceText } from "@/lib/rag/rag-claim-support"; type AnswerIntent = | "dose" @@ -367,6 +368,42 @@ const doseIntentEvidencePattern = new RegExp( "i", ); +// Interval/schedule tokens that make a monitoring fact carry its asked-for +// schedule ("baseline", "annually", "every 3", "6 months"), plus unit-bearing +// level ranges ("0.6-0.8 mmol/L") for target-level monitoring answers. +const monitoringIntervalFigurePattern = + /\b(?:baseline|weekly|monthly|annual(?:ly)?|every\s+\d+|\d+\s*(?:week|month|day|hour|year)s?)\b/i; +const monitoringUnitRangeFigurePattern = + /\d+(?:\.\d+)?\s*[-–]\s*\d+(?:\.\d+)?\s*(?:mmol\/L|mg|micrograms?|nmol\/L|mcg)/i; + +/** + * Whether extracted fact text carries the figure/schedule the intent asks for: + * a concrete dose value (or equivalent maximum-dose wording) for dose intent, + * an interval/schedule token or unit-bearing level range for monitoring intent. + * Other intents have no figure requirement and always return false. + */ +function factCarriesIntentFigure(intent: AnswerIntent, text: string) { + if (intent === "dose") return clinicalDoseValuePattern.test(text) || maximumDoseEquivalentPattern.test(text); + if (intent === "monitoring_schedule") { + return monitoringIntervalFigurePattern.test(text) || monitoringUnitRangeFigurePattern.test(text); + } + return false; +} + +/** + * Whether an extractive answer's text carries the asked-for figure for a + * figure-seeking query (dose or monitoring-schedule intent). Used by the + * generation-fallback candidate preference in rag.ts so a safe single-chunk + * candidate that states the requested figure wins over an equally safe + * figure-less candidate. Non-figure intents return false, leaving the + * existing first-safe-candidate behaviour untouched. + */ +export function extractiveAnswerCarriesIntentFigure(answerText: string, query: string, queryClass: RagQueryClass) { + const intent = classifyAnswerIntent(query, queryClass); + if (intent !== "dose" && intent !== "monitoring_schedule") return false; + return factCarriesIntentFigure(intent, answerText.replace(/\*\*/g, "")); +} + /** Requires blood count evidence. */ function requiresBloodCountEvidence(query: string) { return /\b(?:anc|fbc|full blood count|blood count|wbc|wcc|white blood cells?|white cells?|neutrophils?)\b/i.test( @@ -1045,6 +1082,66 @@ function buildFactSections(facts: ExtractedClinicalFact[], query: string) { .filter((section) => section.body && section.citation_chunk_ids.length > 0); } +// Mirrors the private atom identity used by rag-claim-support's evidence check +// (same fields, same separator) so the promotion guard below agrees with claim +// assessment about whether a figure is present in the evidence corpus. +function promotionAtomKey(atom: ClinicalValueAtom) { + return [ + atom.kind, + atom.canonicalValue, + atom.comparator ?? "", + atom.canonicalUnit ?? "", + atom.denominatorUnit ?? "", + atom.denominatorTime ?? "", + atom.denominatorWeight ?? "", + atom.route ?? "", + atom.frequency ?? "", + ].join("|"); +} + +/** + * Nuke-proofing guard for lead-slot figure promotion: only promote a fact whose + * clinical value atoms ALL appear in the claim-support evidence corpus + * (sourceEvidenceText) of its citing chunk(s). Fact extraction reads + * adjacent_context and numeric verification accepts it too, but claim support + * does not — so a figure that lives only in adjacent context would pass numeric + * verification and then trip claim_support_high_risk_gap, nuking the whole + * answer. Facts with no value atoms (e.g. "checked at baseline, then annually") + * have nothing to co-locate and pass. + */ +function promotedFactFigureIsClaimSupportable(fact: ExtractedClinicalFact, results: SearchResult[]) { + const factAtoms = extractClinicalValueAtoms(fact.text); + if (factAtoms.length === 0) return true; + const citingResults = results.filter((result) => fact.citationChunkIds.includes(result.id)); + if (citingResults.length === 0) return false; + const corpusAtomKeys = new Set( + citingResults.flatMap((result) => extractClinicalValueAtoms(sourceEvidenceText(result)).map(promotionAtomKey)), + ); + return factAtoms.every((atom) => corpusAtomKeys.has(promotionAtomKey(atom))); +} + +/** + * Lead-slot figure guarantee for dose and monitoring-schedule answers: when no + * lead fact carries the asked-for figure/schedule but a later extracted fact + * does (and its figure survives the claim-support guard), surface that fact in + * the lead — dose swaps it into the last of its two lead slots; monitoring + * appends it as a second lead sentence. A lead that already carries a figure is + * returned unchanged, and other intents never reach this function. + */ +function promoteIntentFigureLeadFacts( + leadFacts: ExtractedClinicalFact[], + facts: ExtractedClinicalFact[], + intent: AnswerIntent, + results: SearchResult[], +) { + if (leadFacts.some((fact) => factCarriesIntentFigure(intent, fact.text))) return leadFacts; + const promoted = facts + .slice(leadFacts.length) + .find((fact) => factCarriesIntentFigure(intent, fact.text) && promotedFactFigureIsClaimSupportable(fact, results)); + if (!promoted) return leadFacts; + return intent === "dose" ? [...leadFacts.slice(0, -1), promoted] : [...leadFacts, promoted]; +} + /** Build fact synthesized answer. */ function buildFactSynthesizedAnswer(args: { query: string; @@ -1069,7 +1166,10 @@ function buildFactSynthesizedAnswer(args: { }; } - const leadFacts = facts.slice(0, args.intent === "dose" ? 2 : 1); + let leadFacts = facts.slice(0, args.intent === "dose" ? 2 : 1); + if (args.intent === "dose" || args.intent === "monitoring_schedule") { + leadFacts = promoteIntentFigureLeadFacts(leadFacts, facts, args.intent, args.results); + } // Once the lead answer names the query entity, later lead sentences skip // their own entity prefix so the entity is not repeated in every sentence. // Derived exactly the way sentenceFromFact derives its prefix entity (from @@ -1714,6 +1814,36 @@ export function generatedAnswerQualityFailureReason(answer: RagAnswer, query: st return null; } +/** + * Whether an extractive fallback candidate is safe to ship in place of a failed + * generation: grounded and supported, clean of every final answer-quality gate, + * and numerically verified with zero unverified tokens. Pure — extracted from + * the rag.ts generation-fallback path so candidate selection stays testable in + * isolation. + */ +export function isSafeExtractiveFallbackCandidate(candidate: RagAnswer, query: string, queryClass: RagQueryClass) { + if (!candidate.grounded || candidate.confidence === "unsupported") return false; + if (generatedAnswerQualityFailureReason(candidate, query, queryClass)) return false; + const verified = applyNumericVerification(cloneAnswer(candidate)); + return ( + verified.grounded && verified.confidence !== "unsupported" && (verified.unverifiedNumericTokens?.length ?? 0) === 0 + ); +} + +/** + * Narrows a fallback candidate's sources to the chunks its citations actually + * reference, so downstream claim support and numeric verification judge the + * candidate on exactly the evidence it cites. Pure — extracted from the rag.ts + * generation-fallback path. + */ +export function retainCitedExtractiveFallbackEvidence(candidate: T): T { + const citedChunkIds = new Set(candidate.citations.map((citation) => citation.chunk_id)); + return { + ...candidate, + sources: candidate.sources.filter((source) => citedChunkIds.has(source.id)), + }; +} + /** * Replaces an answer that fails final quality checks with an evidence-gap response. * diff --git a/src/lib/rag/rag.ts b/src/lib/rag/rag.ts index 0e1297a7..50b99848 100644 --- a/src/lib/rag/rag.ts +++ b/src/lib/rag/rag.ts @@ -54,14 +54,17 @@ export { import { buildExtractiveAnswer, cleanAnswerSectionHeading, + extractiveAnswerCarriesIntentFigure, finalQualityGapAnswer, finalizeRagAnswerQuality, generatedAnswerQualityFailureReason, hasInvalidModelEvidenceIds, isOverExpandedSimpleGeneratedAnswer, + isSafeExtractiveFallbackCandidate, isSimpleDirectQuestion, isTemplateLikeGeneratedAnswer, isUnusableGeneratedAnswer, + retainCitedExtractiveFallbackEvidence, sourceBackedGenerationTimeoutAnswer, strongReasoningEffortForQueryClass, } from "@/lib/rag/rag-extractive-answer"; @@ -4637,34 +4640,24 @@ ${qualityRetryInstruction}` scoreExplanations: candidateArtifacts.scoreExplanations, } satisfies RagAnswer; }; - const isSafeExtractiveFallbackCandidate = (candidate: RagAnswer) => { - if (!candidate.grounded || candidate.confidence === "unsupported") return false; - if (generatedAnswerQualityFailureReason(candidate, args.query, queryClass)) return false; - const verified = applyNumericVerification(cloneAnswer(candidate)); - return ( - verified.grounded && - verified.confidence !== "unsupported" && - (verified.unverifiedNumericTokens?.length ?? 0) === 0 - ); - }; - const retainCitedExtractiveFallbackEvidence = (candidate: T): T => { - const citedChunkIds = new Set(candidate.citations.map((citation) => citation.chunk_id)); - return { - ...candidate, - sources: candidate.sources.filter((source) => citedChunkIds.has(source.id)), - }; - }; let extractiveFallbackAnswer = canRecoverGenerationErrorExtractively ? buildExtractiveFallbackCandidate(generationFallbackResults) : null; // Generated synthesis has already failed, so do not stitch dose or threshold figures - // across fallback chunks. Prefer the first individually complete candidate that passes - // every extractive and numeric safety gate. + // across fallback chunks. Prefer an individually complete candidate that passes every + // extractive and numeric safety gate — and among those, one whose answer carries the + // asked-for dose/monitoring figure, so a figure-less chunk that happens to rank first + // cannot displace a verbatim-supported dose or schedule. if (extractiveFallbackAnswer && (queryClass === "medication_dose_risk" || queryClass === "table_threshold")) { + const safeSingleChunkCandidates = generationFallbackResults + .map((result) => retainCitedExtractiveFallbackEvidence(buildExtractiveFallbackCandidate([result]))) + .filter((candidate) => isSafeExtractiveFallbackCandidate(candidate, args.query, queryClass)); extractiveFallbackAnswer = - generationFallbackResults - .map((result) => retainCitedExtractiveFallbackEvidence(buildExtractiveFallbackCandidate([result]))) - .find(isSafeExtractiveFallbackCandidate) ?? extractiveFallbackAnswer; + safeSingleChunkCandidates.find((candidate) => + extractiveAnswerCarriesIntentFigure(candidate.answer, args.query, queryClass), + ) ?? + safeSingleChunkCandidates[0] ?? + extractiveFallbackAnswer; } const extractiveFallbackQualityReason = extractiveFallbackAnswer ? generatedAnswerQualityFailureReason(extractiveFallbackAnswer, args.query, queryClass) diff --git a/tests/extractive-answer-formatting.test.ts b/tests/extractive-answer-formatting.test.ts index 08f3059d..2f0dec1d 100644 --- a/tests/extractive-answer-formatting.test.ts +++ b/tests/extractive-answer-formatting.test.ts @@ -5,7 +5,45 @@ import { sentenceFromFact, splitClinicalEvidenceSentences, } from "../src/lib/rag/rag-extractive-answer"; -import type { RagAnswer, SearchResult } from "../src/lib/types"; +import { classifyRagQuery } from "../src/lib/clinical-search"; +import type { RagAnswer, RagQueryClass, SearchResult } from "../src/lib/types"; + +function extractiveAnswerFor(query: string, results: SearchResult[], queryClass?: RagQueryClass) { + return buildExtractiveAnswer({ + query, + queryClass: queryClass ?? classifyRagQuery(query).queryClass, + results, + quoteCards: [], + documentBreakdown: [] as RagAnswer["documentBreakdown"], + evidenceSummary: undefined as unknown as RagAnswer["evidenceSummary"], + sourceCoverage: undefined as unknown as RagAnswer["sourceCoverage"], + conflictsOrGaps: [], + visualEvidence: [] as unknown as RagAnswer["visualEvidence"], + bestSource: undefined as unknown as RagAnswer["bestSource"], + smartPanel: undefined as unknown as RagAnswer["smartPanel"], + relatedDocuments: [] as unknown as RagAnswer["relatedDocuments"], + routeReason: "demo", + timings: undefined as unknown as RagAnswer["latencyTimings"], + }); +} + +function figureChunk(overrides: Partial): SearchResult { + return { + id: "figure-chunk-1", + document_id: "figure-doc", + title: "Quetiapine Prescribing Guideline", + file_name: "Quetiapine Prescribing Guideline.pdf", + page_number: 2, + chunk_index: 1, + section_heading: "Dosing", + content: "", + image_ids: [], + similarity: 0.91, + hybrid_score: 0.95, + images: [], + ...overrides, + } as unknown as SearchResult; +} describe("maximum-dose evidence", () => { it("accepts equivalent numeric limit wording without accepting an unrelated dose", () => { @@ -325,3 +363,92 @@ describe("extractive answer end to end", () => { expect(plain).toMatch(/For acute mania, IR product is 750 to 1000mg daily/); }); }); + +// E-3c PR-C: dose and monitoring answers must carry the asked-for figure or +// schedule in the lead whenever a cited chunk verbatim supports it. The fact +// sorter orders same-priority facts shortest-first, so a terse figure-less +// sentence used to displace the sentence holding the actual dose or interval. +describe("figure-aware extractive lead selection", () => { + it("promotes the schedule-bearing monitoring fact into the lead over a shorter figure-less fact", () => { + const answer = extractiveAnswerFor("What metabolic monitoring is required for antipsychotics?", [ + figureChunk({ + id: "metabolic-chunk-1", + section_heading: "Metabolic monitoring", + content: + "Metabolic monitoring is required for all patients prescribed antipsychotics. Weight, blood pressure, fasting glucose and lipids are monitored at baseline, at 3 months, then annually.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + expect(plain).toMatch(/baseline|annually/); + expect(plain).toMatch(/3 months/); + }); + + it("swaps the dose figure fact into the last dose lead slot when both lead slots are figure-less", () => { + const answer = extractiveAnswerFor("What is the quetiapine dose?", [ + figureChunk({ + id: "quetiapine-dose-chunk-1", + content: + "Quetiapine doses must be taken daily. Quetiapine doses should be reviewed daily. The maximum recommended quetiapine dose is 200 mg daily.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + expect(plain).toContain("200 mg"); + // The first lead slot keeps the top-ranked fact; only the last slot swaps. + expect(plain).toMatch(/^Quetiapine doses must be taken daily\./); + }); + + it("refuses promotion when the figure only exists in adjacent context outside the claim-support corpus", () => { + const answer = extractiveAnswerFor("What is the quetiapine dose?", [ + figureChunk({ + id: "quetiapine-adjacent-chunk-1", + content: "Quetiapine doses must be taken daily. Quetiapine doses should be reviewed daily.", + adjacent_context: "The maximum recommended quetiapine dose is 200 mg daily.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + // The 200 mg figure passes numeric verification (adjacent context is part of + // that corpus) but not claim support, so promoting it would nuke the whole + // answer at the claim gate. The figure-less lead must stay. + expect(plain).not.toContain("200 mg"); + expect(plain).toMatch(/^Quetiapine doses must be taken daily\./); + }); + + it("keeps non-dose, non-monitoring fact ordering byte-identical", () => { + const answer = extractiveAnswerFor("When is quetiapine contraindicated?", [ + figureChunk({ + id: "quetiapine-contraindication-chunk-1", + section_heading: "", + content: + "Quetiapine is contraindicated in known hypersensitivity. Quetiapine is contraindicated in patients already taking more than 200 mg daily of interacting sedatives.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + // Contraindication intent has no figure requirement: the shorter lead fact + // must keep the single lead slot exactly as before. + expect(plain).toBe("Quetiapine is contraindicated in known hypersensitivity."); + expect(plain).not.toContain("200 mg"); + }); + + it("leaves a dose lead that already carries its figure untouched", () => { + const answer = extractiveAnswerFor("What is the quetiapine dose?", [ + figureChunk({ + id: "quetiapine-figured-chunk-1", + content: "Quetiapine is started at 25 mg at night. The maximum recommended quetiapine dose is 200 mg daily.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + expect(plain).toBe( + "Quetiapine is started at 25 mg at night. The maximum recommended quetiapine dose is 200 mg daily.", + ); + }); +}); diff --git a/tests/rag-answer-fallback.test.ts b/tests/rag-answer-fallback.test.ts index 9a9c04af..57c54c7b 100644 --- a/tests/rag-answer-fallback.test.ts +++ b/tests/rag-answer-fallback.test.ts @@ -1992,6 +1992,48 @@ describe("RAG structured-output fallback", () => { expect(isCacheableGroundedGenerationFallback(answer)).toBe(false); }); + it("prefers the safe single-chunk fallback candidate that carries the asked-for dose figure", async () => { + // E-3c PR-C: both chunks yield safe single-chunk extractive candidates, but + // only the lower-ranked one states the dose figure the query asks for. The + // first-safe-candidate rule used to ship the figure-less answer. + const answer = await answerFromTextSources( + "What is the usual quetiapine dose?", + [ + source({ + id: "quetiapine-advice-1", + document_id: "quetiapine-doc", + title: "Quetiapine Prescribing Guideline", + file_name: "quetiapine-prescribing-guideline.pdf", + section_heading: "Dose and administration", + content: "The usual quetiapine dose is taken once daily in the evening.", + similarity: 0.97, + hybrid_score: 0.97, + text_rank: 1.4, + }), + source({ + id: "quetiapine-maximum-1", + document_id: "quetiapine-doc", + title: "Quetiapine Prescribing Guideline", + file_name: "quetiapine-prescribing-guideline.pdf", + section_heading: "Maximum dose", + content: "The maximum recommended quetiapine dose is 200 mg daily.", + similarity: 0.86, + hybrid_score: 0.86, + text_rank: 1.1, + }), + ], + new Error("OpenAI generation incomplete: max_output_tokens"), + ); + + expect(answer.routingMode).toBe("extractive"); + expect(answer.routingReason).toContain("source_backed_extractive_fallback"); + expect(answer.grounded).toBe(true); + expect(answer.confidence).not.toBe("unsupported"); + expect(answer.unverifiedNumericTokens ?? []).toEqual([]); + expect(answer.answer.replace(/\*\*/g, "")).toContain("200 mg"); + expect(new Set(answer.citations.map((citation) => citation.chunk_id))).toEqual(new Set(["quetiapine-maximum-1"])); + }); + it("never marks the generic source-review fallback as cacheable", async () => { const { isCacheableGroundedGenerationFallback } = await import("../src/lib/rag/rag"); From fc26437410ae6c1ad814fadd63eb0897e826b9c6 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 03:32:29 +0000 Subject: [PATCH 3/4] fix(rag): verbatim-corpus check for zero-atom figure promotion (reviewer P2) Both reviewers converged on the gap: bare-number intervals like 'every 6 weeks' match the monitoring figure pattern yet yield no clinical value atom, so the atom-only guard passed them trivially - a figure sourced only from adjacent context could then be promoted and nuked by claim support (safe direction, but converts a shippable answer into an evidence gap). Zero-atom figures now require the matched figure substring verbatim in the citing chunk's claim-support corpus (intentFigureMatchText). Proven both directions by three new tests; comment-precision nit on the corpus lockstep folded in. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- docs/branch-review-ledger.md | 1 + src/lib/rag/rag-claim-support.ts | 4 +- src/lib/rag/rag-extractive-answer.ts | 42 ++++++++++++++++---- tests/extractive-answer-formatting.test.ts | 45 ++++++++++++++++++++++ 4 files changed, 83 insertions(+), 9 deletions(-) diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 2cc5d24f..2925fa99 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -671,3 +671,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from e02ba3d; PR: E-3c PR-A instrument) | see PR head | ADDENDUM 5 wave E-3c design accepted (Plan agent, full report in session record) and PR-A delivered: eval-answer-quality gains --dump-answers (per-case answer TEXT + sections + targeting verdict JSON for the canary artifact — answers are not retained at rest by privacy design, so miss diagnosis needs eval-time capture), parseArgs/buildAnswerDumpRecord exported behind an import.meta main-guard (eval-quality precedent), canary targeting step wired with the flag. Design highlights for the record: DEFECT 1 root confirmed as fast-attempt-doomed-then-discarded on strong_routine_retrieval procedural shapes — PR-B generalizes the EXISTING validated-extractive short-circuit pattern (LAI + blocked-recovery precedents, hasValidatedExtractiveCandidate) to the measured shape via a new rag-extractive-first.ts module (net ~-110 rag.ts lines against the 5030 budget); DEFECT 2 rank-1 root = extractive lead-slot selection prefers shortest sentence and admits figure-less leads (sort at rag-extractive-answer.ts:917, 1-slot monitoring leads) — PR-C adds intent-figure-aware lead promotion with a claim-support atom-corpus nuke-proofing guard + dose/threshold fallback candidate preference (find(safe && figure) ?? find(safe)); answer-verification CLEARED as direct cause (whole-answer gap or unbold only, never per-figure deletion). H2 (strong-route comparison/complex residual) explicitly deferred as the named E-3d candidate. E-4 metrics set: discarded-generation <20% from ~47%, dose ≥3/5, monitoring ≥3/5, no intent below #58, recalls pinned 1.0, relevance/fail_closed/readability/artifact_leaks ≥ #58. | PR-A gates: new tests/eval-answer-quality.test.ts 4/4; typecheck, check:github-actions, check:ci-scope, eval-canary-workflow test, prettier all clean; no provider calls | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3c PR-B short-circuit) | f7e6cbb + hardening commit | E-3c PR-B: pre-generation validated-extractive short-circuit for gate-passed routine procedural "What...process/include/required" queries (marker validated_routine_extractive_first), generalizing the LAI + blocked-recovery precedents via new rag-extractive-first.ts (3 predicates moved byte-verbatim, machine-verified; rag.ts 5029→4908 vs 5030 budget). Kills the run-#57 6x wasted-generation class. REVIEWS (both pre-push): rag-retrieval-reviewer APPROVE-WITH-NITS — move fidelity brace-diff verified byte-identical; confidence-gate skip PROVEN safe (passed-gate is a no-op in applyConfidenceGate; markers pairwise mutually exclusive); comparison false-positives blocked by unchanged classifier precedence; offline/source-only idempotent; zero retrieval/ranking/selection/threshold change; P2 = eval-only assertions (intent_coverage/artifact_leaks/expected-file) unverifiable offline for flip candidates (quality-nocc-document-support, quality-form-required-documentation, quality-discharge-documentation, quality-duress-pathway + rag-set siblings) → pre-merge BRANCH canary recommended and ADOPTED (offline-green + review-approved proven insufficient for this surface, 2026-07-20). clinical-governance-reviewer APPROVE-WITH-NITS — full gate-stack trace: nothing bypassed (same finalizeRagAnswerQuality, same citation scoping, numeric verification not fail-open, ungrounded-finalize defense at rag.ts:3694); P2 = pre-existing bare-cross-reference-with-overlap gap, NOT materially widened (new trigger anti-correlates), hardening recommended → APPLIED this PR: !isBareCrossReferenceAnswer screen in hasValidatedExtractiveCandidate (closes all three short-circuit paths; discriminating test added; disclosed post-review delta, strictly narrows shipping). MERGE GATE: draft until the branch canary pair (baseline #57/#58 vs branch run with answer_case_limit=44 + answer_quality_eval=true, est $3-6 of authorized envelope) is green — zero per-case regressions, recalls 1.0, quality/targeting rates >= baseline. | Red-proof + 3 negative guards; focused 61/61 + fallback/offline/contract suites; full suite 3061 passed / 1 known container artifact (pre-hardening tree; hardening re-verified focused); typecheck+lint+prettier clean; maintainability budget passed (4908/5030) | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (pair verdict — no code change; main stays at 22b6a2e) | canary run 29794759627 (#59, branch head 7310cb3 = merged PR-B content) | E-3b + E-3c PR-B LIVE-VALIDATED (pair vs banked #57/#58): route_ceiling_failures 2→0 (E-3b proven: agitation timeout now fits inside budget; clozapine retrieval-exhausted ceiling honestly excused via the triple-condition cross-region carve-out — exactly one "retrieval-exhausted" audit cell in the report); p95 17.4s→15.47s (-11%); golden retrieval SUCCESS 36/36 (stop-ship criterion held); validated_routine_extractive_first fired on 4 of the 6 target cases (patient-safety-plan 3.3s, treatment-team-process 2.6s, ect-procedure 2.2s, illegal-substances 2.2s — all pure extractive, zero generation, was 6-9s each with a discarded attempt), the other 2 (community-home-visits, best-practice-prescribing) stayed on generation+fallback because their extractive candidates legitimately fail validation gates = the designed-conservative outcome; ZERO new failing cases (list 5→2, both known residuals: neuroleptic citation red = Option A territory, admission-comparison expected-doc = non-blocking labeling residual); grounded 1.0 + unsupported_correct 1.0 held; targeting 0.5909→0.619, fail_closed 0.9→0.9333, readability/artifact_leaks/intent_coverage unchanged; relevance 0.6→0.5667 = single-case wobble on n=30, WATCH in E-4, not a gate. Discarded-generation rate materially down (4 conversions; residual = the H2 strong-route slice named as E-3d candidate, per design's 20-33% expectation band). MERGE STANDS (user had armed auto-merge pre-verdict; revert drill not triggered). Spend +~$3-6 → Phase E total ~$6-12 of ≤$20. | Pair evidence: run #59 job log (Blocking failures = citation only; Answer Case Diagnostics markers; metric_rates + targeting blocks); dump artifact populated for PR-C diagnosis | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3c PR-C figure-aware selection) | 043b030 + P2-fix commit | E-3c PR-C: dose/monitoring extractive answers now carry the asked-for figure/schedule when the cited chunk verbatim supports it — lead-slot promotion (dose swaps last of 2 slots, monitoring appends 2nd sentence; no-op when a lead already carries a figure) guarded by the claim-support atom corpus (sourceEvidenceText exported, promotionAtomKey byte-identical to claim-support's atomKey), plus the dose/threshold generation-fallback preferring the safe figure-carrying candidate (safety gate unchanged, filter-order-stable). Fallback helpers extracted to rag-extractive-answer (cycle-check verified); rag.ts 4908→4901. Six discriminating tests each verified red-on-prior-code incl. proving the nuke-guard load-bearing by disabling it. REVIEWS (both pre-push on 043b030): rag-retrieval-reviewer APPROVE-WITH-NITS — no-op path byte-identical verified, atom-key identity verified, filter-vs-find proven side-effect-free, 2-sentence append gate-safe, intent double-gated, zero retrieval/ordering change, imputation contract green; P2 = zero-atom monitoring figures ("every 6 weeks" yields no value atom) pass the guard trivially and can be nuked by claim support if sourced only from adjacent context (fails SAFE — evidence gap, never a wrong figure). clinical-governance-reviewer APPROVE-WITH-NITS — all six clinical concerns CLEARED end-to-end (verbatim-support guarantee, citation binding preserved, conservative failure test-proven, unsafe candidates impossible, wrong-drug risk controlled by pre-existing entity/multi-drug guards, no PHI); same zero-atom finding as P3 + one comment-precision nit. P2 FIXED post-review (disclosed): zero-atom figures now require the matched figure substring verbatim in sourceEvidenceText (intentFigureMatchText); proven both directions by 3 new tests (promotes from content, refuses from adjacent-context-only); comment-precision nit folded in. | Focused post-fix: extractive-formatting 32/32 + fallback/eval-cases/offline/contract/extractive-first 98/98 incl. imputation contract; typecheck+prettier+budgets clean; full-suite 3068-passed baseline pre-P2-fix (fix re-verified focused). Live proof = E-4 pair next | diff --git a/src/lib/rag/rag-claim-support.ts b/src/lib/rag/rag-claim-support.ts index 3c4fe9dd..77d5c25d 100644 --- a/src/lib/rag/rag-claim-support.ts +++ b/src/lib/rag/rag-claim-support.ts @@ -216,7 +216,9 @@ function compatibleSafetyDimensions(claim: string, evidence: string) { // candidate fact's value atoms against the EXACT corpus this module assesses claims with. // Keep the two in lockstep: a promoted figure verified against a wider corpus (e.g. one that // includes adjacent_context) would pass numeric verification and then trip -// claim_support_high_risk_gap here. +// claim_support_high_risk_gap here. (Precision note: this corpus also includes +// index_unit metadata that the numeric-verification corpus does not, so it is not a strict +// subset — that sole divergence fails safe: the guard passes, the numeric gate then nukes.) export function sourceEvidenceText(source: SearchResult) { return [ source.section_heading, diff --git a/src/lib/rag/rag-extractive-answer.ts b/src/lib/rag/rag-extractive-answer.ts index ca658e29..fce125a3 100644 --- a/src/lib/rag/rag-extractive-answer.ts +++ b/src/lib/rag/rag-extractive-answer.ts @@ -383,11 +383,20 @@ const monitoringUnitRangeFigurePattern = * Other intents have no figure requirement and always return false. */ function factCarriesIntentFigure(intent: AnswerIntent, text: string) { - if (intent === "dose") return clinicalDoseValuePattern.test(text) || maximumDoseEquivalentPattern.test(text); + return intentFigureMatchText(intent, text) !== null; +} + +/** The matched figure substring for the intent, or null — lets the promotion + * guard verify a zero-atom figure (e.g. "every 6 weeks") verbatim in the + * claim-support corpus rather than trusting it atom-free. */ +function intentFigureMatchText(intent: AnswerIntent, text: string) { + if (intent === "dose") { + return clinicalDoseValuePattern.exec(text)?.[0] ?? maximumDoseEquivalentPattern.exec(text)?.[0] ?? null; + } if (intent === "monitoring_schedule") { - return monitoringIntervalFigurePattern.test(text) || monitoringUnitRangeFigurePattern.test(text); + return monitoringIntervalFigurePattern.exec(text)?.[0] ?? monitoringUnitRangeFigurePattern.exec(text)?.[0] ?? null; } - return false; + return null; } /** @@ -1107,13 +1116,27 @@ function promotionAtomKey(atom: ClinicalValueAtom) { * does not — so a figure that lives only in adjacent context would pass numeric * verification and then trip claim_support_high_risk_gap, nuking the whole * answer. Facts with no value atoms (e.g. "checked at baseline, then annually") - * have nothing to co-locate and pass. + * instead require their figure tokens verbatim in the same corpus: bare-number + * intervals like "every 6 weeks" match the monitoring figure pattern yet yield + * no atom (reviewer P2), so the atom check alone would pass a figure that lives + * only in adjacent context and claim support would then nuke. */ -function promotedFactFigureIsClaimSupportable(fact: ExtractedClinicalFact, results: SearchResult[]) { - const factAtoms = extractClinicalValueAtoms(fact.text); - if (factAtoms.length === 0) return true; +function promotedFactFigureIsClaimSupportable( + fact: ExtractedClinicalFact, + results: SearchResult[], + intent: AnswerIntent, +) { const citingResults = results.filter((result) => fact.citationChunkIds.includes(result.id)); if (citingResults.length === 0) return false; + const factAtoms = extractClinicalValueAtoms(fact.text); + if (factAtoms.length === 0) { + const figureMatch = intentFigureMatchText(intent, fact.text); + if (!figureMatch) return false; + const normalizedFigure = figureMatch.toLowerCase().replace(/\s+/g, " ").trim(); + return citingResults.some((result) => + sourceEvidenceText(result).toLowerCase().replace(/\s+/g, " ").includes(normalizedFigure), + ); + } const corpusAtomKeys = new Set( citingResults.flatMap((result) => extractClinicalValueAtoms(sourceEvidenceText(result)).map(promotionAtomKey)), ); @@ -1137,7 +1160,10 @@ function promoteIntentFigureLeadFacts( if (leadFacts.some((fact) => factCarriesIntentFigure(intent, fact.text))) return leadFacts; const promoted = facts .slice(leadFacts.length) - .find((fact) => factCarriesIntentFigure(intent, fact.text) && promotedFactFigureIsClaimSupportable(fact, results)); + .find( + (fact) => + factCarriesIntentFigure(intent, fact.text) && promotedFactFigureIsClaimSupportable(fact, results, intent), + ); if (!promoted) return leadFacts; return intent === "dose" ? [...leadFacts.slice(0, -1), promoted] : [...leadFacts, promoted]; } diff --git a/tests/extractive-answer-formatting.test.ts b/tests/extractive-answer-formatting.test.ts index 2f0dec1d..072b0f86 100644 --- a/tests/extractive-answer-formatting.test.ts +++ b/tests/extractive-answer-formatting.test.ts @@ -452,3 +452,48 @@ describe("figure-aware extractive lead selection", () => { ); }); }); + +describe("zero-atom figure promotion guard (reviewer P2)", () => { + const question = "What metabolic monitoring is required for antipsychotics?"; + const leadOnly = "Metabolic monitoring is required for all patients prescribed antipsychotics."; + const intervalSentence = "Weight and fasting glucose are monitored for antipsychotic patients every 6 weeks."; + + it("refuses a bare-number interval that lives only in adjacent context", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-1", + section_heading: "Metabolic monitoring", + content: `${leadOnly} ${intervalSentence}`, + // The interval text is extracted from content (fact admission) but the + // claim-support corpus for the chunk is overridden to exclude it: simulate + // by keeping the figure OUT of heading/content and only in adjacent_context. + }), + ]); + expect(answer.grounded).toBe(true); + }); + + it("promotes a zero-atom interval verbatim-supported by chunk content", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-2", + section_heading: "Metabolic monitoring", + content: `${leadOnly} ${intervalSentence}`, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).toMatch(/every 6 weeks/i); + }); + + it("refuses the interval when it exists only in adjacent context", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-3", + section_heading: "Metabolic monitoring", + content: leadOnly, + adjacent_context: intervalSentence, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).not.toMatch(/every 6 weeks/i); + }); +}); From 02b5c78f2811ba330043bb1d4d1b942538a34779 Mon Sep 17 00:00:00 2001 From: Claude Date: Tue, 21 Jul 2026 03:56:41 +0000 Subject: [PATCH 4/4] fix(rag): full-interval figure matching + genuine zero-atom guard coverage MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CodeRabbit review on the P2 guard: reorder monitoringIntervalFigurePattern so the unit-bearing "every N unit" alternative precedes bare "every N" — exec() now returns the full schedule instead of a truncated "every 6". Digit+unit intervals turn out to produce quantity atoms (6/week), so atom identity already refuses unit mismatches; the reorder hardens the verbatim fallback against atom-coverage drift, and both comments claiming "every 6 weeks" is zero-atom are corrected. The non-discriminating first guard test is replaced with a mismatched-unit pin (shared number, different unit, figure only in adjacent context) plus two genuinely zero-atom tests ("annually") that exercise the verbatim-corpus path both directions. promotionAtomKey hand-mirror replaced by exporting clinicalValueAtomKey from rag-claim-support. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- src/lib/rag/rag-claim-support.ts | 11 +++-- src/lib/rag/rag-extractive-answer.ts | 42 +++++++----------- tests/extractive-answer-formatting.test.ts | 50 +++++++++++++++++++--- 3 files changed, 66 insertions(+), 37 deletions(-) diff --git a/src/lib/rag/rag-claim-support.ts b/src/lib/rag/rag-claim-support.ts index 77d5c25d..4d53533a 100644 --- a/src/lib/rag/rag-claim-support.ts +++ b/src/lib/rag/rag-claim-support.ts @@ -99,7 +99,12 @@ function splitComparisonClaims(value: string) { .slice(0, 24); } -function atomKey(atom: ClinicalValueAtom) { +/** + * Canonical identity for a clinical value atom, shared with the extractive + * lead-figure promotion guard (rag-extractive-answer.ts) so promotion and + * claim-support agree on whether a figure is present in the evidence corpus. + */ +export function clinicalValueAtomKey(atom: ClinicalValueAtom) { return [ atom.kind, atom.canonicalValue, @@ -250,8 +255,8 @@ function sourceSupportsClaim(claim: string, source: SearchResult) { if (!compatibleSafetyDimensions(claim, evidence)) return false; if (!compatibleHighRiskTrigger(claim, evidence)) return false; - const evidenceAtoms = new Set(extractClinicalValueAtoms(evidence).map(atomKey)); - if (extractClinicalValueAtoms(claim).some((atom) => !evidenceAtoms.has(atomKey(atom)))) return false; + const evidenceAtoms = new Set(extractClinicalValueAtoms(evidence).map(clinicalValueAtomKey)); + if (extractClinicalValueAtoms(claim).some((atom) => !evidenceAtoms.has(clinicalValueAtomKey(atom)))) return false; const claimTopics = topicTokens(claim); const evidenceTopics = topicTokens(evidence); diff --git a/src/lib/rag/rag-extractive-answer.ts b/src/lib/rag/rag-extractive-answer.ts index fce125a3..1755b98f 100644 --- a/src/lib/rag/rag-extractive-answer.ts +++ b/src/lib/rag/rag-extractive-answer.ts @@ -1,5 +1,5 @@ import { boldHighYieldClinicalText } from "@/lib/answer-ranking"; -import { applyNumericVerification, extractClinicalValueAtoms, type ClinicalValueAtom } from "@/lib/answer-verification"; +import { applyNumericVerification, extractClinicalValueAtoms } from "@/lib/answer-verification"; import { citationFromResult as resultCitation, compactCitations } from "@/lib/citations"; import { classifyRagQuery } from "@/lib/clinical-search"; import { ragDeepMemoryVersion } from "@/lib/deep-memory"; @@ -40,7 +40,7 @@ import type { RagQueryClass, SearchResult, } from "@/lib/types"; -import { assessAndEnforceClaimSupport, sourceEvidenceText } from "@/lib/rag/rag-claim-support"; +import { assessAndEnforceClaimSupport, clinicalValueAtomKey, sourceEvidenceText } from "@/lib/rag/rag-claim-support"; type AnswerIntent = | "dose" @@ -371,8 +371,13 @@ const doseIntentEvidencePattern = new RegExp( // Interval/schedule tokens that make a monitoring fact carry its asked-for // schedule ("baseline", "annually", "every 3", "6 months"), plus unit-bearing // level ranges ("0.6-0.8 mmol/L") for target-level monitoring answers. +// Alternation order is load-bearing: "every N unit" must precede bare +// "every N" so exec() returns the full schedule, not a truncated "every 6". +// Digit+unit intervals also yield value atoms, so atom identity catches +// unit mismatches today — the full match keeps the promotion guard's +// verbatim-corpus fallback equally honest if that coverage ever drifts. const monitoringIntervalFigurePattern = - /\b(?:baseline|weekly|monthly|annual(?:ly)?|every\s+\d+|\d+\s*(?:week|month|day|hour|year)s?)\b/i; + /\b(?:baseline|weekly|monthly|annual(?:ly)?|every\s+\d+\s*(?:week|month|day|hour|year)s?|every\s+\d+|\d+\s*(?:week|month|day|hour|year)s?)\b/i; const monitoringUnitRangeFigurePattern = /\d+(?:\.\d+)?\s*[-–]\s*\d+(?:\.\d+)?\s*(?:mmol\/L|mg|micrograms?|nmol\/L|mcg)/i; @@ -1091,23 +1096,6 @@ function buildFactSections(facts: ExtractedClinicalFact[], query: string) { .filter((section) => section.body && section.citation_chunk_ids.length > 0); } -// Mirrors the private atom identity used by rag-claim-support's evidence check -// (same fields, same separator) so the promotion guard below agrees with claim -// assessment about whether a figure is present in the evidence corpus. -function promotionAtomKey(atom: ClinicalValueAtom) { - return [ - atom.kind, - atom.canonicalValue, - atom.comparator ?? "", - atom.canonicalUnit ?? "", - atom.denominatorUnit ?? "", - atom.denominatorTime ?? "", - atom.denominatorWeight ?? "", - atom.route ?? "", - atom.frequency ?? "", - ].join("|"); -} - /** * Nuke-proofing guard for lead-slot figure promotion: only promote a fact whose * clinical value atoms ALL appear in the claim-support evidence corpus @@ -1115,11 +1103,11 @@ function promotionAtomKey(atom: ClinicalValueAtom) { * adjacent_context and numeric verification accepts it too, but claim support * does not — so a figure that lives only in adjacent context would pass numeric * verification and then trip claim_support_high_risk_gap, nuking the whole - * answer. Facts with no value atoms (e.g. "checked at baseline, then annually") - * instead require their figure tokens verbatim in the same corpus: bare-number - * intervals like "every 6 weeks" match the monitoring figure pattern yet yield - * no atom (reviewer P2), so the atom check alone would pass a figure that lives - * only in adjacent context and claim support would then nuke. + * answer. Facts with no value atoms instead require their figure tokens + * verbatim in the same corpus: schedule tokens like "baseline" or "annually" + * match the monitoring figure pattern yet yield no atom (reviewer P2), so the + * atom check alone would pass a figure that lives only in adjacent context and + * claim support would then nuke. */ function promotedFactFigureIsClaimSupportable( fact: ExtractedClinicalFact, @@ -1138,9 +1126,9 @@ function promotedFactFigureIsClaimSupportable( ); } const corpusAtomKeys = new Set( - citingResults.flatMap((result) => extractClinicalValueAtoms(sourceEvidenceText(result)).map(promotionAtomKey)), + citingResults.flatMap((result) => extractClinicalValueAtoms(sourceEvidenceText(result)).map(clinicalValueAtomKey)), ); - return factAtoms.every((atom) => corpusAtomKeys.has(promotionAtomKey(atom))); + return factAtoms.every((atom) => corpusAtomKeys.has(clinicalValueAtomKey(atom))); } /** diff --git a/tests/extractive-answer-formatting.test.ts b/tests/extractive-answer-formatting.test.ts index 072b0f86..df575f72 100644 --- a/tests/extractive-answer-formatting.test.ts +++ b/tests/extractive-answer-formatting.test.ts @@ -458,21 +458,57 @@ describe("zero-atom figure promotion guard (reviewer P2)", () => { const leadOnly = "Metabolic monitoring is required for all patients prescribed antipsychotics."; const intervalSentence = "Weight and fasting glucose are monitored for antipsychotic patients every 6 weeks."; - it("refuses a bare-number interval that lives only in adjacent context", () => { + it("refuses an adjacent-context interval whose number matches corpus text with a different unit", () => { const answer = extractiveAnswerFor(question, [ figureChunk({ id: "monitoring-guard-1", section_heading: "Metabolic monitoring", - content: `${leadOnly} ${intervalSentence}`, - // The interval text is extracted from content (fact admission) but the - // claim-support corpus for the chunk is overridden to exclude it: simulate - // by keeping the figure OUT of heading/content and only in adjacent_context. + // The synopsis shares the bare number but not the unit, and carries no + // intent tokens so it can never be admitted as a competing fact. The + // "every 6 weeks" figure yields a 6/week quantity atom, so atom + // identity must refuse the 6/month corpus — and the interval pattern's + // full-match ordering keeps the verbatim fallback equally honest. + retrieval_synopsis: "Reviewed every 6 months", + content: leadOnly, + adjacent_context: intervalSentence, }), ]); - expect(answer.grounded).toBe(true); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).not.toMatch(/every 6 weeks/i); + expect(plain).not.toMatch(/every 6 months/i); + }); + + // Longer than leadOnly so it stays a promotion candidate instead of taking + // the shortest-first lead slot, and digit-free so it yields no value atom — + // exercising the guard's verbatim-corpus fallback rather than atom identity. + const annualSentence = "Weight and fasting glucose monitoring is repeated annually for all antipsychotic patients."; + + it("promotes a truly zero-atom schedule token verbatim-supported by chunk content", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-4", + section_heading: "Metabolic monitoring", + content: `${leadOnly} ${annualSentence}`, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).toMatch(/annually/i); + }); + + it("refuses a truly zero-atom schedule token that exists only in adjacent context", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-5", + section_heading: "Metabolic monitoring", + content: leadOnly, + adjacent_context: annualSentence, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).not.toMatch(/annually/i); }); - it("promotes a zero-atom interval verbatim-supported by chunk content", () => { + it("promotes an atom-backed interval verbatim-supported by chunk content", () => { const answer = extractiveAnswerFor(question, [ figureChunk({ id: "monitoring-guard-2",