diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index cb5a950b..2925fa99 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -670,3 +670,5 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3b budget-aware generation deadlines) | 314d03f (+eb08ec5 review row) | ADDENDUM 5 wave E-3b implemented per the design-agent plan: generation attempts clamped to route budget minus a measured 2s recovery reserve (single call-site, all four attempt kinds); truncation self-heal gated on retry viability (reserve+5s floor) with observable truncation_retry_skipped_budget_reserve marker; additive route_budget_exhausted_by_retrieval runtime flag; eval route-ceiling gains the triple-condition cross-region carve-out (context + runtime flag + zero generation) with retrieval-exhausted audit cells — local/release gates provably strict. Fixes I3 (54ms budget overrun after 22.6s provider timeout), I5 (82s truncation waste class), resolves I2 (clozapine 13.3s retrieval vs 12s runtime budget = geography, now suppressed ONLY in the sanctioned cross-region context with full auditability). Reviewer verdicts: rag-retrieval-reviewer APPROVE-WITH-NITS (2 P3: prod-dead requestTimeoutMs retained for symmetry; report-cell coupling cosmetic; cached-replay invariant PROVEN — budget-exhausted answers never cached, carve-out unreachable via replay; marker isolation proven — SLO counters key on fallback_reason not answer_retry_reasons); clinical-governance-reviewer APPROVE-WITH-NITS (prior row) — internal-timeout→PublicApiError→existing-fallback path verified, all safety gates still applied to recovery answers. ALSO BANKED — E-2 targeting baseline (canary run #58, 29788404357, all-green incl. first execution of the !cancelled()-fixed instrument, ~$1-2): metric_rates relevance 0.6 / readability 1.0 / artifact_leaks 1.0 / intent_coverage 0.9333 / fail_closed 0.9; targeting_rate 0.5909 (13/22); by intent: document_lookup 5/5, red_result_action 3/3, contraindication 2/2, dose 1/5, monitoring_schedule 1/5, pathway_referral 1/2; all 9 misses = missing dose figure/schedule-interval (answer lengths 73-232 chars) → E-3c co-primary target alongside the wasted-generation class. Phase E spend ≈$3-6 of ≤$20. | Red-proofs: reserve pinned 3 independent ways (exact 23000ms grant, deadline flag clear, total under budget); self-heal skip pins exact marker + single provider call; offline flag pinned true/false. Focused: route-budget 9/9, eval-quality 27/27, fallback+offline 52/52, parser/abort regressions 15/15. Full suite 3043 passed / 1 known container pdf artifact. typecheck+lint+prettier clean. No provider calls; live proof = E-4 paired run | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from e02ba3d; PR: E-3c PR-A instrument) | see PR head | ADDENDUM 5 wave E-3c design accepted (Plan agent, full report in session record) and PR-A delivered: eval-answer-quality gains --dump-answers (per-case answer TEXT + sections + targeting verdict JSON for the canary artifact — answers are not retained at rest by privacy design, so miss diagnosis needs eval-time capture), parseArgs/buildAnswerDumpRecord exported behind an import.meta main-guard (eval-quality precedent), canary targeting step wired with the flag. Design highlights for the record: DEFECT 1 root confirmed as fast-attempt-doomed-then-discarded on strong_routine_retrieval procedural shapes — PR-B generalizes the EXISTING validated-extractive short-circuit pattern (LAI + blocked-recovery precedents, hasValidatedExtractiveCandidate) to the measured shape via a new rag-extractive-first.ts module (net ~-110 rag.ts lines against the 5030 budget); DEFECT 2 rank-1 root = extractive lead-slot selection prefers shortest sentence and admits figure-less leads (sort at rag-extractive-answer.ts:917, 1-slot monitoring leads) — PR-C adds intent-figure-aware lead promotion with a claim-support atom-corpus nuke-proofing guard + dose/threshold fallback candidate preference (find(safe && figure) ?? find(safe)); answer-verification CLEARED as direct cause (whole-answer gap or unbold only, never per-figure deletion). H2 (strong-route comparison/complex residual) explicitly deferred as the named E-3d candidate. E-4 metrics set: discarded-generation <20% from ~47%, dose ≥3/5, monitoring ≥3/5, no intent below #58, recalls pinned 1.0, relevance/fail_closed/readability/artifact_leaks ≥ #58. | PR-A gates: new tests/eval-answer-quality.test.ts 4/4; typecheck, check:github-actions, check:ci-scope, eval-canary-workflow test, prettier all clean; no provider calls | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3c PR-B short-circuit) | f7e6cbb + hardening commit | E-3c PR-B: pre-generation validated-extractive short-circuit for gate-passed routine procedural "What...process/include/required" queries (marker validated_routine_extractive_first), generalizing the LAI + blocked-recovery precedents via new rag-extractive-first.ts (3 predicates moved byte-verbatim, machine-verified; rag.ts 5029→4908 vs 5030 budget). Kills the run-#57 6x wasted-generation class. REVIEWS (both pre-push): rag-retrieval-reviewer APPROVE-WITH-NITS — move fidelity brace-diff verified byte-identical; confidence-gate skip PROVEN safe (passed-gate is a no-op in applyConfidenceGate; markers pairwise mutually exclusive); comparison false-positives blocked by unchanged classifier precedence; offline/source-only idempotent; zero retrieval/ranking/selection/threshold change; P2 = eval-only assertions (intent_coverage/artifact_leaks/expected-file) unverifiable offline for flip candidates (quality-nocc-document-support, quality-form-required-documentation, quality-discharge-documentation, quality-duress-pathway + rag-set siblings) → pre-merge BRANCH canary recommended and ADOPTED (offline-green + review-approved proven insufficient for this surface, 2026-07-20). clinical-governance-reviewer APPROVE-WITH-NITS — full gate-stack trace: nothing bypassed (same finalizeRagAnswerQuality, same citation scoping, numeric verification not fail-open, ungrounded-finalize defense at rag.ts:3694); P2 = pre-existing bare-cross-reference-with-overlap gap, NOT materially widened (new trigger anti-correlates), hardening recommended → APPLIED this PR: !isBareCrossReferenceAnswer screen in hasValidatedExtractiveCandidate (closes all three short-circuit paths; discriminating test added; disclosed post-review delta, strictly narrows shipping). MERGE GATE: draft until the branch canary pair (baseline #57/#58 vs branch run with answer_case_limit=44 + answer_quality_eval=true, est $3-6 of authorized envelope) is green — zero per-case regressions, recalls 1.0, quality/targeting rates >= baseline. | Red-proof + 3 negative guards; focused 61/61 + fallback/offline/contract suites; full suite 3061 passed / 1 known container artifact (pre-hardening tree; hardening re-verified focused); typecheck+lint+prettier clean; maintainability budget passed (4908/5030) | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (pair verdict — no code change; main stays at 22b6a2e) | canary run 29794759627 (#59, branch head 7310cb3 = merged PR-B content) | E-3b + E-3c PR-B LIVE-VALIDATED (pair vs banked #57/#58): route_ceiling_failures 2→0 (E-3b proven: agitation timeout now fits inside budget; clozapine retrieval-exhausted ceiling honestly excused via the triple-condition cross-region carve-out — exactly one "retrieval-exhausted" audit cell in the report); p95 17.4s→15.47s (-11%); golden retrieval SUCCESS 36/36 (stop-ship criterion held); validated_routine_extractive_first fired on 4 of the 6 target cases (patient-safety-plan 3.3s, treatment-team-process 2.6s, ect-procedure 2.2s, illegal-substances 2.2s — all pure extractive, zero generation, was 6-9s each with a discarded attempt), the other 2 (community-home-visits, best-practice-prescribing) stayed on generation+fallback because their extractive candidates legitimately fail validation gates = the designed-conservative outcome; ZERO new failing cases (list 5→2, both known residuals: neuroleptic citation red = Option A territory, admission-comparison expected-doc = non-blocking labeling residual); grounded 1.0 + unsupported_correct 1.0 held; targeting 0.5909→0.619, fail_closed 0.9→0.9333, readability/artifact_leaks/intent_coverage unchanged; relevance 0.6→0.5667 = single-case wobble on n=30, WATCH in E-4, not a gate. Discarded-generation rate materially down (4 conversions; residual = the H2 strong-route slice named as E-3d candidate, per design's 20-33% expectation band). MERGE STANDS (user had armed auto-merge pre-verdict; revert drill not triggered). Spend +~$3-6 → Phase E total ~$6-12 of ≤$20. | Pair evidence: run #59 job log (Blocking failures = citation only; Answer Case Diagnostics markers; metric_rates + targeting blocks); dump artifact populated for PR-C diagnosis | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3c PR-C figure-aware selection) | 043b030 + P2-fix commit | E-3c PR-C: dose/monitoring extractive answers now carry the asked-for figure/schedule when the cited chunk verbatim supports it — lead-slot promotion (dose swaps last of 2 slots, monitoring appends 2nd sentence; no-op when a lead already carries a figure) guarded by the claim-support atom corpus (sourceEvidenceText exported, promotionAtomKey byte-identical to claim-support's atomKey), plus the dose/threshold generation-fallback preferring the safe figure-carrying candidate (safety gate unchanged, filter-order-stable). Fallback helpers extracted to rag-extractive-answer (cycle-check verified); rag.ts 4908→4901. Six discriminating tests each verified red-on-prior-code incl. proving the nuke-guard load-bearing by disabling it. REVIEWS (both pre-push on 043b030): rag-retrieval-reviewer APPROVE-WITH-NITS — no-op path byte-identical verified, atom-key identity verified, filter-vs-find proven side-effect-free, 2-sentence append gate-safe, intent double-gated, zero retrieval/ordering change, imputation contract green; P2 = zero-atom monitoring figures ("every 6 weeks" yields no value atom) pass the guard trivially and can be nuked by claim support if sourced only from adjacent context (fails SAFE — evidence gap, never a wrong figure). clinical-governance-reviewer APPROVE-WITH-NITS — all six clinical concerns CLEARED end-to-end (verbatim-support guarantee, citation binding preserved, conservative failure test-proven, unsafe candidates impossible, wrong-drug risk controlled by pre-existing entity/multi-drug guards, no PHI); same zero-atom finding as P3 + one comment-precision nit. P2 FIXED post-review (disclosed): zero-atom figures now require the matched figure substring verbatim in sourceEvidenceText (intentFigureMatchText); proven both directions by 3 new tests (promotes from content, refuses from adjacent-context-only); comment-precision nit folded in. | Focused post-fix: extractive-formatting 32/32 + fallback/eval-cases/offline/contract/extractive-first 98/98 incl. imputation contract; typecheck+prettier+budgets clean; full-suite 3068-passed baseline pre-P2-fix (fix re-verified focused). Live proof = E-4 pair next | diff --git a/src/lib/rag/rag-claim-support.ts b/src/lib/rag/rag-claim-support.ts index 845dd7aa..4d53533a 100644 --- a/src/lib/rag/rag-claim-support.ts +++ b/src/lib/rag/rag-claim-support.ts @@ -99,7 +99,12 @@ function splitComparisonClaims(value: string) { .slice(0, 24); } -function atomKey(atom: ClinicalValueAtom) { +/** + * Canonical identity for a clinical value atom, shared with the extractive + * lead-figure promotion guard (rag-extractive-answer.ts) so promotion and + * claim-support agree on whether a figure is present in the evidence corpus. + */ +export function clinicalValueAtomKey(atom: ClinicalValueAtom) { return [ atom.kind, atom.canonicalValue, @@ -212,7 +217,14 @@ function compatibleSafetyDimensions(claim: string, evidence: string) { return [...safetyDimensions(claim)].every((dimension) => evidenceDimensions.has(dimension)); } -function sourceEvidenceText(source: SearchResult) { +// Exported so the extractive figure-promotion guard (rag-extractive-answer.ts) can check a +// candidate fact's value atoms against the EXACT corpus this module assesses claims with. +// Keep the two in lockstep: a promoted figure verified against a wider corpus (e.g. one that +// includes adjacent_context) would pass numeric verification and then trip +// claim_support_high_risk_gap here. (Precision note: this corpus also includes +// index_unit metadata that the numeric-verification corpus does not, so it is not a strict +// subset — that sole divergence fails safe: the guard passes, the numeric gate then nukes.) +export function sourceEvidenceText(source: SearchResult) { return [ source.section_heading, source.content, @@ -243,8 +255,8 @@ function sourceSupportsClaim(claim: string, source: SearchResult) { if (!compatibleSafetyDimensions(claim, evidence)) return false; if (!compatibleHighRiskTrigger(claim, evidence)) return false; - const evidenceAtoms = new Set(extractClinicalValueAtoms(evidence).map(atomKey)); - if (extractClinicalValueAtoms(claim).some((atom) => !evidenceAtoms.has(atomKey(atom)))) return false; + const evidenceAtoms = new Set(extractClinicalValueAtoms(evidence).map(clinicalValueAtomKey)); + if (extractClinicalValueAtoms(claim).some((atom) => !evidenceAtoms.has(clinicalValueAtomKey(atom)))) return false; const claimTopics = topicTokens(claim); const evidenceTopics = topicTokens(evidence); diff --git a/src/lib/rag/rag-extractive-answer.ts b/src/lib/rag/rag-extractive-answer.ts index f641a910..1755b98f 100644 --- a/src/lib/rag/rag-extractive-answer.ts +++ b/src/lib/rag/rag-extractive-answer.ts @@ -1,5 +1,5 @@ import { boldHighYieldClinicalText } from "@/lib/answer-ranking"; -import { applyNumericVerification } from "@/lib/answer-verification"; +import { applyNumericVerification, extractClinicalValueAtoms } from "@/lib/answer-verification"; import { citationFromResult as resultCitation, compactCitations } from "@/lib/citations"; import { classifyRagQuery } from "@/lib/clinical-search"; import { ragDeepMemoryVersion } from "@/lib/deep-memory"; @@ -23,6 +23,7 @@ import { sanitizeAnswerText, splitBalancedWords, } from "@/lib/rag/rag-answer-text"; +import { cloneAnswer } from "@/lib/rag/rag-cache"; import { ragProviderMode } from "@/lib/rag/rag-provider"; import { isLowYieldClinicalText, @@ -39,7 +40,7 @@ import type { RagQueryClass, SearchResult, } from "@/lib/types"; -import { assessAndEnforceClaimSupport } from "@/lib/rag/rag-claim-support"; +import { assessAndEnforceClaimSupport, clinicalValueAtomKey, sourceEvidenceText } from "@/lib/rag/rag-claim-support"; type AnswerIntent = | "dose" @@ -367,6 +368,56 @@ const doseIntentEvidencePattern = new RegExp( "i", ); +// Interval/schedule tokens that make a monitoring fact carry its asked-for +// schedule ("baseline", "annually", "every 3", "6 months"), plus unit-bearing +// level ranges ("0.6-0.8 mmol/L") for target-level monitoring answers. +// Alternation order is load-bearing: "every N unit" must precede bare +// "every N" so exec() returns the full schedule, not a truncated "every 6". +// Digit+unit intervals also yield value atoms, so atom identity catches +// unit mismatches today — the full match keeps the promotion guard's +// verbatim-corpus fallback equally honest if that coverage ever drifts. +const monitoringIntervalFigurePattern = + /\b(?:baseline|weekly|monthly|annual(?:ly)?|every\s+\d+\s*(?:week|month|day|hour|year)s?|every\s+\d+|\d+\s*(?:week|month|day|hour|year)s?)\b/i; +const monitoringUnitRangeFigurePattern = + /\d+(?:\.\d+)?\s*[-–]\s*\d+(?:\.\d+)?\s*(?:mmol\/L|mg|micrograms?|nmol\/L|mcg)/i; + +/** + * Whether extracted fact text carries the figure/schedule the intent asks for: + * a concrete dose value (or equivalent maximum-dose wording) for dose intent, + * an interval/schedule token or unit-bearing level range for monitoring intent. + * Other intents have no figure requirement and always return false. + */ +function factCarriesIntentFigure(intent: AnswerIntent, text: string) { + return intentFigureMatchText(intent, text) !== null; +} + +/** The matched figure substring for the intent, or null — lets the promotion + * guard verify a zero-atom figure (e.g. "every 6 weeks") verbatim in the + * claim-support corpus rather than trusting it atom-free. */ +function intentFigureMatchText(intent: AnswerIntent, text: string) { + if (intent === "dose") { + return clinicalDoseValuePattern.exec(text)?.[0] ?? maximumDoseEquivalentPattern.exec(text)?.[0] ?? null; + } + if (intent === "monitoring_schedule") { + return monitoringIntervalFigurePattern.exec(text)?.[0] ?? monitoringUnitRangeFigurePattern.exec(text)?.[0] ?? null; + } + return null; +} + +/** + * Whether an extractive answer's text carries the asked-for figure for a + * figure-seeking query (dose or monitoring-schedule intent). Used by the + * generation-fallback candidate preference in rag.ts so a safe single-chunk + * candidate that states the requested figure wins over an equally safe + * figure-less candidate. Non-figure intents return false, leaving the + * existing first-safe-candidate behaviour untouched. + */ +export function extractiveAnswerCarriesIntentFigure(answerText: string, query: string, queryClass: RagQueryClass) { + const intent = classifyAnswerIntent(query, queryClass); + if (intent !== "dose" && intent !== "monitoring_schedule") return false; + return factCarriesIntentFigure(intent, answerText.replace(/\*\*/g, "")); +} + /** Requires blood count evidence. */ function requiresBloodCountEvidence(query: string) { return /\b(?:anc|fbc|full blood count|blood count|wbc|wcc|white blood cells?|white cells?|neutrophils?)\b/i.test( @@ -1045,6 +1096,66 @@ function buildFactSections(facts: ExtractedClinicalFact[], query: string) { .filter((section) => section.body && section.citation_chunk_ids.length > 0); } +/** + * Nuke-proofing guard for lead-slot figure promotion: only promote a fact whose + * clinical value atoms ALL appear in the claim-support evidence corpus + * (sourceEvidenceText) of its citing chunk(s). Fact extraction reads + * adjacent_context and numeric verification accepts it too, but claim support + * does not — so a figure that lives only in adjacent context would pass numeric + * verification and then trip claim_support_high_risk_gap, nuking the whole + * answer. Facts with no value atoms instead require their figure tokens + * verbatim in the same corpus: schedule tokens like "baseline" or "annually" + * match the monitoring figure pattern yet yield no atom (reviewer P2), so the + * atom check alone would pass a figure that lives only in adjacent context and + * claim support would then nuke. + */ +function promotedFactFigureIsClaimSupportable( + fact: ExtractedClinicalFact, + results: SearchResult[], + intent: AnswerIntent, +) { + const citingResults = results.filter((result) => fact.citationChunkIds.includes(result.id)); + if (citingResults.length === 0) return false; + const factAtoms = extractClinicalValueAtoms(fact.text); + if (factAtoms.length === 0) { + const figureMatch = intentFigureMatchText(intent, fact.text); + if (!figureMatch) return false; + const normalizedFigure = figureMatch.toLowerCase().replace(/\s+/g, " ").trim(); + return citingResults.some((result) => + sourceEvidenceText(result).toLowerCase().replace(/\s+/g, " ").includes(normalizedFigure), + ); + } + const corpusAtomKeys = new Set( + citingResults.flatMap((result) => extractClinicalValueAtoms(sourceEvidenceText(result)).map(clinicalValueAtomKey)), + ); + return factAtoms.every((atom) => corpusAtomKeys.has(clinicalValueAtomKey(atom))); +} + +/** + * Lead-slot figure guarantee for dose and monitoring-schedule answers: when no + * lead fact carries the asked-for figure/schedule but a later extracted fact + * does (and its figure survives the claim-support guard), surface that fact in + * the lead — dose swaps it into the last of its two lead slots; monitoring + * appends it as a second lead sentence. A lead that already carries a figure is + * returned unchanged, and other intents never reach this function. + */ +function promoteIntentFigureLeadFacts( + leadFacts: ExtractedClinicalFact[], + facts: ExtractedClinicalFact[], + intent: AnswerIntent, + results: SearchResult[], +) { + if (leadFacts.some((fact) => factCarriesIntentFigure(intent, fact.text))) return leadFacts; + const promoted = facts + .slice(leadFacts.length) + .find( + (fact) => + factCarriesIntentFigure(intent, fact.text) && promotedFactFigureIsClaimSupportable(fact, results, intent), + ); + if (!promoted) return leadFacts; + return intent === "dose" ? [...leadFacts.slice(0, -1), promoted] : [...leadFacts, promoted]; +} + /** Build fact synthesized answer. */ function buildFactSynthesizedAnswer(args: { query: string; @@ -1069,7 +1180,10 @@ function buildFactSynthesizedAnswer(args: { }; } - const leadFacts = facts.slice(0, args.intent === "dose" ? 2 : 1); + let leadFacts = facts.slice(0, args.intent === "dose" ? 2 : 1); + if (args.intent === "dose" || args.intent === "monitoring_schedule") { + leadFacts = promoteIntentFigureLeadFacts(leadFacts, facts, args.intent, args.results); + } // Once the lead answer names the query entity, later lead sentences skip // their own entity prefix so the entity is not repeated in every sentence. // Derived exactly the way sentenceFromFact derives its prefix entity (from @@ -1714,6 +1828,36 @@ export function generatedAnswerQualityFailureReason(answer: RagAnswer, query: st return null; } +/** + * Whether an extractive fallback candidate is safe to ship in place of a failed + * generation: grounded and supported, clean of every final answer-quality gate, + * and numerically verified with zero unverified tokens. Pure — extracted from + * the rag.ts generation-fallback path so candidate selection stays testable in + * isolation. + */ +export function isSafeExtractiveFallbackCandidate(candidate: RagAnswer, query: string, queryClass: RagQueryClass) { + if (!candidate.grounded || candidate.confidence === "unsupported") return false; + if (generatedAnswerQualityFailureReason(candidate, query, queryClass)) return false; + const verified = applyNumericVerification(cloneAnswer(candidate)); + return ( + verified.grounded && verified.confidence !== "unsupported" && (verified.unverifiedNumericTokens?.length ?? 0) === 0 + ); +} + +/** + * Narrows a fallback candidate's sources to the chunks its citations actually + * reference, so downstream claim support and numeric verification judge the + * candidate on exactly the evidence it cites. Pure — extracted from the rag.ts + * generation-fallback path. + */ +export function retainCitedExtractiveFallbackEvidence(candidate: T): T { + const citedChunkIds = new Set(candidate.citations.map((citation) => citation.chunk_id)); + return { + ...candidate, + sources: candidate.sources.filter((source) => citedChunkIds.has(source.id)), + }; +} + /** * Replaces an answer that fails final quality checks with an evidence-gap response. * diff --git a/src/lib/rag/rag.ts b/src/lib/rag/rag.ts index 0e1297a7..50b99848 100644 --- a/src/lib/rag/rag.ts +++ b/src/lib/rag/rag.ts @@ -54,14 +54,17 @@ export { import { buildExtractiveAnswer, cleanAnswerSectionHeading, + extractiveAnswerCarriesIntentFigure, finalQualityGapAnswer, finalizeRagAnswerQuality, generatedAnswerQualityFailureReason, hasInvalidModelEvidenceIds, isOverExpandedSimpleGeneratedAnswer, + isSafeExtractiveFallbackCandidate, isSimpleDirectQuestion, isTemplateLikeGeneratedAnswer, isUnusableGeneratedAnswer, + retainCitedExtractiveFallbackEvidence, sourceBackedGenerationTimeoutAnswer, strongReasoningEffortForQueryClass, } from "@/lib/rag/rag-extractive-answer"; @@ -4637,34 +4640,24 @@ ${qualityRetryInstruction}` scoreExplanations: candidateArtifacts.scoreExplanations, } satisfies RagAnswer; }; - const isSafeExtractiveFallbackCandidate = (candidate: RagAnswer) => { - if (!candidate.grounded || candidate.confidence === "unsupported") return false; - if (generatedAnswerQualityFailureReason(candidate, args.query, queryClass)) return false; - const verified = applyNumericVerification(cloneAnswer(candidate)); - return ( - verified.grounded && - verified.confidence !== "unsupported" && - (verified.unverifiedNumericTokens?.length ?? 0) === 0 - ); - }; - const retainCitedExtractiveFallbackEvidence = (candidate: T): T => { - const citedChunkIds = new Set(candidate.citations.map((citation) => citation.chunk_id)); - return { - ...candidate, - sources: candidate.sources.filter((source) => citedChunkIds.has(source.id)), - }; - }; let extractiveFallbackAnswer = canRecoverGenerationErrorExtractively ? buildExtractiveFallbackCandidate(generationFallbackResults) : null; // Generated synthesis has already failed, so do not stitch dose or threshold figures - // across fallback chunks. Prefer the first individually complete candidate that passes - // every extractive and numeric safety gate. + // across fallback chunks. Prefer an individually complete candidate that passes every + // extractive and numeric safety gate — and among those, one whose answer carries the + // asked-for dose/monitoring figure, so a figure-less chunk that happens to rank first + // cannot displace a verbatim-supported dose or schedule. if (extractiveFallbackAnswer && (queryClass === "medication_dose_risk" || queryClass === "table_threshold")) { + const safeSingleChunkCandidates = generationFallbackResults + .map((result) => retainCitedExtractiveFallbackEvidence(buildExtractiveFallbackCandidate([result]))) + .filter((candidate) => isSafeExtractiveFallbackCandidate(candidate, args.query, queryClass)); extractiveFallbackAnswer = - generationFallbackResults - .map((result) => retainCitedExtractiveFallbackEvidence(buildExtractiveFallbackCandidate([result]))) - .find(isSafeExtractiveFallbackCandidate) ?? extractiveFallbackAnswer; + safeSingleChunkCandidates.find((candidate) => + extractiveAnswerCarriesIntentFigure(candidate.answer, args.query, queryClass), + ) ?? + safeSingleChunkCandidates[0] ?? + extractiveFallbackAnswer; } const extractiveFallbackQualityReason = extractiveFallbackAnswer ? generatedAnswerQualityFailureReason(extractiveFallbackAnswer, args.query, queryClass) diff --git a/tests/extractive-answer-formatting.test.ts b/tests/extractive-answer-formatting.test.ts index 08f3059d..df575f72 100644 --- a/tests/extractive-answer-formatting.test.ts +++ b/tests/extractive-answer-formatting.test.ts @@ -5,7 +5,45 @@ import { sentenceFromFact, splitClinicalEvidenceSentences, } from "../src/lib/rag/rag-extractive-answer"; -import type { RagAnswer, SearchResult } from "../src/lib/types"; +import { classifyRagQuery } from "../src/lib/clinical-search"; +import type { RagAnswer, RagQueryClass, SearchResult } from "../src/lib/types"; + +function extractiveAnswerFor(query: string, results: SearchResult[], queryClass?: RagQueryClass) { + return buildExtractiveAnswer({ + query, + queryClass: queryClass ?? classifyRagQuery(query).queryClass, + results, + quoteCards: [], + documentBreakdown: [] as RagAnswer["documentBreakdown"], + evidenceSummary: undefined as unknown as RagAnswer["evidenceSummary"], + sourceCoverage: undefined as unknown as RagAnswer["sourceCoverage"], + conflictsOrGaps: [], + visualEvidence: [] as unknown as RagAnswer["visualEvidence"], + bestSource: undefined as unknown as RagAnswer["bestSource"], + smartPanel: undefined as unknown as RagAnswer["smartPanel"], + relatedDocuments: [] as unknown as RagAnswer["relatedDocuments"], + routeReason: "demo", + timings: undefined as unknown as RagAnswer["latencyTimings"], + }); +} + +function figureChunk(overrides: Partial): SearchResult { + return { + id: "figure-chunk-1", + document_id: "figure-doc", + title: "Quetiapine Prescribing Guideline", + file_name: "Quetiapine Prescribing Guideline.pdf", + page_number: 2, + chunk_index: 1, + section_heading: "Dosing", + content: "", + image_ids: [], + similarity: 0.91, + hybrid_score: 0.95, + images: [], + ...overrides, + } as unknown as SearchResult; +} describe("maximum-dose evidence", () => { it("accepts equivalent numeric limit wording without accepting an unrelated dose", () => { @@ -325,3 +363,173 @@ describe("extractive answer end to end", () => { expect(plain).toMatch(/For acute mania, IR product is 750 to 1000mg daily/); }); }); + +// E-3c PR-C: dose and monitoring answers must carry the asked-for figure or +// schedule in the lead whenever a cited chunk verbatim supports it. The fact +// sorter orders same-priority facts shortest-first, so a terse figure-less +// sentence used to displace the sentence holding the actual dose or interval. +describe("figure-aware extractive lead selection", () => { + it("promotes the schedule-bearing monitoring fact into the lead over a shorter figure-less fact", () => { + const answer = extractiveAnswerFor("What metabolic monitoring is required for antipsychotics?", [ + figureChunk({ + id: "metabolic-chunk-1", + section_heading: "Metabolic monitoring", + content: + "Metabolic monitoring is required for all patients prescribed antipsychotics. Weight, blood pressure, fasting glucose and lipids are monitored at baseline, at 3 months, then annually.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + expect(plain).toMatch(/baseline|annually/); + expect(plain).toMatch(/3 months/); + }); + + it("swaps the dose figure fact into the last dose lead slot when both lead slots are figure-less", () => { + const answer = extractiveAnswerFor("What is the quetiapine dose?", [ + figureChunk({ + id: "quetiapine-dose-chunk-1", + content: + "Quetiapine doses must be taken daily. Quetiapine doses should be reviewed daily. The maximum recommended quetiapine dose is 200 mg daily.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + expect(plain).toContain("200 mg"); + // The first lead slot keeps the top-ranked fact; only the last slot swaps. + expect(plain).toMatch(/^Quetiapine doses must be taken daily\./); + }); + + it("refuses promotion when the figure only exists in adjacent context outside the claim-support corpus", () => { + const answer = extractiveAnswerFor("What is the quetiapine dose?", [ + figureChunk({ + id: "quetiapine-adjacent-chunk-1", + content: "Quetiapine doses must be taken daily. Quetiapine doses should be reviewed daily.", + adjacent_context: "The maximum recommended quetiapine dose is 200 mg daily.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + // The 200 mg figure passes numeric verification (adjacent context is part of + // that corpus) but not claim support, so promoting it would nuke the whole + // answer at the claim gate. The figure-less lead must stay. + expect(plain).not.toContain("200 mg"); + expect(plain).toMatch(/^Quetiapine doses must be taken daily\./); + }); + + it("keeps non-dose, non-monitoring fact ordering byte-identical", () => { + const answer = extractiveAnswerFor("When is quetiapine contraindicated?", [ + figureChunk({ + id: "quetiapine-contraindication-chunk-1", + section_heading: "", + content: + "Quetiapine is contraindicated in known hypersensitivity. Quetiapine is contraindicated in patients already taking more than 200 mg daily of interacting sedatives.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + // Contraindication intent has no figure requirement: the shorter lead fact + // must keep the single lead slot exactly as before. + expect(plain).toBe("Quetiapine is contraindicated in known hypersensitivity."); + expect(plain).not.toContain("200 mg"); + }); + + it("leaves a dose lead that already carries its figure untouched", () => { + const answer = extractiveAnswerFor("What is the quetiapine dose?", [ + figureChunk({ + id: "quetiapine-figured-chunk-1", + content: "Quetiapine is started at 25 mg at night. The maximum recommended quetiapine dose is 200 mg daily.", + }), + ]); + + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(answer.grounded).toBe(true); + expect(plain).toBe( + "Quetiapine is started at 25 mg at night. The maximum recommended quetiapine dose is 200 mg daily.", + ); + }); +}); + +describe("zero-atom figure promotion guard (reviewer P2)", () => { + const question = "What metabolic monitoring is required for antipsychotics?"; + const leadOnly = "Metabolic monitoring is required for all patients prescribed antipsychotics."; + const intervalSentence = "Weight and fasting glucose are monitored for antipsychotic patients every 6 weeks."; + + it("refuses an adjacent-context interval whose number matches corpus text with a different unit", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-1", + section_heading: "Metabolic monitoring", + // The synopsis shares the bare number but not the unit, and carries no + // intent tokens so it can never be admitted as a competing fact. The + // "every 6 weeks" figure yields a 6/week quantity atom, so atom + // identity must refuse the 6/month corpus — and the interval pattern's + // full-match ordering keeps the verbatim fallback equally honest. + retrieval_synopsis: "Reviewed every 6 months", + content: leadOnly, + adjacent_context: intervalSentence, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).not.toMatch(/every 6 weeks/i); + expect(plain).not.toMatch(/every 6 months/i); + }); + + // Longer than leadOnly so it stays a promotion candidate instead of taking + // the shortest-first lead slot, and digit-free so it yields no value atom — + // exercising the guard's verbatim-corpus fallback rather than atom identity. + const annualSentence = "Weight and fasting glucose monitoring is repeated annually for all antipsychotic patients."; + + it("promotes a truly zero-atom schedule token verbatim-supported by chunk content", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-4", + section_heading: "Metabolic monitoring", + content: `${leadOnly} ${annualSentence}`, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).toMatch(/annually/i); + }); + + it("refuses a truly zero-atom schedule token that exists only in adjacent context", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-5", + section_heading: "Metabolic monitoring", + content: leadOnly, + adjacent_context: annualSentence, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).not.toMatch(/annually/i); + }); + + it("promotes an atom-backed interval verbatim-supported by chunk content", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-2", + section_heading: "Metabolic monitoring", + content: `${leadOnly} ${intervalSentence}`, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).toMatch(/every 6 weeks/i); + }); + + it("refuses the interval when it exists only in adjacent context", () => { + const answer = extractiveAnswerFor(question, [ + figureChunk({ + id: "monitoring-guard-3", + section_heading: "Metabolic monitoring", + content: leadOnly, + adjacent_context: intervalSentence, + }), + ]); + const plain = (answer.answer ?? "").replace(/\*\*/g, ""); + expect(plain).not.toMatch(/every 6 weeks/i); + }); +}); diff --git a/tests/rag-answer-fallback.test.ts b/tests/rag-answer-fallback.test.ts index 9a9c04af..57c54c7b 100644 --- a/tests/rag-answer-fallback.test.ts +++ b/tests/rag-answer-fallback.test.ts @@ -1992,6 +1992,48 @@ describe("RAG structured-output fallback", () => { expect(isCacheableGroundedGenerationFallback(answer)).toBe(false); }); + it("prefers the safe single-chunk fallback candidate that carries the asked-for dose figure", async () => { + // E-3c PR-C: both chunks yield safe single-chunk extractive candidates, but + // only the lower-ranked one states the dose figure the query asks for. The + // first-safe-candidate rule used to ship the figure-less answer. + const answer = await answerFromTextSources( + "What is the usual quetiapine dose?", + [ + source({ + id: "quetiapine-advice-1", + document_id: "quetiapine-doc", + title: "Quetiapine Prescribing Guideline", + file_name: "quetiapine-prescribing-guideline.pdf", + section_heading: "Dose and administration", + content: "The usual quetiapine dose is taken once daily in the evening.", + similarity: 0.97, + hybrid_score: 0.97, + text_rank: 1.4, + }), + source({ + id: "quetiapine-maximum-1", + document_id: "quetiapine-doc", + title: "Quetiapine Prescribing Guideline", + file_name: "quetiapine-prescribing-guideline.pdf", + section_heading: "Maximum dose", + content: "The maximum recommended quetiapine dose is 200 mg daily.", + similarity: 0.86, + hybrid_score: 0.86, + text_rank: 1.1, + }), + ], + new Error("OpenAI generation incomplete: max_output_tokens"), + ); + + expect(answer.routingMode).toBe("extractive"); + expect(answer.routingReason).toContain("source_backed_extractive_fallback"); + expect(answer.grounded).toBe(true); + expect(answer.confidence).not.toBe("unsupported"); + expect(answer.unverifiedNumericTokens ?? []).toEqual([]); + expect(answer.answer.replace(/\*\*/g, "")).toContain("200 mg"); + expect(new Set(answer.citations.map((citation) => citation.chunk_id))).toEqual(new Set(["quetiapine-maximum-1"])); + }); + it("never marks the generic source-review fallback as cacheable", async () => { const { isCacheableGroundedGenerationFallback } = await import("../src/lib/rag/rag");