diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index cc05fc6e..cb5a950b 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -669,3 +669,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3b budget-aware generation deadlines) | 314d03f | Clinical-governance review of E-3b diff `git diff origin/main...HEAD` (commits 1078264 + 314d03f): answer-side generation timing/gating + telemetry — reserve-aware generation timeout (`generationRequestTimeoutMs`), truncation self-heal budget gate (`deadlineAllowsGenerationRetry`), `route_budget_exhausted_by_retrieval` telemetry, and the cross-region eval carve-out in scripts/eval-quality.ts. | APPROVE-WITH-NITS. No P0/P1/P2. (a) Conservative failure preserved: reserve-aware timeout fires ~2s early producing the SAME error type — an internal SDK timeout is mapped by mapOpenAIError to PublicApiError(openai_timeout) (openai.ts:525-529), NOT a bare DOMException, so the rag.ts:4636 re-throw guard is not tripped and the existing source-backed fallback/extractive recovery is reached; truncation-skip falls through to the terminal throw (rag.ts:4309-4313) into the same catch. No new answer-producing path. (b) Safety gates intact: all recovery answers finalize through finalizeAnswer→finalizeRagAnswerQuality and isSafeExtractiveFallbackCandidate (grounded/confidence/quality/numeric) — none touched. (c) Eval carve-out env-gated: crossRegionRunner && budgetExhaustedByRetrieval && generationMs===0; EVAL_LATENCY_CONTEXT set only in eval-canary.yml:164, prod caller (eval-quality.ts:1095) passes no options → inert in release/local; generationMs===0 requirement means a generation-side failure (generationMs>0) is never suppressed. (d)/(e) Telemetry additions non-PHI (boolean + mechanical retry-reason strings); no privacy/query-privacy/cross-border/verification source files touched; no new provider call. Nits (P3, non-blocking): carve-out also excuses fast/strong routes when generationMs===0 (sound — provably no generation ran); `requestTimeoutMs` now prod-dead (test-only); report label "retrieval-exhausted" (routeDeadlineExceeded && flag) is broader than actual gate suppression (audit label only, gate stays strict). | 88 offline unit tests PASS (tests/rag-route-budget.test.ts, tests/eval-quality.test.ts, tests/rag-offline-answer.test.ts, tests/rag-answer-fallback.test.ts); no provider/Supabase/OpenAI calls; no files mutated except this ledger row | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3b budget-aware generation deadlines) | 314d03f (+eb08ec5 review row) | ADDENDUM 5 wave E-3b implemented per the design-agent plan: generation attempts clamped to route budget minus a measured 2s recovery reserve (single call-site, all four attempt kinds); truncation self-heal gated on retry viability (reserve+5s floor) with observable truncation_retry_skipped_budget_reserve marker; additive route_budget_exhausted_by_retrieval runtime flag; eval route-ceiling gains the triple-condition cross-region carve-out (context + runtime flag + zero generation) with retrieval-exhausted audit cells — local/release gates provably strict. Fixes I3 (54ms budget overrun after 22.6s provider timeout), I5 (82s truncation waste class), resolves I2 (clozapine 13.3s retrieval vs 12s runtime budget = geography, now suppressed ONLY in the sanctioned cross-region context with full auditability). Reviewer verdicts: rag-retrieval-reviewer APPROVE-WITH-NITS (2 P3: prod-dead requestTimeoutMs retained for symmetry; report-cell coupling cosmetic; cached-replay invariant PROVEN — budget-exhausted answers never cached, carve-out unreachable via replay; marker isolation proven — SLO counters key on fallback_reason not answer_retry_reasons); clinical-governance-reviewer APPROVE-WITH-NITS (prior row) — internal-timeout→PublicApiError→existing-fallback path verified, all safety gates still applied to recovery answers. ALSO BANKED — E-2 targeting baseline (canary run #58, 29788404357, all-green incl. first execution of the !cancelled()-fixed instrument, ~$1-2): metric_rates relevance 0.6 / readability 1.0 / artifact_leaks 1.0 / intent_coverage 0.9333 / fail_closed 0.9; targeting_rate 0.5909 (13/22); by intent: document_lookup 5/5, red_result_action 3/3, contraindication 2/2, dose 1/5, monitoring_schedule 1/5, pathway_referral 1/2; all 9 misses = missing dose figure/schedule-interval (answer lengths 73-232 chars) → E-3c co-primary target alongside the wasted-generation class. Phase E spend ≈$3-6 of ≤$20. | Red-proofs: reserve pinned 3 independent ways (exact 23000ms grant, deadline flag clear, total under budget); self-heal skip pins exact marker + single provider call; offline flag pinned true/false. Focused: route-budget 9/9, eval-quality 27/27, fallback+offline 52/52, parser/abort regressions 15/15. Full suite 3043 passed / 1 known container pdf artifact. typecheck+lint+prettier clean. No provider calls; live proof = E-4 paired run | | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (restarted from e02ba3d; PR: E-3c PR-A instrument) | see PR head | ADDENDUM 5 wave E-3c design accepted (Plan agent, full report in session record) and PR-A delivered: eval-answer-quality gains --dump-answers (per-case answer TEXT + sections + targeting verdict JSON for the canary artifact — answers are not retained at rest by privacy design, so miss diagnosis needs eval-time capture), parseArgs/buildAnswerDumpRecord exported behind an import.meta main-guard (eval-quality precedent), canary targeting step wired with the flag. Design highlights for the record: DEFECT 1 root confirmed as fast-attempt-doomed-then-discarded on strong_routine_retrieval procedural shapes — PR-B generalizes the EXISTING validated-extractive short-circuit pattern (LAI + blocked-recovery precedents, hasValidatedExtractiveCandidate) to the measured shape via a new rag-extractive-first.ts module (net ~-110 rag.ts lines against the 5030 budget); DEFECT 2 rank-1 root = extractive lead-slot selection prefers shortest sentence and admits figure-less leads (sort at rag-extractive-answer.ts:917, 1-slot monitoring leads) — PR-C adds intent-figure-aware lead promotion with a claim-support atom-corpus nuke-proofing guard + dose/threshold fallback candidate preference (find(safe && figure) ?? find(safe)); answer-verification CLEARED as direct cause (whole-answer gap or unbold only, never per-figure deletion). H2 (strong-route comparison/complex residual) explicitly deferred as the named E-3d candidate. E-4 metrics set: discarded-generation <20% from ~47%, dose ≥3/5, monitoring ≥3/5, no intent below #58, recalls pinned 1.0, relevance/fail_closed/readability/artifact_leaks ≥ #58. | PR-A gates: new tests/eval-answer-quality.test.ts 4/4; typecheck, check:github-actions, check:ci-scope, eval-canary-workflow test, prettier all clean; no provider calls | +| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: E-3c PR-B short-circuit) | f7e6cbb + hardening commit | E-3c PR-B: pre-generation validated-extractive short-circuit for gate-passed routine procedural "What...process/include/required" queries (marker validated_routine_extractive_first), generalizing the LAI + blocked-recovery precedents via new rag-extractive-first.ts (3 predicates moved byte-verbatim, machine-verified; rag.ts 5029→4908 vs 5030 budget). Kills the run-#57 6x wasted-generation class. REVIEWS (both pre-push): rag-retrieval-reviewer APPROVE-WITH-NITS — move fidelity brace-diff verified byte-identical; confidence-gate skip PROVEN safe (passed-gate is a no-op in applyConfidenceGate; markers pairwise mutually exclusive); comparison false-positives blocked by unchanged classifier precedence; offline/source-only idempotent; zero retrieval/ranking/selection/threshold change; P2 = eval-only assertions (intent_coverage/artifact_leaks/expected-file) unverifiable offline for flip candidates (quality-nocc-document-support, quality-form-required-documentation, quality-discharge-documentation, quality-duress-pathway + rag-set siblings) → pre-merge BRANCH canary recommended and ADOPTED (offline-green + review-approved proven insufficient for this surface, 2026-07-20). clinical-governance-reviewer APPROVE-WITH-NITS — full gate-stack trace: nothing bypassed (same finalizeRagAnswerQuality, same citation scoping, numeric verification not fail-open, ungrounded-finalize defense at rag.ts:3694); P2 = pre-existing bare-cross-reference-with-overlap gap, NOT materially widened (new trigger anti-correlates), hardening recommended → APPLIED this PR: !isBareCrossReferenceAnswer screen in hasValidatedExtractiveCandidate (closes all three short-circuit paths; discriminating test added; disclosed post-review delta, strictly narrows shipping). MERGE GATE: draft until the branch canary pair (baseline #57/#58 vs branch run with answer_case_limit=44 + answer_quality_eval=true, est $3-6 of authorized envelope) is green — zero per-case regressions, recalls 1.0, quality/targeting rates >= baseline. | Red-proof + 3 negative guards; focused 61/61 + fallback/offline/contract suites; full suite 3061 passed / 1 known container artifact (pre-hardening tree; hardening re-verified focused); typecheck+lint+prettier clean; maintainability budget passed (4908/5030) | diff --git a/src/lib/rag/rag-extractive-first.ts b/src/lib/rag/rag-extractive-first.ts new file mode 100644 index 00000000..3eb1a030 --- /dev/null +++ b/src/lib/rag/rag-extractive-first.ts @@ -0,0 +1,218 @@ +import { + buildExtractiveAnswer, + finalizeRagAnswerQuality, + isBareCrossReferenceAnswer, +} from "@/lib/rag/rag-extractive-answer"; +import type { RagQueryClass, RetrievalConfidenceGateStatus, SearchResult } from "@/lib/types"; + +/** + * Pre-generation validated-extractive short-circuit. + * + * Each predicate here targets a measured wasted-generation shape: a query whose + * deterministic extractive answer independently passes every final quality and + * grounding gate, so the paid model call adds latency and cost without adding + * trust. A predicate may flip the answer route to `extractive` only when its + * validated candidate has already cleared those gates; retrieval, ranking, and + * selection behaviour are never touched. + */ + +type ShortCircuitRoute = { mode: "unsupported" | "extractive" | "fast" | "strong"; reason: string }; + +/** + * Allow a score-blocked routine document-content query to use the deterministic + * answer only when that answer independently passes the final safety gates. + * + * This is deliberately narrower than the normal extractive router: it cannot + * recover medication, threshold, comparison, broad-summary, complex, or weakly + * related queries. The retrieval diagnostic remains blocked so the UI still + * presents the recovered answer with low-trust guidance. + */ +export function hasValidatedExtractiveCandidate(args: { + query: string; + queryClass: RagQueryClass; + results: SearchResult[]; + routeReason: string; +}) { + const candidate = finalizeRagAnswerQuality( + buildExtractiveAnswer({ + query: args.query, + queryClass: args.queryClass, + results: args.results, + quoteCards: [], + documentBreakdown: [], + evidenceSummary: undefined, + sourceCoverage: undefined, + conflictsOrGaps: [], + visualEvidence: [], + bestSource: null, + smartPanel: undefined, + relatedDocuments: [], + routeReason: args.routeReason, + timings: undefined, + }), + args.query, + args.queryClass, + ); + + return ( + candidate.grounded && + candidate.confidence !== "unsupported" && + candidate.citations.length > 0 && + candidate.responseMode !== "evidence_gap" && + !/final_quality_gate:/.test(candidate.routingReason ?? "") && + // Governance-review hardening (PR-B P2): a cross-reference lead that shares query terms + // passes the overlap gate yet answers nothing ("Refer to the X procedure for..."). Screen + // it here so no short-circuit path can ship a redirect-only answer; the query then stays + // on model synthesis (or the existing fallback chain) instead. + !isBareCrossReferenceAnswer(candidate.answer ?? "") + ); +} + +/** Recover only routine, source-backed document lookups whose deterministic answer passes every final gate. */ +export function hasValidatedRoutineExtractiveRecovery(args: { + query: string; + queryClass: RagQueryClass; + results: SearchResult[]; + route: { mode: "unsupported" | "extractive" | "fast" | "strong"; reason: string }; + sourceBacked: boolean; +}) { + if ( + args.queryClass !== "document_lookup" || + args.route.mode !== "fast" || + args.route.reason !== "strong_routine_retrieval" || + !args.sourceBacked + ) { + return false; + } + + return hasValidatedExtractiveCandidate({ + query: args.query, + queryClass: args.queryClass, + results: args.results, + routeReason: `${args.route.reason}; validated_routine_extractive_recovery`, + }); +} + +/** + * Generic LAI-management questions repeatedly time out in generation despite strong direct + * source support. Skip that paid tail only when the question asks no specific clinical detail + * and the deterministic answer independently passes the same final quality and grounding gates. + */ +export function hasValidatedGenericLaiManagementExtractiveAnswer(args: { + query: string; + queryClass: RagQueryClass; + results: SearchResult[]; + route: { mode: "unsupported" | "extractive" | "fast" | "strong"; reason: string }; + sourceBacked: boolean; +}) { + const genericLaiManagementQuery = /^\s*how (?:are|should) long[- ]acting injectables? (?:be )?managed\??\s*$/i.test( + args.query, + ); + + if ( + !genericLaiManagementQuery || + args.queryClass !== "medication_dose_risk" || + args.route.mode !== "fast" || + args.route.reason !== "clinical_fast_grounded_synthesis" || + !args.sourceBacked + ) { + return false; + } + + return hasValidatedExtractiveCandidate({ + query: args.query, + queryClass: args.queryClass, + results: args.results, + routeReason: `${args.route.reason}; validated_generic_lai_management_extractive_answer`, + }); +} + +const routineProceduralLeadPattern = /^\s*what\b/i; +const routineProceduralKeywordPattern = + /\b(?:process|procedure|steps?|includes?|include|required?|requires?|requirements?|documentation)\b/i; + +/** + * Routine procedural document-content shape: a "What ..." question asking what a + * process/procedure includes or requires. Deliberately EXCLUDES "How is X + * handled/managed?" shapes — those stay on model synthesis (see the gate-passed + * routine document-content contract in tests/rag-answer-fallback.test.ts). + */ +export const routineProceduralContentPattern = { + test(query: string) { + return routineProceduralLeadPattern.test(query) && routineProceduralKeywordPattern.test(query); + }, +}; + +/** + * Gate-passed routine procedural document-content questions repeatedly pay for model + * synthesis that the deterministic extractive answer already covers. Skip that paid tail + * only for the strong routine fast route, only for routine lookup/general classes (never + * dose, threshold, comparison, or broad-summary classes), and only when the deterministic + * answer independently passes every final quality and grounding gate. + */ +export function hasValidatedRoutineProceduralExtractiveAnswer(args: { + query: string; + queryClass: RagQueryClass; + results: SearchResult[]; + route: ShortCircuitRoute; + sourceBacked: boolean; + gateStatus: RetrievalConfidenceGateStatus; +}) { + if ( + args.route.mode !== "fast" || + args.route.reason !== "strong_routine_retrieval" || + args.gateStatus !== "passed" || + !args.sourceBacked || + (args.queryClass !== "document_lookup" && args.queryClass !== "unsupported_or_general") || + args.results.length === 0 || + !routineProceduralContentPattern.test(args.query) + ) { + return false; + } + + return hasValidatedExtractiveCandidate({ + query: args.query, + queryClass: args.queryClass, + results: args.results, + routeReason: `${args.route.reason}; validated_routine_extractive_first`, + }); +} + +/** + * Choose the first applicable validated-extractive short-circuit for the current route. + * + * Precedence is fixed: the generic LAI-management skip (gate passed), then the + * score-blocked routine recovery (gate blocked), then the gate-passed routine + * procedural short-circuit. Returns the routing-reason marker to append, or null + * when generation should proceed unchanged. + */ +export function chooseValidatedExtractiveShortCircuit(args: { + query: string; + queryClass: RagQueryClass; + results: SearchResult[]; + route: ShortCircuitRoute; + sourceBacked: boolean; + gateStatus: RetrievalConfidenceGateStatus; +}): { reasonMarker: string } | null { + const predicateArgs = { + query: args.query, + queryClass: args.queryClass, + results: args.results, + route: args.route, + sourceBacked: args.sourceBacked, + }; + + if (args.gateStatus === "passed" && hasValidatedGenericLaiManagementExtractiveAnswer(predicateArgs)) { + return { reasonMarker: "validated_generic_lai_management_extractive_answer" }; + } + + if (args.gateStatus === "blocked" && hasValidatedRoutineExtractiveRecovery(predicateArgs)) { + return { reasonMarker: "validated_routine_extractive_recovery" }; + } + + if (hasValidatedRoutineProceduralExtractiveAnswer(args)) { + return { reasonMarker: "validated_routine_extractive_first" }; + } + + return null; +} diff --git a/src/lib/rag/rag.ts b/src/lib/rag/rag.ts index 75d1043a..0e1297a7 100644 --- a/src/lib/rag/rag.ts +++ b/src/lib/rag/rag.ts @@ -65,6 +65,7 @@ import { sourceBackedGenerationTimeoutAnswer, strongReasoningEffortForQueryClass, } from "@/lib/rag/rag-extractive-answer"; +import { chooseValidatedExtractiveShortCircuit } from "@/lib/rag/rag-extractive-first"; import { buildComparisonAnswer, buildComparisonEvidenceGapAnswer, @@ -907,110 +908,6 @@ function applyConfidenceGate( }; } -/** - * Allow a score-blocked routine document-content query to use the deterministic - * answer only when that answer independently passes the final safety gates. - * - * This is deliberately narrower than the normal extractive router: it cannot - * recover medication, threshold, comparison, broad-summary, complex, or weakly - * related queries. The retrieval diagnostic remains blocked so the UI still - * presents the recovered answer with low-trust guidance. - */ -function hasValidatedExtractiveCandidate(args: { - query: string; - queryClass: RagQueryClass; - results: SearchResult[]; - routeReason: string; -}) { - const candidate = finalizeRagAnswerQuality( - buildExtractiveAnswer({ - query: args.query, - queryClass: args.queryClass, - results: args.results, - quoteCards: [], - documentBreakdown: [], - evidenceSummary: undefined, - sourceCoverage: undefined, - conflictsOrGaps: [], - visualEvidence: [], - bestSource: null, - smartPanel: undefined, - relatedDocuments: [], - routeReason: args.routeReason, - timings: undefined, - }), - args.query, - args.queryClass, - ); - - return ( - candidate.grounded && - candidate.confidence !== "unsupported" && - candidate.citations.length > 0 && - candidate.responseMode !== "evidence_gap" && - !/final_quality_gate:/.test(candidate.routingReason ?? "") - ); -} - -/** Recover only routine, source-backed document lookups whose deterministic answer passes every final gate. */ -function hasValidatedRoutineExtractiveRecovery(args: { - query: string; - queryClass: RagQueryClass; - results: SearchResult[]; - route: { mode: "unsupported" | "extractive" | "fast" | "strong"; reason: string }; - sourceBacked: boolean; -}) { - if ( - args.queryClass !== "document_lookup" || - args.route.mode !== "fast" || - args.route.reason !== "strong_routine_retrieval" || - !args.sourceBacked - ) { - return false; - } - - return hasValidatedExtractiveCandidate({ - query: args.query, - queryClass: args.queryClass, - results: args.results, - routeReason: `${args.route.reason}; validated_routine_extractive_recovery`, - }); -} - -/** - * Generic LAI-management questions repeatedly time out in generation despite strong direct - * source support. Skip that paid tail only when the question asks no specific clinical detail - * and the deterministic answer independently passes the same final quality and grounding gates. - */ -function hasValidatedGenericLaiManagementExtractiveAnswer(args: { - query: string; - queryClass: RagQueryClass; - results: SearchResult[]; - route: { mode: "unsupported" | "extractive" | "fast" | "strong"; reason: string }; - sourceBacked: boolean; -}) { - const genericLaiManagementQuery = /^\s*how (?:are|should) long[- ]acting injectables? (?:be )?managed\??\s*$/i.test( - args.query, - ); - - if ( - !genericLaiManagementQuery || - args.queryClass !== "medication_dose_risk" || - args.route.mode !== "fast" || - args.route.reason !== "clinical_fast_grounded_synthesis" || - !args.sourceBacked - ) { - return false; - } - - return hasValidatedExtractiveCandidate({ - query: args.query, - queryClass: args.queryClass, - results: args.results, - routeReason: `${args.route.reason}; validated_generic_lai_management_extractive_answer`, - }); -} - /** Clamp confidence. */ function clampConfidence( proposed: RagAnswer["confidence"] | undefined, @@ -3425,43 +3322,25 @@ async function answerQuestionWithScopeUncoalesced( results: answerInputResults, answerMode: routeFromRouting.mode, }); - const validatedGenericLaiManagementExtractiveAnswer = - initialRetrievalDiagnostics.gateStatus === "passed" && - hasValidatedGenericLaiManagementExtractiveAnswer({ - query: args.query, - queryClass, - results: answerInputResults, - route: routeFromRouting, - sourceBacked: relevance.isSourceBacked, - }); - const validatedRoutineExtractiveRecovery = - initialRetrievalDiagnostics.gateStatus === "blocked" && - hasValidatedRoutineExtractiveRecovery({ - query: args.query, - queryClass, - results: answerInputResults, - route: routeFromRouting, - sourceBacked: relevance.isSourceBacked, - }); - const routeBeforeConfidenceGate = validatedGenericLaiManagementExtractiveAnswer + const validatedExtractiveShortCircuit = chooseValidatedExtractiveShortCircuit({ + query: args.query, + queryClass, + results: answerInputResults, + route: routeFromRouting, + sourceBacked: relevance.isSourceBacked, + gateStatus: initialRetrievalDiagnostics.gateStatus, + }); + const routeBeforeConfidenceGate = validatedExtractiveShortCircuit ? { ...routeFromRouting, mode: "extractive" as const, model: null, - reason: `${routeFromRouting.reason}; validated_generic_lai_management_extractive_answer`, + reason: `${routeFromRouting.reason}; ${validatedExtractiveShortCircuit.reasonMarker}`, } - : validatedRoutineExtractiveRecovery - ? { - ...routeFromRouting, - mode: "extractive" as const, - model: null, - reason: `${routeFromRouting.reason}; validated_routine_extractive_recovery`, - } - : routeFromRouting; - const gatedRoute = - validatedGenericLaiManagementExtractiveAnswer || validatedRoutineExtractiveRecovery - ? { route: routeBeforeConfidenceGate } - : applyConfidenceGate(routeBeforeConfidenceGate, queryClass, initialRetrievalDiagnostics); + : routeFromRouting; + const gatedRoute = validatedExtractiveShortCircuit + ? { route: routeBeforeConfidenceGate } + : applyConfidenceGate(routeBeforeConfidenceGate, queryClass, initialRetrievalDiagnostics); // In source-only mode (offline, or auto with no usable key) we never call the model. Route to // the deterministic extractive path when evidence is usable, but preserve the confidence gate's // "unsupported" decision so weak evidence still fails closed to a source-gap answer rather than diff --git a/tests/rag-answer-fallback.test.ts b/tests/rag-answer-fallback.test.ts index ccb49dbb..9a9c04af 100644 --- a/tests/rag-answer-fallback.test.ts +++ b/tests/rag-answer-fallback.test.ts @@ -2215,6 +2215,203 @@ describe("RAG structured-output fallback", () => { expect(answer.grounded).toBe(true); }); + // Pre-generation validated-extractive short-circuit (rag-extractive-first). The titles below + // deliberately share no topic token with the query so routing cannot take the existing + // title-supported extractive branch: the route must be fast/"strong_routine_retrieval" with a + // passed gate — the measured wasted-generation shape. + const proceduralFirstSources = () => [ + source({ + id: "procedural-first-reference", + document_id: "procedural-first-doc", + title: "Consumer Crisis Response Guideline", + file_name: "consumer-crisis-response.pdf", + section_heading: "Related procedures", + content: + "Related procedures and guidelines. Women's and Perinatal Mental Health Referral and Management Guideline.", + similarity: 0.9, + hybrid_score: 0.9, + text_rank: 0.12, + }), + source({ + id: "procedural-first-requirements", + document_id: "procedural-first-doc", + title: "Consumer Crisis Response Guideline", + file_name: "consumer-crisis-response.pdf", + section_heading: "Safety planning for identified risks", + content: + "The Consumer Safety Plan must be developed in collaboration with the consumer, involve carers and family where appropriate, identify actions for a crisis and who is responsible, and be reviewed when clinical status changes.", + similarity: 0.9, + hybrid_score: 0.9, + text_rank: 0.11, + }), + ]; + + it("short-circuits a validated gate-passed routine procedural answer before model generation", async () => { + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.stubEnv("RAG_SEARCH_CACHE_TTL_MS", "0"); + vi.stubEnv("RAG_ANSWER_CACHE_TTL_MS", "0"); + + const sources = proceduralFirstSources(); + const rpc = vi.fn(async (name: string) => { + if (retrievalRpcBaseName(name) === "match_document_chunks_text") return { data: sources, error: null }; + if (retrievalRpcBaseName(name) === "get_related_document_metadata") return { data: [], error: null }; + return { data: [], error: null }; + }); + const generateStructuredTextResult = vi.fn(); + + vi.doMock("@/lib/supabase/admin", () => ({ + createAdminClient: () => ({ + rpc, + from: vi.fn(() => new EmptyQuery()), + }), + })); + vi.doMock("@/lib/openai", () => ({ + embedTextWithTelemetry: vi.fn(async () => ({ embedding: [0.1, 0.2, 0.3], cacheHit: false })), + generateStructuredTextResult, + })); + + const { answerQuestionWithScope } = await import("../src/lib/rag/rag"); + const answer = await answerQuestionWithScope({ + query: "What should a patient safety plan include?", + ownerId: undefined, + logQuery: false, + skipCache: true, + }); + + expect(generateStructuredTextResult).not.toHaveBeenCalled(); + expect(answer.routingMode).toBe("extractive"); + expect(answer.routingReason).toContain("validated_routine_extractive_first"); + expect(answer.retrievalDiagnostics).toMatchObject({ gateStatus: "passed", topScore: 0.9 }); + expect(answer.grounded).toBe(true); + expect(answer.confidence).not.toBe("unsupported"); + expect(answer.citations.length).toBeGreaterThan(0); + }); + + it("keeps gate-passed 'How is…' document-content queries on model synthesis under the procedural short-circuit", async () => { + const answer = await answerFromTextSources("How is patient safety planning handled?", proceduralFirstSources(), { + answer: + "Patient safety planning is handled collaboratively with the consumer and reviewed when clinical status changes.", + grounded: true, + confidence: "medium", + answerSections: [], + citations: [{ chunk_id: "procedural-first-requirements" }], + quoteCards: [], + conflictsOrGaps: [], + }); + + expect(answer.routingMode).toBe("fast"); + expect(answer.routingReason).toBe("strong_routine_retrieval"); + expect(answer.routingReason).not.toContain("validated_routine_extractive_first"); + expect(answer.modelUsed).not.toBeNull(); + expect(answer.openAIRequestIds).toEqual(["req_answer_from_text_sources"]); + }); + + it("keeps a procedural query on model synthesis when the only extractive candidate fails the final gates", async () => { + vi.stubEnv("OPENAI_API_KEY", "test-key"); + vi.stubEnv("RAG_SEARCH_CACHE_TTL_MS", "0"); + vi.stubEnv("RAG_ANSWER_CACHE_TTL_MS", "0"); + + // Only a bare cross-reference chunk is available: it redirects to another named document + // (see isBareCrossReferenceAnswer) and answers nothing itself, so the extractive candidate + // fails the final gates and the short-circuit must refuse — generation still runs. + const sources = [ + source({ + id: "procedural-first-cross-reference", + document_id: "procedural-first-doc", + title: "Consumer Crisis Response Guideline", + file_name: "consumer-crisis-response.pdf", + section_heading: "Related procedures", + content: + "Refer to the Women's and Perinatal Mental Health Referral and Management Guideline for further information about related procedures.", + similarity: 0.9, + hybrid_score: 0.9, + text_rank: 0.12, + }), + ]; + const rpc = vi.fn(async (name: string) => { + if (retrievalRpcBaseName(name) === "match_document_chunks_text") return { data: sources, error: null }; + if (retrievalRpcBaseName(name) === "get_related_document_metadata") return { data: [], error: null }; + return { data: [], error: null }; + }); + const generateStructuredTextResult = vi.fn(async () => ({ + text: JSON.stringify({ + answer: "A patient safety plan should include collaboratively developed actions for identified risks.", + grounded: true, + confidence: "medium", + answerSections: [], + citations: [{ chunk_id: "procedural-first-cross-reference" }], + quoteCards: [], + conflictsOrGaps: [], + }), + model: "gpt-4.1-mini", + operation: "answer", + latencyMs: 12, + requestId: "req_procedural_first_negative", + usage: { input_tokens: 120, output_tokens: 80, total_tokens: 200 }, + })); + + vi.doMock("@/lib/supabase/admin", () => ({ + createAdminClient: () => ({ + rpc, + from: vi.fn(() => new EmptyQuery()), + }), + })); + vi.doMock("@/lib/openai", () => ({ + embedTextWithTelemetry: vi.fn(async () => ({ embedding: [0.1, 0.2, 0.3], cacheHit: false })), + generateStructuredTextResult, + })); + + const { answerQuestionWithScope } = await import("../src/lib/rag/rag"); + const answer = await answerQuestionWithScope({ + query: "What should a patient safety plan include?", + ownerId: undefined, + logQuery: false, + skipCache: true, + }); + + expect(generateStructuredTextResult).toHaveBeenCalled(); + expect(answer.routingReason).not.toContain("validated_routine_extractive_first"); + }); + + it("keeps dose-class procedural queries on model synthesis under the procedural short-circuit", async () => { + const query = "What is the maximum lithium dose process?"; + const { classifyRagQuery } = await import("../src/lib/clinical-search"); + // Pin the classification this negative depends on: dose-risk classes must never be + // eligible for the routine procedural short-circuit. + expect(classifyRagQuery(query).queryClass).toBe("medication_dose_risk"); + + const answer = await answerFromTextSources( + query, + [ + source({ + id: "lithium-dose-1", + document_id: "lithium-doc", + title: "Lithium Therapy Guideline", + file_name: "lithium-therapy.pdf", + section_heading: "Dosing", + content: + "Lithium dosing follows the documented titration process with plasma level monitoring and clinical review before any dose change.", + similarity: 0.9, + hybrid_score: 0.9, + text_rank: 0.12, + }), + ], + { + answer: "The maximum lithium dose process is described in the cited dosing guidance.", + grounded: true, + confidence: "medium", + answerSections: [], + citations: [{ chunk_id: "lithium-dose-1" }], + quoteCards: [], + conflictsOrGaps: [], + }, + ); + + expect(answer.routingReason).not.toContain("validated_routine_extractive_first"); + expect(answer.modelUsed).not.toBeNull(); + expect(answer.openAIRequestIds).toEqual(["req_answer_from_text_sources"]); + }); + it("does not gate-block a moderate score clustered across several distinct documents", async () => { // Regression: a topic with rich coverage (e.g. clozapine) returns many relevant // documents whose scores cluster tightly at a moderate value. A tiny spread there diff --git a/tests/rag-extractive-first.test.ts b/tests/rag-extractive-first.test.ts new file mode 100644 index 00000000..53fc14f1 --- /dev/null +++ b/tests/rag-extractive-first.test.ts @@ -0,0 +1,204 @@ +import { describe, expect, it } from "vitest"; + +import { + chooseValidatedExtractiveShortCircuit, + hasValidatedExtractiveCandidate, + hasValidatedRoutineProceduralExtractiveAnswer, + routineProceduralContentPattern, +} from "../src/lib/rag/rag-extractive-first"; +import type { SearchResult } from "../src/lib/types"; + +function source(overrides: Partial = {}): SearchResult { + return { + id: "procedural-chunk-1", + document_id: "procedural-doc", + title: "Consumer Crisis Response Guideline", + file_name: "consumer-crisis-response.pdf", + page_number: 4, + chunk_index: 0, + section_heading: "Safety planning for identified risks", + content: + "The Consumer Safety Plan must be developed in collaboration with the consumer, involve carers and family where appropriate, identify actions for a crisis and who is responsible, and be reviewed when clinical status changes.", + image_ids: [], + similarity: 0.9, + hybrid_score: 0.9, + text_rank: 0.11, + source_metadata: { + source_title: "Crisis response source", + publisher: "Local service", + jurisdiction: "Australia/WA", + version: "1", + publication_date: null, + review_date: null, + uploaded_at: null, + indexed_at: null, + uploaded_by: null, + document_status: "current", + clinical_validation_status: "approved", + extraction_quality: "good", + }, + images: [], + ...overrides, + }; +} + +const proceduralQuery = "What should a patient safety plan include?"; + +function proceduralArgs(overrides: Partial[0]> = {}) { + return { + query: proceduralQuery, + queryClass: "document_lookup" as const, + results: [source()], + route: { mode: "fast" as const, reason: "strong_routine_retrieval" }, + sourceBacked: true, + gateStatus: "passed" as const, + ...overrides, + }; +} + +describe("routineProceduralContentPattern", () => { + it("matches 'What …' questions that ask about process, inclusion, or requirements", () => { + expect(routineProceduralContentPattern.test("What should a patient safety plan include?")).toBe(true); + expect(routineProceduralContentPattern.test("what steps are required for seclusion documentation")).toBe(true); + expect(routineProceduralContentPattern.test(" What is the admission procedure?")).toBe(true); + expect(routineProceduralContentPattern.test("What does the transfer process involve?")).toBe(true); + }); + + it("deliberately excludes 'How is X handled/managed?' shapes and non-procedural questions", () => { + expect(routineProceduralContentPattern.test("How is patient safety planning handled?")).toBe(false); + expect(routineProceduralContentPattern.test("How are long acting injectables managed?")).toBe(false); + expect(routineProceduralContentPattern.test("What is clozapine?")).toBe(false); + expect(routineProceduralContentPattern.test("Describe the discharge process")).toBe(false); + expect(routineProceduralContentPattern.test("Somewhat unclear process question")).toBe(false); + }); +}); + +describe("hasValidatedRoutineProceduralExtractiveAnswer", () => { + it("accepts the gate-passed strong routine procedural shape whose candidate validates", () => { + expect(hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs())).toBe(true); + }); + + it("accepts the unsupported_or_general query class alongside document_lookup", () => { + expect( + hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs({ queryClass: "unsupported_or_general" })), + ).toBe(true); + }); + + it("refuses every off-shape input", () => { + expect( + hasValidatedRoutineProceduralExtractiveAnswer( + proceduralArgs({ route: { mode: "fast", reason: "clinical_fast_grounded_synthesis" } }), + ), + ).toBe(false); + expect( + hasValidatedRoutineProceduralExtractiveAnswer( + proceduralArgs({ route: { mode: "strong", reason: "strong_routine_retrieval" } }), + ), + ).toBe(false); + expect(hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs({ gateStatus: "blocked" }))).toBe(false); + expect(hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs({ sourceBacked: false }))).toBe(false); + expect(hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs({ queryClass: "medication_dose_risk" }))).toBe( + false, + ); + expect(hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs({ results: [] }))).toBe(false); + expect( + hasValidatedRoutineProceduralExtractiveAnswer( + proceduralArgs({ query: "How is patient safety planning handled?" }), + ), + ).toBe(false); + }); + + it("refuses when the only extractive candidate is a bare cross-reference that fails the final gates", () => { + const crossReferenceOnly = [ + source({ + id: "procedural-cross-reference", + section_heading: "Related procedures", + content: + "Refer to the Women's and Perinatal Mental Health Referral and Management Guideline for further information about related procedures.", + }), + ]; + expect(hasValidatedRoutineProceduralExtractiveAnswer(proceduralArgs({ results: crossReferenceOnly }))).toBe(false); + }); +}); + +describe("chooseValidatedExtractiveShortCircuit", () => { + it("returns the generic LAI marker first for the gate-passed LAI-management shape", () => { + const laiResults = [ + source({ + id: "lai-management-1", + document_id: "lai-doc", + title: "Long Acting Injectable Medication", + file_name: "MHSP.LongActingInjectable.pdf", + section_heading: "Management process", + content: + "Long acting injectables are managed through a documented medication pathway covering prescribing, administration, observation, follow-up, and clinical review.", + match_explanation: { titleHit: true, contentHit: true, reasons: ["title", "content"] }, + }), + source({ + id: "lai-management-2", + document_id: "lai-doc", + title: "Long Acting Injectable Medication", + file_name: "MHSP.LongActingInjectable.pdf", + section_heading: "Follow-up", + content: + "The long acting injectable medication record documents the prescription and administration, with follow-up and review arranged through the treating team.", + match_explanation: { titleHit: true, contentHit: true, reasons: ["title", "content"] }, + }), + ]; + const args = proceduralArgs({ + query: "How are long acting injectables managed?", + queryClass: "medication_dose_risk", + results: laiResults, + route: { mode: "fast", reason: "clinical_fast_grounded_synthesis" }, + }); + + expect(chooseValidatedExtractiveShortCircuit(args)).toEqual({ + reasonMarker: "validated_generic_lai_management_extractive_answer", + }); + // The LAI skip is only offered while the retrieval gate passed. + expect(chooseValidatedExtractiveShortCircuit({ ...args, gateStatus: "blocked" })).toBeNull(); + }); + + it("returns the blocked-recovery marker for the score-blocked routine document lookup", () => { + expect(chooseValidatedExtractiveShortCircuit(proceduralArgs({ gateStatus: "blocked" }))).toEqual({ + reasonMarker: "validated_routine_extractive_recovery", + }); + }); + + it("returns the procedural-first marker for the gate-passed routine procedural shape", () => { + expect(chooseValidatedExtractiveShortCircuit(proceduralArgs())).toEqual({ + reasonMarker: "validated_routine_extractive_first", + }); + }); + + it("returns null when no validated short-circuit applies", () => { + expect( + chooseValidatedExtractiveShortCircuit(proceduralArgs({ query: "How is patient safety planning handled?" })), + ).toBeNull(); + expect( + chooseValidatedExtractiveShortCircuit( + proceduralArgs({ route: { mode: "strong", reason: "limited_retrieval_strength" } }), + ), + ).toBeNull(); + expect(chooseValidatedExtractiveShortCircuit(proceduralArgs({ sourceBacked: false }))).toBeNull(); + }); +}); + +describe("cross-reference hardening (governance P2)", () => { + it("rejects a candidate whose lead is a query-overlapping bare cross-reference", () => { + const results = [ + source({ + content: + "Refer to the Patient Safety Planning Procedure for further information about what a patient safety plan should include.", + }), + ]; + expect( + hasValidatedExtractiveCandidate({ + query: "What should a patient safety plan include?", + queryClass: "document_lookup", + results, + routeReason: "strong_routine_retrieval", + }), + ).toBe(false); + }); +});