fix(rag): recover grounded source-backed canary answers#603
Conversation
Use validated deterministic recovery for routine document-content queries and scope clinical-value checks to claim-supporting citations. Focused RAG safety tests and the offline RAG contract pass.
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
📝 WalkthroughWalkthroughRAG answer finalization now supports claim-scoped numeric verification, document lookup intent is narrowed, and blocked routine document retrievals can use validated extractive recovery. Tests cover numeric grounding, source selection, routing outcomes, and model bypass behavior. ChangesRAG quality and routing
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related PRs
Sequence Diagram(s)sequenceDiagram
participant RAGAnswerFlow
participant RetrievalDiagnostics
participant buildExtractiveAnswer
participant finalizeRagAnswerQuality
RAGAnswerFlow->>RetrievalDiagnostics: inspect blocked routine retrieval
RAGAnswerFlow->>buildExtractiveAnswer: build extractive candidate
buildExtractiveAnswer->>finalizeRagAnswerQuality: finalize candidate
finalizeRagAnswerQuality-->>RAGAnswerFlow: return grounded candidate
RAGAnswerFlow->>RAGAnswerFlow: select extractive route before confidence gating
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5638931fdb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. After fixing or dispositioning a thread, reply in that thread with as the first line, followed by a concise summary; that marker authorizes the workflow to close that exact thread. If human input or new authorization is required, do not use the marker and leave the thread open with the blocker. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation. |
Preserve model synthesis for gate-passed routine document queries while retaining validated extractive recovery for blocked low-signal cases. Focused tests, ESLint, and Prettier pass.
|
@copilot resolve the merge conflicts in this pull request |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c0eb3e7aa1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…very # Conflicts: # src/lib/answer-verification.ts
Resolved on |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e476df734d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const claimScopedValues = (answer.supportedClaims ?? []).filter( | ||
| (claim) => extractClinicalValueAtoms(claim.text).length > 0, | ||
| ); | ||
| if (claimScopedValues.length > 0) { |
There was a problem hiding this comment.
When any supported claim contains a clinical value, this branch skips the previous raw answer/section scan and only verifies values that made it into supportedClaims. claimInputs derives those claims via splitClaims, which drops short snippets under eight characters, so a model section like Maximum dose: 500 mg (body 500 mg) can bypass numeric faithfulness whenever another longer supported numeric claim is present; before this change every section body was scanned directly. Please keep the claim-scoped check but also scan answer/section text for clinical value atoms that were not covered by a claim.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
src/lib/rag-extractive-answer.ts (1)
335-352: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated evidence regex between
answerIntentEvidencePatterndefault andfactSupportsAnswerIntentgeneral branch.The default-intent regex here (Lines 350) is copy-pasted verbatim into
factSupportsAnswerIntent's general/default branch (Lines 743-745). Since both lists gate what counts as "general" clinical evidence, any future tuning of one without the other will silently desynchronize evidence-pattern matching from fact-support filtering.♻️ Extract a shared constant
+const GENERAL_INTENT_EVIDENCE_PATTERN = + /\b(?:assess|arrange|check|collaborat\w*|complete|conduct|continue|develop|diagnos\w*|document|dose|ensure|identify|include|incorporate|involve|link|manage|monitor|provide|record|refer|revise|review\w*|risk|share|therapy|treat|update)\b/i; + function answerIntentEvidencePattern(intent: AnswerIntent) { switch (intent) { ... default: - return /\b(?:assess|arrange|check|...)\b/i; + return GENERAL_INTENT_EVIDENCE_PATTERN; } }Then reuse
GENERAL_INTENT_EVIDENCE_PATTERNat Line 743-745.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag-extractive-answer.ts` around lines 335 - 352, Extract the default regex from answerIntentEvidencePattern into a shared GENERAL_INTENT_EVIDENCE_PATTERN constant, then return or reuse that constant for the default case. Update factSupportsAnswerIntent’s general/default branch to use the same constant, preserving the existing matching behavior while eliminating the duplicated pattern.src/lib/rag.ts (1)
916-961: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy liftTrial candidate duplicates the full extractive build+finalize pass that runs again for the real answer.
hasValidatedRoutineExtractiveRecoverycallsbuildExtractiveAnswer+finalizeRagAnswerQualitypurely to validate; when it returnstrue, the extractive branch at Lines 4463-4538 rebuilds the samebuildExtractiveAnswer(with fuller metadata) and callsfinalizeRagAnswerQualityagain — meaning fact synthesis, citation compaction, memory-card ranking, claim-support assessment, and numeric verification each run twice for exactly the queries this PR is designed to recover (i.e. this is expected to become a real, non-rare path). SincerelatedDocumentsisn't resolved yet at validation time, fully caching/reusing the trial candidate would require restructuring the async ordering; a smaller win is available now: the outer-scopequoteCards(Line 4166) is already computed by validation time and could be passed in here instead of[]to avoid one extraextractQuoteCardscall.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag.ts` around lines 916 - 961, Update hasValidatedRoutineExtractiveRecovery to accept the already-computed outer-scope quoteCards and pass them into its buildExtractiveAnswer call instead of creating an empty list. Update every invocation to provide quoteCards, preserving the existing validation flow while eliminating the duplicate extractQuoteCards work.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/lib/rag-extractive-answer.ts`:
- Around line 335-352: Extract the default regex from
answerIntentEvidencePattern into a shared GENERAL_INTENT_EVIDENCE_PATTERN
constant, then return or reuse that constant for the default case. Update
factSupportsAnswerIntent’s general/default branch to use the same constant,
preserving the existing matching behavior while eliminating the duplicated
pattern.
In `@src/lib/rag.ts`:
- Around line 916-961: Update hasValidatedRoutineExtractiveRecovery to accept
the already-computed outer-scope quoteCards and pass them into its
buildExtractiveAnswer call instead of creating an empty list. Update every
invocation to provide quoteCards, preserving the existing validation flow while
eliminating the duplicate extractQuoteCards work.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: aaf7f029-58c8-4203-9ea3-f920314691c1
📒 Files selected for processing (5)
src/lib/answer-verification.tssrc/lib/rag-extractive-answer.tssrc/lib/rag.tstests/rag-answer-fallback.test.tstests/rag-trust.test.ts
Summary
strong_routine_retrievaldocument-content answers whose deterministic candidate passes the existing final quality, citation, claim-support, and numeric gates; retrieval-blocked recoveries remain visibly low-trust.Touched areas:
src/lib/rag.ts,src/lib/rag-extractive-answer.ts,src/lib/answer-verification.ts, and focused RAG trust/fallback tests.Verification
npm run verify:pr-local— not run locally because the sharednode_modulesjunction predates main's new@axe-core/playwrightdependency; clean hosted CI will install the committed lockfile.node scripts/run-vitest.mjs run tests/answer-verification.test.ts tests/rag-trust.test.ts tests/rag-answer-fallback.test.ts tests/rag-claim-support.test.ts tests/retrieval-query-variants.test.ts tests/rag-variant-early-exit.test.ts --reporter=dot— 6 files, 190 tests passed on current main.npm run eval:rag:offline— 36 fixture schemas; 21 files and 267 tests passed on current main.--max-warnings 0— passed for all five changed files.--check— passed for all five changed files.npm run eval:retrieval:quality— no retrieval-weight or ranking change; the prior live run for fix(rag): require co-located medication evidence #581 was 36/36 and exposed only answer-quality/latency failures.npm run eval:rag -- --limit 15+npm run eval:quality -- --rag-only— branch canary required before merge. One authorized single-case retry hit its 240-second provider/backoff timeout and produced no result; it was not repeated.npm run check:production-readiness— run with production secrets after merge/deployment if the branch canary and required CI pass.npm run verify:ui— not applicable; no UI/routing/styling changes.npm run verify:release— deferred until CI/canary completion.Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)Notes
Summary by CodeRabbit
New Features
Bug Fixes