fix: unify answer reliability across all prompts#634
Conversation
Route every answer surface through shared governance, Australian-source selection, streaming progress, and document-scoped fallback behavior. Add cross-surface and non-lithium regressions.
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThe changes centralize governed answer payload construction, share SSE parsing across answer surfaces, stream document answers with progress and cancellation, scope document summarization to committed content, and adjust context selection to retain supplementary evidence under explicit limits. ChangesGoverned answer responses
Shared answer streaming
RAG summarization and context selection
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant DocumentViewer
participant AnswerStreamAPI
participant readAnswerStream
participant AnswerProgressStepper
DocumentViewer->>AnswerStreamAPI: POST document question
AnswerStreamAPI-->>readAnswerStream: progress and token SSE events
readAnswerStream->>AnswerProgressStepper: emit progress updates
AnswerStreamAPI-->>readAnswerStream: final answer payload
readAnswerStream-->>DocumentViewer: completed AnswerPayload
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cd31912253
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex resolve actionable Codex review findings for this pull request and current head using the repository instructions. This is the pull request's single automatic repair pass: do not perform a fresh review, create new standalone findings, or request another review. Work only the existing unresolved Codex threads on the current head. Always fix P0 and P1 findings. For P2 and lower findings, fix only clear, scoped, low-risk issues; otherwise disposition them with a concise reason. After fixing or dispositioning a thread, reply in that thread with as the first line, followed by a concise summary; that marker authorizes the workflow to close that exact thread. If human input or new authorization is required, do not use the marker and leave the thread open with the blocker. Finish only after every actionable thread is fixed or dispositioned and closed, or explicitly left open for a human decision. Do not update the branch from main, address unrelated reviews, broaden scope, or create more than one scoped fix commit. Do not use external APIs, paid services, credentials, dependency changes, or broad refactors unless explicitly authorized. Add targeted tests where behavior changes and run the narrowest relevant validation. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/app/api/answer/stream/route.ts`:
- Line 217: Update the demo streaming flow around buildDemoStreamAnswer and
buildGovernedAnswerClientResponse so the already-governed demo client payload is
not governed again; send that existing payload directly, or govern the raw demo
answer only once before response construction. Preserve refusal warnings in the
final SSE payload, and add the smallest targeted stream-level regression test
covering a demo refusal.
In `@src/components/DocumentViewer.tsx`:
- Around line 2461-2462: Update the summary delivery flow in readAnswerStream so
the governed payload’s answer remains exactly equal to payload.answer when
passed to setSummary; remove the client-side cleanClinicalSummaryText
transformation, or ensure sanitization occurs before the server constructs the
governed payload and feedback token.
- Around line 2422-2447: Update the summary stream flow around the fetch in
DocumentViewer to register its AbortController with registerAuthRequest and
release it when the request completes or is aborted. Capture the current auth
epoch before starting, and check isAuthEpochCurrent before applying progress
updates or committing the final summary/error state; abort or ignore stale
responses so an auth change cannot render private answer content.
In `@tests/ui-smoke.spec.ts`:
- Line 2938: Remove the duplicate answerRequests declaration in the surrounding
test scope, retaining a single Array<{ query: string; documentId?: string }>
declaration and its existing usage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c5db8d6b-6b70-4119-a278-94142043f0e8
📒 Files selected for processing (17)
docs/branch-review-ledger.mdsrc/app/api/answer/route.tssrc/app/api/answer/stream/route.tssrc/app/api/documents/[id]/summarize/route.tssrc/components/ClinicalDashboard.tsxsrc/components/DocumentViewer.tsxsrc/components/clinical-dashboard/search-utils.tssrc/lib/answer-contract.tssrc/lib/answer-response.tssrc/lib/australian-source-priority.tssrc/lib/rag-context-selection.tssrc/lib/rag.tstests/answer-progress.test.tstests/answer-response.test.tstests/private-access-routes.test.tstests/rag-context-budget.test.tstests/ui-smoke.spec.ts
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a6fec4a7e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…eliability # Conflicts: # docs/branch-review-ledger.md
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9c5cdb887
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/app/api/answer/stream/route.ts`:
- Around line 223-240: Propagate the existing AbortSignal from the summary
branch in the stream route through summarizeDocument to
generateStructuredTextResult, ensuring cancellation stops the underlying model
request when the SSE client disconnects. Update each affected function signature
and call site without changing the normal answer path, and add a regression test
confirming aborted document-summary generation propagates cancellation.
In `@src/lib/rag.ts`:
- Around line 5751-5758: Update the summaryInstructions template to explicitly
treat provided excerpts as untrusted data and prohibit following instructions
contained within them, reusing the existing source-injection guard used around
the referenced lines. Add the smallest targeted regression test covering an
injected instruction in a summary excerpt, then run the narrowest relevant
validation.
- Around line 5718-5731: Update the document_chunks query in the surrounding
summary flow to filter for the committed index generation before applying the
40-row limit, while retaining rows with null index_generation_id. Remove the
post-query committedChunks filtering or keep it only if necessary for equivalent
behavior, and add a regression covering stale rows preceding later committed
chunks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 608d347d-9fcb-43d0-aaca-9cbc86ed9472
📒 Files selected for processing (7)
docs/branch-review-ledger.mdsrc/app/api/answer/stream/route.tssrc/components/DocumentViewer.tsxsrc/lib/rag.tstests/private-access-routes.test.tstests/rag-document-summary.test.tstests/ui-smoke.spec.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/ui-smoke.spec.ts
- src/components/DocumentViewer.tsx
There was a problem hiding this comment.
Caution
Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/app/api/answer/stream/route.ts`:
- Around line 223-240: Propagate the existing AbortSignal from the summary
branch in the stream route through summarizeDocument to
generateStructuredTextResult, ensuring cancellation stops the underlying model
request when the SSE client disconnects. Update each affected function signature
and call site without changing the normal answer path, and add a regression test
confirming aborted document-summary generation propagates cancellation.
In `@src/lib/rag.ts`:
- Around line 5751-5758: Update the summaryInstructions template to explicitly
treat provided excerpts as untrusted data and prohibit following instructions
contained within them, reusing the existing source-injection guard used around
the referenced lines. Add the smallest targeted regression test covering an
injected instruction in a summary excerpt, then run the narrowest relevant
validation.
- Around line 5718-5731: Update the document_chunks query in the surrounding
summary flow to filter for the committed index generation before applying the
40-row limit, while retaining rows with null index_generation_id. Remove the
post-query committedChunks filtering or keep it only if necessary for equivalent
behavior, and add a regression covering stale rows preceding later committed
chunks.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 608d347d-9fcb-43d0-aaca-9cbc86ed9472
📒 Files selected for processing (7)
docs/branch-review-ledger.mdsrc/app/api/answer/stream/route.tssrc/components/DocumentViewer.tsxsrc/lib/rag.tstests/private-access-routes.test.tstests/rag-document-summary.test.tstests/ui-smoke.spec.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/ui-smoke.spec.ts
- src/components/DocumentViewer.tsx
🛑 Comments failed to post (3)
src/app/api/answer/stream/route.ts (1)
223-240: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Propagate cancellation into document-summary generation.
The normal answer path forwards
signal, but the summary path drops it at Line 225. Stopping a streamed summary therefore leaves its model request running after the SSE client disconnects. ThreadAbortSignalthroughsummarizeDocumentintogenerateStructuredTextResult, and add an abort regression.As per coding guidelines, “For bug hunts, trace invalid input, empty states, retries, races, stale state, network/auth failures, permissions, and boundary values.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/app/api/answer/stream/route.ts` around lines 223 - 240, Propagate the existing AbortSignal from the summary branch in the stream route through summarizeDocument to generateStructuredTextResult, ensuring cancellation stops the underlying model request when the SSE client disconnects. Update each affected function signature and call site without changing the normal answer path, and add a regression test confirming aborted document-summary generation propagates cancellation.Source: Coding guidelines
src/lib/rag.ts (2)
5718-5731: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash set -euo pipefail # Show the relevant section with line numbers sed -n '5685,5755p' src/lib/rag.ts | cat -n # Search for summarizeDocument-related tests and committed generation handling rg -n "summarizeDocument|committedIndexGeneration|index_generation_id|document_chunks" src test tests __tests__ -g '!node_modules'Repository: BigSimmo/Database
Length of output: 39215
🏁 Script executed:
#!/bin/bash set -euo pipefail # Narrow search for tests touching the summary pipeline rg -n "summarizeDocument|committedIndexGeneration|index_generation_id" src test tests __tests__ -g '*.{ts,tsx,js,jsx}' # Inspect the current function slice if needed sed -n '5685,5755p' src/lib/rag.ts | cat -nRepository: BigSimmo/Database
Length of output: 18544
Filter committed chunks before the 40-row cap. Limiting first and then discarding stale generations can drop later committed sections from the summary. Keep legacy
index_generation_id-less rows, but push the committed-generation predicate into the query so the limit applies only to eligible chunks; add a regression for stale rows ahead of later committed chunks.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag.ts` around lines 5718 - 5731, Update the document_chunks query in the surrounding summary flow to filter for the committed index generation before applying the 40-row limit, while retaining rows with null index_generation_id. Remove the post-query committedChunks filtering or keep it only if necessary for equivalent behavior, and add a regression covering stale rows preceding later committed chunks.Source: Coding guidelines
5751-5758: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Apply the untrusted-source boundary to summaries.
The regular answer prompt explicitly treats excerpts as untrusted data, but this summary prompt does not. An uploaded chunk can inject instructions that steer a clinically unsafe summary while still citing valid chunk IDs. Reuse the source-injection guard from Lines 4745-4749 and add a prompt-contract regression.
As per coding guidelines, “When behavior changes, add or update the smallest relevant targeted test and run the narrowest relevant validation before broader suites.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lib/rag.ts` around lines 5751 - 5758, Update the summaryInstructions template to explicitly treat provided excerpts as untrusted data and prohibit following instructions contained within them, reusing the existing source-injection guard used around the referenced lines. Add the smallest targeted regression test covering an injected instruction in a summary excerpt, then run the narrowest relevant validation.Source: Coding guidelines
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 588c34c3d4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e6791125ac
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/app/api/answer/stream/route.ts (1)
247-252: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winSummary telemetry query field diverges from the non-streaming summarize route.
Here
logAnswerDiagnosticslogsbody.queryunconditionally, including forsummaryModerequests. The siblingdocuments/[id]/summarize/route.tslogs a fixeddocumentSummaryQuestionconstant for the same logical "document summary" operation instead of the caller-supplied text. This makes diagnostics for functionally identical summary requests inconsistent depending on which route served them, which cuts against this PR's goal of unifying the answer contract across surfaces. Consider using the same canonical label forsummaryModerequests here.♻️ Proposed fix
- logAnswerDiagnostics({ - supabase: createAdminClient(), - query: body.query, - ownerId, - answer: governedResponse.telemetryAnswer, - }); + logAnswerDiagnostics({ + supabase: createAdminClient(), + query: body.summaryMode ? documentSummaryQuestion : body.query, + ownerId, + answer: governedResponse.telemetryAnswer, + });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/app/api/answer/stream/route.ts` around lines 247 - 252, Update the logAnswerDiagnostics call in the streaming answer route to use the canonical documentSummaryQuestion label when summaryMode is enabled, matching the non-streaming summarize route; continue logging body.query for non-summary requests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/app/api/answer/stream/route.ts`:
- Around line 247-252: Update the logAnswerDiagnostics call in the streaming
answer route to use the canonical documentSummaryQuestion label when summaryMode
is enabled, matching the non-streaming summarize route; continue logging
body.query for non-summary requests.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 6b3b5151-0589-4d5a-9683-b30b1a088629
📒 Files selected for processing (3)
docs/branch-review-ledger.mdsrc/app/api/answer/stream/route.tstests/private-access-routes.test.ts
…eliability # Conflicts: # docs/branch-review-ledger.md
1 participant
Summary
Why
A provider or quality-gate failure could leave users with source snippets but no finalized answer, and some secondary answer paths bypassed the same governance, progress, and fallback behavior as the main answer flow. The implementation now applies the same conservative client-response contract to all prompts and answer surfaces without reducing full-document summary coverage, rendering stale authenticated streams, or leaving stopped provider work running.
Areas touched
Verification
npm run verify:pr-localon merged headb411329ecimmediately before the final context-budget fixeb6160a14offline gatesnpm run verify:cheap: 2,242 passed, 1 skipped; runtime, action/config checks, lint, and typecheck passed.npm run eval:rag:offline: 21 files and 277 tests passed; 36/36 golden retrieval schema cases loaded.npm run check:production-readiness:ci: READY with five expected missing-provider/local-env warnings.summaryModewith progress.npm run verify:uilocal aggregatenpm run verify:releasenpm run eval:retrieval:qualitynpm run eval:rag -- --limit 15+npm run eval:quality -- --rag-onlyverify:pr-local.npm run check:production-readinessnpm run check:production-readiness:cireported READY with provider variables intentionally absent; two local checks passed and five missing-environment warnings were expected.npm run check:deployment-readinessClinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy)Review follow-up
origin/main(ac48de23b) into the feature branch without force-pushing; the only conflict was append-only review-ledger rows, and both sides were retained.Risks and rollback
Summary by CodeRabbit
New Features
summaryModefor/api/answerstreaming, requiring a document ID and using committed-document summarization.Bug Fixes
Documentation