diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index a6af85953..9f2bee673 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -20,6 +20,8 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | Date | Branch or ref | Reviewed HEAD | Scope | Outcome | Checks | | ---------- | -------------------------------------------------------- | ---------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 2026-07-24 | PR #1125 / `codex/answer-relevance-fail-closed` | `8d9fb2408f13e305138749655214baa0020fcfd4` | Follow-up: clear comparison/`documentBreakdown` in untrusted clinical notes | APPROVE for the scoped P2. `trustGatedAnswerForClinicalNotes` now clears `documentBreakdown`, `comparisonMatrix`, and `comparisonEvaluationState` when relevance is not source-backed, so Clinical Notes → ClinicalOutputPanel cannot rebuild comparison-detail tables from raw `best_quote` values. Prior visual/section/quote gates remain. Residual risk is still deliberate low-trust rendering for legacy payloads without `isSourceBacked: true`. | Focused jsdom/policy regressions: `tests/visual-evidence-tabs.dom.test.tsx` 5/5 after hardening the comparison case (caption + matrix values absent). Thread disposition posted and resolved. No live RAG/OpenAI/Supabase mutation. | +| 2026-07-24 | `codex/answer-relevance-fail-closed` | `ac0d4305478a0bc8fef03894b78ec5911912c08a` | Missing answer-relevance metadata across render policy and live dashboard grounding | APPROVE after resolving two review P2s. A shared `isAnswerSourceBacked` predicate now requires explicit `isSourceBacked: true`; missing or explicitly negative relevance cannot retain high render trust, a grounded dashboard state, visual/table evidence, or a clinical-notes table bypass. Explicitly source-backed answers preserve supported behavior. Retrieval, ranking, generation, source selection and stored data are unchanged. Highest residual risk is deliberate compatibility tightening for older answer payloads without relevance metadata; they render low-trust and expose review sources rather than richer evidence blocks. | Initial red policy proof failed with `high`; two later red proofs exposed retained visual evidence and the clinical-notes raw-table affordance, then passed after both render-model gates. Focused render/provenance/clinical-safety tests passed 37/37; the focused DOM/policy pair passed 31/31. Offline RAG passed 21 suites/308 tests and 36/36 fixtures; production-readiness was READY against `Clinical KB Database` read-only; `verify:pr-local` passed runtime, formatting, lint, typecheck, all 366 test files (3,254 passed/1 skipped), production build (1,677 pages), client-bundle secret scan and RAG fixture validation; the earlier local `verify:ui` passed 267/267. After the final UI fix, `verify:cheap` again passed all 20 non-test gates, lint, typecheck and 3,254 tests, with only tracked issue #067 timing out under machine load; its isolated retry also exceeded the same 30-second limit and was not repeated. Fresh exact-head hosted checks are required. No live RAG, OpenAI request, Supabase mutation, Railway action, production data operation or deployment ran. | | 2026-07-24 | `codex/safety-plan-no-patient-data-contract` | `b94987c94537f3114a3429848fa908bdecd1d80a` | Safety Plan Generator identifier, local-state, copy, print, privacy-notice and PIA contract | APPROVE. No P0-P2 finding. The patient name/initials field is removed; the builder now asks for identifier-free minimum content, retains working state only in the mounted React component, and makes clipboard/print/PDF export an explicit handling boundary. The PIA and product privacy copy distinguish this local-only tool from provider-backed questions. Highest residual risk is outside Clinical KB: users must handle exported copies under an approved clinical-record process, which the UI now states at the export controls. | Privacy/component DOM 3/3 plus updated privacy-copy 2/2; focused Chromium copy/print/no-fetch-or-XHR 1/1; `verify:cheap` passed all 21 gates, 366 files and 3,245 tests with 1 skip; production-readiness READY using the existing canonical environment without a provider call; production build and client-bundle secret scan passed; offline RAG fixture/manifest 36 cases/21 suites passed. `verify:pr-local` passed runtime, formatting, lint and typecheck, then stopped on the unrelated load-sensitive `reconciliation-preflight` 30-second timeout; that test passed 5/5 isolated and the preceding full suite passed, so the unchanged five-minute gate was not retried. No Supabase, OpenAI, Railway, live RAG, production data or deployment action ran. | | 2026-07-24 | PR #1114 / `codex/universal-ledger-main-followup-20260724` | `eedbe594a89eaa0aad8a517ee8808b089cb5ff57` + reviewed working diff | Streamlined-sidebar handoff in the merged universal ledger | APPROVE. PRs #1112, #1116, and #1117 landed the universal-ledger baseline, two preserved UI tasks, and credential/hook hardening; this follow-up preserves that current-main schema and adds only the still-missing sidebar delivery as collision-free `#066`. The queue remains gap-free and every recommended ID has one open row. Sidebar implementation stays recoverable from remote branch `origin/codex/sidebar-test-fix-20260723` at full commit `cd54e68fbf7b07b5dffe3220e36af2caa528da54` until its local/browser/build/hosted merge proof is complete. | Focused sidebar/favourites tests previously passed 18/18 on the recorded implementation commit; this ledger follow-up passed Prettier, remote-ref recovery, 36-row contiguous-order/46-open-ID/next-ID invariants, and `git diff --check`. The broader reconciled precursor passed `verify:cheap` (365 files, 3,242 passed/1 skipped); fresh exact-head hosted CI is required on PR #1114. No OpenAI, Supabase, Railway, deployment, live-app, credential, or production-data action ran. | | 2026-07-24 | `codex/universal-ledger-main-final` | `527988c2ccabc98b4d0673d33360c971df65fa0e` + reviewed working diff | Current-main universal-ledger reconciliation after PR #1106 superseded PR #1109 | READY. Kept PR #1106's current-main ledger and IDs, carried forward only non-duplicate recommended work from the superseded branch, and fixed the confirmed SessionStart empty-open defect. Every active recommendation now has a durable open ID; exposed-GitHub-token containment is the first A1 item; the Safety Plan privacy contract, absent-relevance fail-closed rule, stranded-upload recovery, threshold-conflict design, catalogue-toolbar convergence, and Current Clinical Work brief are retained without duplicating #1106's legal/config/release/staging/seed packages. Resolved #014/#034 claims stay archived, and PR #1110's scheduled-diagnostics priority remains intact. Highest residual risk is manual queue/open-table drift. | Empty-open fixture and real-ledger Bash execution passed; 33 contiguous recommendations reference tracked open IDs; 43 open items; no duplicate queued IDs; `docs:check-links`, `docs:check-index`, `docs:check-scripts`, `check:skills`, Prettier, and `git diff --check` passed. No OpenAI, Supabase, Railway, production, deployment, live-app, or credential action. | diff --git a/src/components/ClinicalDashboard.tsx b/src/components/ClinicalDashboard.tsx index 51a31a707..a95f2fbc9 100644 --- a/src/components/ClinicalDashboard.tsx +++ b/src/components/ClinicalDashboard.tsx @@ -206,7 +206,7 @@ import { maxStoredAnswerTurns, savePersistedAnswerThread, } from "@/lib/answer-thread-storage"; -import { buildAnswerRenderModel } from "@/lib/answer-render-policy"; +import { buildAnswerRenderModel, isAnswerSourceBacked } from "@/lib/answer-render-policy"; import { frontendSourceGovernanceWarnings, groupSourceGovernanceWarnings, @@ -2907,7 +2907,7 @@ export function ClinicalDashboard({ const answerGrounded = answer?.grounded === true && answer.confidence !== "unsupported" && - currentRelevance?.isSourceBacked !== false && + isAnswerSourceBacked(answer) && answerRenderModel?.trust !== "unsupported"; const sourceLookup = useMemo(() => new Map(sources.map((source) => [source.id, source])), [sources]); const answerPreformatted = isPreformattedGroundedAnswer(answer); diff --git a/src/components/clinical-dashboard/answer-result-surface.tsx b/src/components/clinical-dashboard/answer-result-surface.tsx index 6051ac0e0..644bda03b 100644 --- a/src/components/clinical-dashboard/answer-result-surface.tsx +++ b/src/components/clinical-dashboard/answer-result-surface.tsx @@ -291,6 +291,7 @@ function StagedAnswerResultSurfaceImpl({ > item.accessibleTableMarkdown || item.tableRows?.length, - ).length; -} - function clinicalNotesRowsForTab( sections: ClinicalDetailSection[], tab: ClinicalNotesTabId, @@ -574,6 +568,36 @@ function clinicalNotesAvailableTabs(sections: ClinicalDetailSection[]) { .filter((tab) => tab.count > 0); } +/** + * Align clinical-notes inputs with the fail-closed render model: when an answer + * is not explicitly source-backed, strip structured clinical payloads so the + * notes sheet cannot reconstruct actionable monitoring/escalation/comparison + * content from untrusted sections, quotes, or documentBreakdown (visual + * evidence is passed separately). + */ +function trustGatedAnswerForClinicalNotes( + answer: RagAnswer, + visualEvidence: VisualEvidenceCard[] = answer.visualEvidence ?? [], +): RagAnswer { + if (isAnswerSourceBacked(answer)) { + return { + ...answer, + visualEvidence, + smartPanel: answer.smartPanel ? { ...answer.smartPanel, visualEvidence } : answer.smartPanel, + }; + } + return { + ...answer, + answerSections: [], + quoteCards: [], + documentBreakdown: [], + comparisonMatrix: undefined, + comparisonEvaluationState: undefined, + visualEvidence, + smartPanel: answer.smartPanel ? { ...answer.smartPanel, visualEvidence, quotes: [] } : answer.smartPanel, + }; +} + /** * Builds the non-empty clinical detail sections used by the clinical notes view. * @@ -598,13 +622,16 @@ function clinicalNotesDetailSectionsForAnswer(answer: RagAnswer, viewMode: Answe } export function clinicalNotesDisplayCountForAnswer(answer: RagAnswer, viewMode: AnswerViewMode, fallback: number) { - const tabs = clinicalNotesAvailableTabs(clinicalNotesDetailSectionsForAnswer(answer, viewMode)); + const tabs = clinicalNotesAvailableTabs( + clinicalNotesDetailSectionsForAnswer(trustGatedAnswerForClinicalNotes(answer), viewMode), + ); const largestTabCount = tabs.reduce((largest, tab) => Math.max(largest, tab.count), 0); return Math.max(1, largestTabCount || fallback); } export function ClinicalNotesChecklistPanel({ answer, + visualEvidence, viewMode, evidenceMapRows, sourceLinks = [], @@ -614,6 +641,7 @@ export function ClinicalNotesChecklistPanel({ onOpenTables, }: { answer: RagAnswer; + visualEvidence: VisualEvidenceCard[]; viewMode: AnswerViewMode; evidenceMapRows: AnswerEvidenceMapRow[]; sourceLinks?: SourceLink[]; @@ -622,7 +650,8 @@ export function ClinicalNotesChecklistPanel({ onCopy: () => void; onOpenTables?: () => void; }) { - const detailSections = clinicalNotesDetailSectionsForAnswer(answer, viewMode); + const renderableAnswer = trustGatedAnswerForClinicalNotes(answer, visualEvidence); + const detailSections = clinicalNotesDetailSectionsForAnswer(renderableAnswer, viewMode); const tabs = clinicalNotesAvailableTabs(detailSections); const defaultTab = tabs.find((tab) => tab.id === "actions")?.id ?? tabs[0]?.id ?? "actions"; const [requestedTab, setRequestedTab] = useState(defaultTab); @@ -632,14 +661,21 @@ export function ClinicalNotesChecklistPanel({ const notesPanelId = `${tabBaseId}-panel`; const activeTab = tabs.some((tab) => tab.id === requestedTab) ? requestedTab : defaultTab; const rows = clinicalNotesRowsForTab(detailSections, activeTab, sourceLinks, bestSource); - const tableEvidenceCount = clinicalNotesTableEvidenceCount(answer); + const tableEvidenceCount = visualEvidence.filter( + (item) => item.accessibleTableMarkdown || item.tableRows?.length, + ).length; const [added, setAdded] = useState(false); const warningRows = clinicalNotesRowsForTab(detailSections, "safety", sourceLinks, bestSource); const warningCount = warningRows.filter((row) => row.tone === "warn").length || warningRows.length; if (!tabs.length || rows.length === 0) { return ( - + ); } @@ -1027,7 +1063,7 @@ export function evidenceTabCount({ } export function clinicalNotesCount(answer: RagAnswer) { - return buildHighYieldClinicalOutputSections(answer).filter((section) => + return buildHighYieldClinicalOutputSections(trustGatedAnswerForClinicalNotes(answer)).filter((section) => ["action", "escalation", "thresholds", "cautions", "monitoring", "medication", "source-gap"].includes(section.id), ).length; } diff --git a/src/lib/answer-render-policy.ts b/src/lib/answer-render-policy.ts index 642144598..7973ca92d 100644 --- a/src/lib/answer-render-policy.ts +++ b/src/lib/answer-render-policy.ts @@ -139,10 +139,13 @@ function answerRelevance(answer: RagAnswer): EvidenceRelevance | undefined { return answer.relevance ?? answer.smartPanel?.relevance; } +export function isAnswerSourceBacked(answer: RagAnswer): boolean { + return answerRelevance(answer)?.isSourceBacked === true; +} + function deriveTrust(answer: RagAnswer): AnswerRenderTrust { - const relevance = answerRelevance(answer); const retrievalBlocked = answer.retrievalDiagnostics?.gateStatus === "blocked"; - const sourceBacked = relevance?.isSourceBacked !== false; + const sourceBacked = isAnswerSourceBacked(answer); const hasFaithfulnessWarning = Boolean(answer.faithfulnessWarning || answer.unverifiedNumericTokens?.length); const evidenceGap = answer.responseMode === "evidence_gap"; @@ -742,7 +745,8 @@ export function buildAnswerRenderModel( const rawQuotes = answer.quoteCards ?? answer.smartPanel?.quotes ?? []; const rawVisualEvidence = answer.visualEvidence ?? answer.smartPanel?.visualEvidence ?? []; const rawRelatedDocuments = answer.relatedDocuments ?? answer.smartPanel?.relatedDocuments ?? []; - const visualLimit = hasDirectVisualNeed(answer) || trust === "high" ? caps.visual : 0; + const visualLimit = + isAnswerSourceBacked(answer) && (hasDirectVisualNeed(answer) || trust === "high") ? caps.visual : 0; const quoteCards = dedupeQuotes(rawQuotes, primarySources, caps.quotes); const visualEvidence = dedupeVisualEvidence(rawVisualEvidence, primarySources, visualLimit); const tables = buildCanonicalTables(visualEvidence); diff --git a/tests/answer-render-policy.test.ts b/tests/answer-render-policy.test.ts index 917b7db36..d7c73345d 100644 --- a/tests/answer-render-policy.test.ts +++ b/tests/answer-render-policy.test.ts @@ -97,6 +97,17 @@ function answer(overrides: Partial = {}): RagAnswer { answer: "For red-range blood results, withhold clozapine and contact the monitoring service.", grounded: true, confidence: "high", + relevance: { + verdict: "direct", + label: "Direct source support", + matchedTerms: ["clozapine", "monitoring"], + missingTerms: [], + directSourceCount: 1, + weakSourceCount: 0, + score: 0.95, + supportReason: "A direct source supports the answer.", + isSourceBacked: true, + }, citations: [citation()], sources: [baseSource], answerSections: [ @@ -278,6 +289,23 @@ describe("answer render policy", () => { expect(model.warnings).not.toContain("A supporting source is due for review."); }); + it("fails closed when answer relevance metadata is absent", () => { + const model = buildAnswerRenderModel( + answer({ + relevance: undefined, + smartPanel: undefined, + }), + ); + + expect(model.trust).toBe("low"); + expect(model.allowedBlocks).not.toContain("quoteCards"); + expect(model.allowedBlocks).not.toContain("visualEvidence"); + expect(model.allowedBlocks).not.toContain("relatedDocuments"); + expect(model.visualEvidence).toEqual([]); + expect(model.tables).toEqual([]); + expect(model.copyText).not.toContain("Displayed table evidence"); + }); + it("does not render high trust for high-risk claims supported only by unverified evidence", () => { const model = buildAnswerRenderModel( answer({ diff --git a/tests/visual-evidence-tabs.dom.test.tsx b/tests/visual-evidence-tabs.dom.test.tsx index 9db75dbec..6855c6821 100644 --- a/tests/visual-evidence-tabs.dom.test.tsx +++ b/tests/visual-evidence-tabs.dom.test.tsx @@ -2,6 +2,7 @@ import { render, screen, within } from "@testing-library/react"; import userEvent from "@testing-library/user-event"; import { describe, expect, it, vi } from "vitest"; +import { ClinicalNotesChecklistPanel } from "@/components/clinical-dashboard/evidence-panels"; import { MobileEvidenceSheetContent } from "@/components/clinical-dashboard/visual-evidence"; import type { AnswerRenderModel } from "@/lib/answer-render-policy"; import type { RagAnswer } from "@/lib/types"; @@ -137,3 +138,210 @@ describe("MobileEvidenceSheetContent tabs (jsdom)", () => { expect(claims).toHaveAttribute("tabindex", "-1"); }); }); + +describe("ClinicalNotesChecklistPanel visual-evidence boundary (jsdom)", () => { + it("does not expose raw table evidence suppressed by the render model", () => { + const answerWithRawTable: RagAnswer = { + ...answer, + answer: "Monitor renal function and escalate review for vomiting, dehydration, tremor, confusion, or ataxia.", + queryClass: "table_threshold", + responseMode: "threshold_table", + citations: [ + { + chunk_id: "chunk-1", + document_id: "doc-1", + title: "Lithium source", + file_name: "lithium.pdf", + page_number: 1, + chunk_index: 0, + }, + ], + answerSections: [ + { + heading: "Monitoring", + body: "Check lithium level, renal function, thyroid function, calcium, and interacting medicines.", + citation_chunk_ids: ["chunk-1"], + kind: "monitoring_timing", + supportLevel: "direct", + }, + { + heading: "Escalation", + body: "Escalate review for vomiting, dehydration, tremor, confusion, or ataxia.", + citation_chunk_ids: ["chunk-1"], + kind: "escalation_risk", + supportLevel: "direct", + }, + ], + visualEvidence: [ + { + id: "raw-table", + document_id: "doc-1", + image_id: "image-1", + source_chunk_id: "chunk-1", + title: "Raw threshold table", + file_name: "raw-table.pdf", + page_number: 1, + chunk_index: 0, + caption: "Raw table", + tableColumns: ["Threshold", "Action"], + tableRows: [ + ["0.49", "Withhold"], + ["1.0", "Monitor"], + ], + signed_url_endpoint: "/api/images/image-1", + viewer_href: "/documents/doc-1?page=1", + }, + ], + }; + + render( + , + ); + + expect(screen.queryByRole("button", { name: "Tables" })).not.toBeInTheDocument(); + expect(screen.queryByText("0.49")).not.toBeInTheDocument(); + expect(screen.queryByText("Withhold")).not.toBeInTheDocument(); + }); + + it("does not reconstruct clinical-notes sections from untrusted answerSections", () => { + const untrusted: RagAnswer = { + ...answer, + grounded: true, + // Missing relevance.isSourceBacked → fail closed for structured clinical UI. + answerSections: [ + { + heading: "Monitoring", + body: "Check lithium level every 3 months when stable.", + citation_chunk_ids: ["chunk-1"], + kind: "monitoring_timing", + supportLevel: "direct", + }, + { + heading: "Escalation", + body: "Escalate for tremor, confusion, or ataxia.", + citation_chunk_ids: ["chunk-1"], + kind: "escalation_risk", + supportLevel: "direct", + }, + ], + quoteCards: [ + { + chunk_id: "chunk-1", + document_id: "doc-1", + title: "Lithium source", + file_name: "lithium.pdf", + page_number: 1, + chunk_index: 0, + section_heading: "Monitoring", + quote: "Check lithium level every 3 months when stable.", + }, + ], + }; + + render( + , + ); + + expect(screen.queryByText("Check lithium level every 3 months when stable.")).not.toBeInTheDocument(); + expect(screen.queryByText("Escalate for tremor, confusion, or ataxia.")).not.toBeInTheDocument(); + }); + + it("does not render comparison tables from untrusted documentBreakdown", () => { + const comparisonAnswer: RagAnswer = { + ...answer, + grounded: true, + // Missing relevance.isSourceBacked → fail closed; documentBreakdown/comparison + // metadata must not rebuild ClinicalOutputPanel comparison-detail tables. + responseMode: "comparison_matrix", + queryClass: "comparison", + documentBreakdown: [ + { + document_id: "doc-a", + title: "Guideline A", + file_name: "a.pdf", + top_similarity: 0.9, + source_strength: "strong", + source_count: 1, + quote_count: 1, + pages: [1], + best_quote: "Guideline A prefers weekly lithium monitoring when unstable.", + }, + { + document_id: "doc-b", + title: "Guideline B", + file_name: "b.pdf", + top_similarity: 0.8, + source_strength: "moderate", + source_count: 1, + quote_count: 1, + pages: [2], + best_quote: "Guideline B prefers monthly lithium monitoring when stable.", + }, + ], + comparisonMatrix: { + documents: [ + { documentId: "doc-a", title: "Guideline A", fileName: "a.pdf" }, + { documentId: "doc-b", title: "Guideline B", fileName: "b.pdf" }, + ], + rows: [ + { + parameter: "Monitoring interval", + status: "conflict", + entries: [ + { + documentId: "doc-a", + chunkIds: ["chunk-a"], + value: "weekly when unstable", + qualifiers: [], + }, + { + documentId: "doc-b", + chunkIds: ["chunk-b"], + value: "monthly when stable", + qualifiers: [], + }, + ], + }, + ], + }, + comparisonEvaluationState: "evaluated", + }; + + render( + , + ); + + expect(screen.queryByText("Clinical comparison detail")).not.toBeInTheDocument(); + expect(screen.queryByText("Guideline A prefers weekly lithium monitoring when unstable.")).not.toBeInTheDocument(); + expect(screen.queryByText("Guideline B prefers monthly lithium monitoring when stable.")).not.toBeInTheDocument(); + expect(screen.queryByText("weekly when unstable")).not.toBeInTheDocument(); + expect(screen.queryByText("monthly when stable")).not.toBeInTheDocument(); + }); +});