From cbf2873c1a2f56fe3df94b0b06e408209423a9c4 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:16:36 +0800 Subject: [PATCH 1/6] docs(issues): re-diagnose #018 (generation, not retrieval); close #019 Free diagnosis from the run #61 + #57 eval-canary artifacts (no spend, no provider calls, read-only). - #019 closed as NOT REPRODUCIBLE: both runs place "Admission of Community Patients (AKG).pdf" at ranks 4-5 with missingDocumentSubstrings [], documentRecallAt5 1, contentRecallAt5 1, and zero failing cases across the 36-case retrieval suite. The claim is false at both endpoints of the stated #57-#61 range. - #018 re-scoped from "retrieval-depth" to answer generation. Retrieval is healthy (grounded, 4-5 citations, 100% doc/content recall, content_mrr_at_10 0.9344). Two of the three cases return the source_backed_review_fallback boilerplate, which cannot carry a schedule; the third emits degenerate extractive text and is misclassified as document_lookup. Suite-wide rather than monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are route=extractive across dose/monitoring/pathway intents - substantially the same phenomenon as #021. No retrieval canary warranted. Docs-only. Co-Authored-By: Claude Opus 4.8 --- docs/outstanding-issues.md | 56 +++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index 8b2b6a2b..60d5a9ad 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -31,36 +31,36 @@ Durable, cross-session memory of everything still outstanding for this repo: ope ## Open items -| ID | Pri | Type | Summary | Detail / next action | Source | Added | -| ---- | --- | ----- | ------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -| #001 | P2 | task | Semantic reranking still gated off | `RAG_SEMANTIC_RERANK_ENABLED=false` from PR #901. Do not enable until the provider-backed 36/36 retrieval-quality gate **and** an ambiguity-focused canary are explicitly approved and recorded. | `docs/process-hardening.md` (Semantic reranking rollout debt); PR #901 | 2026-07-21 | -| #005 | P3 | rec | `finalScore` saturates at clamp ceiling | Base + ~40 stacked boosts routinely exceed 1.0, so strong matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. If ranking is ever revisited, break ties by the **pre-clamp** score rather than raising the `[0,1]` ceiling (downstream gates assume `[0,1]`). Ordering already sorts by the unbounded pre-clamp `rankScore` (`clinical-search.ts:1735,1927,1950-1955`), so the clamp confines only the reported confidence value, not result order. Not a defect on the current golden set; any change here is a protected RAG surface (canary required). | `docs/rag-hybrid-findings-and-todo.md` P1 item 4; `src/lib/clinical-search.ts:1735` | 2026-07-21 | -| #006 | P2 | issue | Globe "Language & region" button had no handler | Fixed in this PR to the disabled "Coming soon" placeholder convention (`aria-disabled` + sr-only note). Wire to a real language/region settings screen when one exists, then drop the placeholder state. | `src/components/clinical-dashboard/master-search-header.tsx:1829`; session 2026-07-21 | 2026-07-21 | -| #007 | P3 | rec | `/tools` vs `/?mode=tools` parallel Tools entry points | `/tools` (standalone `ApplicationsLauncherPage`) has no inbound in-app link; the sidebar Tools item uses `/?mode=tools`. Decide the canonical entry point and wire nav consistently, or drop the standalone `/tools` page + `/applications` redirect. Currently allowlisted in `tests/route-reachability.test.ts`. | `src/app/tools/page.tsx`; `src/app/applications/route.ts` | 2026-07-21 | -| #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | -| #010 | P3 | task | Un-built "Coming soon" controls across forms/favourites | ~10 disabled placeholders (forms refine/reset, favourites sort/add/new-set, move-to-set, remove-favourite). Correctly flagged (`aria-disabled` + "Coming soon"), not defects — wire when the underlying features land. | `forms-search-results-page.tsx`; `favourites-hub.tsx`; `favourites-command-library-page.tsx` | 2026-07-21 | -| #011 | P3 | task | Auth DB-connection allocation is operator-only | Supabase Auth (GoTrue) is capped at ~10 absolute DB connections (Supabase perf advisor). Switch to **percentage-based** allocation in the Supabase **dashboard** before the first compute scale-up — **not settable via SQL/MCP** (operator-owned). Verify via a staging soak + an approval-gated read-only advisor re-check. | `docs/auth-connection-cap-runbook.md`; `docs/process-hardening.md` (Known follow-up debts) | 2026-07-21 | -| #012 | P3 | rec | Slim the lazy cross-mode differentials chunk | `cross-mode-differentials.ts` is dynamically imported (correctly code-split **out** of the initial/dashboard bundle — verified), but it pulls the full ~860 KB differentials snapshot (~125 KB gzip lazy chunk) just to build a tiny `{slug,title,clinicalHinge}` + presentations + aliases catalog. A precomputed lightweight index (generator + drift check, like the `specifiers-content` split / medications `fields=index`) would cut that lazy chunk ~5–10×. Not a bundle leak — an M-effort slim. | `src/lib/cross-mode-differentials.ts`; `src/components/clinical-dashboard/cross-mode-links.tsx:150`; session 2026-07-21 (build:analyze) | 2026-07-21 | -| #013 | P3 | rec | Route-chunk + mockup catalogue JSON weight | `build:analyze`: `/specifiers` ships `specifiers-search-index.json` (~180 KB parsed), `/forms` ships `forms-catalog.json` (~132 KB), `/formulation` ships `formulation-content.json` (~52 KB, client-side local search — needs index/full split or a search endpoint, architectural). All route-scoped (not initial bundle). Also `*-mockups.tsx` (~100 KB across chunks) build though `/mockups` 404s in prod — exclude from the prod artifact. | session 2026-07-21 (build:analyze) | 2026-07-21 | -| #014 | P3 | rec | Realize the `next/image` win on signed previews | `next.config` `images` (AVIF + `*.supabase.co` `remotePatterns` pinned to the project host, from #1024) is currently inert — signed document/image previews still render as raw ``. Route them through `next/image` to actually get AVIF + lazy optimization. | #1024; `src/components/clinical-dashboard/signed-image.tsx`; session 2026-07-21 | 2026-07-21 | -| #016 | P3 | rec | "Big but not easy" structural + motion perf | Deferred larger levers: (a) nonce-CSP forces every product route to `ƒ Dynamic` (zero static generation) — evaluate Partial Prerendering / static shells for the static clinical catalogues (DSM/differentials/therapy/specifiers/formulation); (b) sidebar expand/collapse animates `grid-template-columns` (biggest smoothness cost, motion-gated — needs a transform-overlay rethink); (c) Therapy Compass fetches 692 KB / 2.5 MB JSON client-side (defer until interaction + confirm brotli); (d) settings/setup/admin dialogs static-imported into the home chunk (`next/dynamic` them). | session 2026-07-21 (build route table + design audit) | 2026-07-21 | -| #017 | P3 | task | Field Web-Vitals baseline via live Lighthouse | In-sandbox runtime vitals were blocked (prod server hard-requires Supabase secrets; dev-mode CLS measured excellent at 0.00–0.04, content-first pages 0.000). Run Lighthouse against `psychiatry.tools` for real LCP/INP/CLS to prioritize #012–#016 by measured impact rather than reasoning. | session 2026-07-21 (measurement pass) | 2026-07-21 | -| #018 | P2 | task | Monitoring retrieval-depth trio misses schedules | quality-lithium-monitoring-range, quality-antipsychotic-metabolic-monitoring, quality-adhd-medication-monitoring still fail targeting ("no schedule/interval") after the #1040 gate parity — their live chunk sets apparently surface no admissible schedule sentence. Diagnose via read-only chunk queries + run #61 dump artifact (30d retention) to split corpus-content vs retrieval-depth vs fact-extraction; fix shape TBD after diagnosis. A retrieval-side fix is a protected surface (canary pair, fresh ~$2-4 authorization). | run #61 targeting misses; branch-review-ledger 2026-07-21 | 2026-07-21 | -| #019 | P2 | task | Comparison-class coverage: admission doc never in top-5 | admission-discharge-comparison misses its expected admission-side doc on every canary (#57-#61) — top-5 carries discharge-planning + unrelated docs only. Same "why doesn't retrieval surface X" species as #018; diagnose jointly. Ground-truth already widened (#1020); the residual is genuine comparison-class retrieval coverage. | runs #57-#61; PR #1020 replay verdict | 2026-07-21 | -| #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | -| #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | -| #022 | P2 | task | Source-governance metadata refresh (operator) | Governance warning rate ~0.84 across canaries: stale/review-required/unknown source metadata on most top results. Operator work (document review-status attestation in the app), not a code defect. Next action: generate the prioritized refresh worklist ($0 read-only) and schedule a metadata pass. | runs #57-#61 Source Governance tables; docs/observability-slos.md | 2026-07-21 | -| #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | -| #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | +| ID | Pri | Type | Summary | Detail / next action | Source | Added | +| ---- | --- | ----- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| #001 | P2 | task | Semantic reranking still gated off | `RAG_SEMANTIC_RERANK_ENABLED=false` from PR #901. Do not enable until the provider-backed 36/36 retrieval-quality gate **and** an ambiguity-focused canary are explicitly approved and recorded. | `docs/process-hardening.md` (Semantic reranking rollout debt); PR #901 | 2026-07-21 | +| #005 | P3 | rec | `finalScore` saturates at clamp ceiling | Base + ~40 stacked boosts routinely exceed 1.0, so strong matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. If ranking is ever revisited, break ties by the **pre-clamp** score rather than raising the `[0,1]` ceiling (downstream gates assume `[0,1]`). Ordering already sorts by the unbounded pre-clamp `rankScore` (`clinical-search.ts:1735,1927,1950-1955`), so the clamp confines only the reported confidence value, not result order. Not a defect on the current golden set; any change here is a protected RAG surface (canary required). | `docs/rag-hybrid-findings-and-todo.md` P1 item 4; `src/lib/clinical-search.ts:1735` | 2026-07-21 | +| #006 | P2 | issue | Globe "Language & region" button had no handler | Fixed in this PR to the disabled "Coming soon" placeholder convention (`aria-disabled` + sr-only note). Wire to a real language/region settings screen when one exists, then drop the placeholder state. | `src/components/clinical-dashboard/master-search-header.tsx:1829`; session 2026-07-21 | 2026-07-21 | +| #007 | P3 | rec | `/tools` vs `/?mode=tools` parallel Tools entry points | `/tools` (standalone `ApplicationsLauncherPage`) has no inbound in-app link; the sidebar Tools item uses `/?mode=tools`. Decide the canonical entry point and wire nav consistently, or drop the standalone `/tools` page + `/applications` redirect. Currently allowlisted in `tests/route-reachability.test.ts`. | `src/app/tools/page.tsx`; `src/app/applications/route.ts` | 2026-07-21 | +| #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | +| #010 | P3 | task | Un-built "Coming soon" controls across forms/favourites | ~10 disabled placeholders (forms refine/reset, favourites sort/add/new-set, move-to-set, remove-favourite). Correctly flagged (`aria-disabled` + "Coming soon"), not defects — wire when the underlying features land. | `forms-search-results-page.tsx`; `favourites-hub.tsx`; `favourites-command-library-page.tsx` | 2026-07-21 | +| #011 | P3 | task | Auth DB-connection allocation is operator-only | Supabase Auth (GoTrue) is capped at ~10 absolute DB connections (Supabase perf advisor). Switch to **percentage-based** allocation in the Supabase **dashboard** before the first compute scale-up — **not settable via SQL/MCP** (operator-owned). Verify via a staging soak + an approval-gated read-only advisor re-check. | `docs/auth-connection-cap-runbook.md`; `docs/process-hardening.md` (Known follow-up debts) | 2026-07-21 | +| #012 | P3 | rec | Slim the lazy cross-mode differentials chunk | `cross-mode-differentials.ts` is dynamically imported (correctly code-split **out** of the initial/dashboard bundle — verified), but it pulls the full ~860 KB differentials snapshot (~125 KB gzip lazy chunk) just to build a tiny `{slug,title,clinicalHinge}` + presentations + aliases catalog. A precomputed lightweight index (generator + drift check, like the `specifiers-content` split / medications `fields=index`) would cut that lazy chunk ~5–10×. Not a bundle leak — an M-effort slim. | `src/lib/cross-mode-differentials.ts`; `src/components/clinical-dashboard/cross-mode-links.tsx:150`; session 2026-07-21 (build:analyze) | 2026-07-21 | +| #013 | P3 | rec | Route-chunk + mockup catalogue JSON weight | `build:analyze`: `/specifiers` ships `specifiers-search-index.json` (~180 KB parsed), `/forms` ships `forms-catalog.json` (~132 KB), `/formulation` ships `formulation-content.json` (~52 KB, client-side local search — needs index/full split or a search endpoint, architectural). All route-scoped (not initial bundle). Also `*-mockups.tsx` (~100 KB across chunks) build though `/mockups` 404s in prod — exclude from the prod artifact. | session 2026-07-21 (build:analyze) | 2026-07-21 | +| #014 | P3 | rec | Realize the `next/image` win on signed previews | `next.config` `images` (AVIF + `*.supabase.co` `remotePatterns` pinned to the project host, from #1024) is currently inert — signed document/image previews still render as raw ``. Route them through `next/image` to actually get AVIF + lazy optimization. | #1024; `src/components/clinical-dashboard/signed-image.tsx`; session 2026-07-21 | 2026-07-21 | +| #016 | P3 | rec | "Big but not easy" structural + motion perf | Deferred larger levers: (a) nonce-CSP forces every product route to `ƒ Dynamic` (zero static generation) — evaluate Partial Prerendering / static shells for the static clinical catalogues (DSM/differentials/therapy/specifiers/formulation); (b) sidebar expand/collapse animates `grid-template-columns` (biggest smoothness cost, motion-gated — needs a transform-overlay rethink); (c) Therapy Compass fetches 692 KB / 2.5 MB JSON client-side (defer until interaction + confirm brotli); (d) settings/setup/admin dialogs static-imported into the home chunk (`next/dynamic` them). | session 2026-07-21 (build route table + design audit) | 2026-07-21 | +| #017 | P3 | task | Field Web-Vitals baseline via live Lighthouse | In-sandbox runtime vitals were blocked (prod server hard-requires Supabase secrets; dev-mode CLS measured excellent at 0.00–0.04, content-first pages 0.000). Run Lighthouse against `psychiatry.tools` for real LCP/INP/CLS to prioritize #012–#016 by measured impact rather than reasoning. | session 2026-07-21 (measurement pass) | 2026-07-21 | +| #018 | P2 | task | Monitoring targeting misses are generation, not retrieval | **Re-diagnosed 2026-07-22 from the run #61 + #57 eval-canary artifacts (no spend).** NOT retrieval-depth: all three cases are grounded with 4-5 citations, and the 36-case retrieval suite is 100 pct doc/content recall in both runs (content_mrr_at_10 0.9344, zero failing cases). Cause is answer generation: lithium-monitoring-range and adhd-medication-monitoring return the `source_backed_review_fallback` boilerplate ("a full written answer could not be completed just now"), which cannot carry a schedule; antipsychotic-metabolic emits degenerate extractive text ("The guidance for metabolic is that compliance, monitoring and evaluation.") and is classified `document_lookup` when the case expects `medication_dose_risk`. Suite-wide, not monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are route=extractive across dose/monitoring/pathway intents - substantially the same phenomenon as #021. **No retrieval canary warranted.** Next: treat jointly with #021 as one generation-quality item; the cheap standalone fix is the antipsychotic query_class drift. | run #61/#57 eval-canary artifacts; session 2026-07-22 | 2026-07-21 | +| #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | +| #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | +| #022 | P2 | task | Source-governance metadata refresh (operator) | Governance warning rate ~0.84 across canaries: stale/review-required/unknown source metadata on most top results. Operator work (document review-status attestation in the app), not a code defect. Next action: generate the prioritized refresh worklist ($0 read-only) and schedule a metadata pass. | runs #57-#61 Source Governance tables; docs/observability-slos.md | 2026-07-21 | +| #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | +| #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | ## Resolved / archive Move resolved rows here with the resolution date and a one-line outcome. Keep them — do not delete. -| ID | Type | Summary | Outcome | Resolved | -| ---- | ---- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -| #003 | task | Staging tenancy release evidence outstanding | Ran GitHub Action and validated isolation | 2026-07-21 | -| #002 | task | Process-ownership fix not yet isolated on `main` | Fixed process isolation using child.pid termination | 2026-07-21 | -| #008 | rec | Dead href builders in `document-flow-routes.ts` | Not dead code (false positive): `documentReaderHref`/`documentEvidenceHref` are live via the mock wrappers in `src/components/document-search-mockups.tsx` + `src/components/master-document-flow-mockups.tsx` (rendered under `src/app/mockups/document-search/`) and covered by `tests/document-flow-routes.test.ts`; removing breaks the build. Only the production non-mock hrefs are unlinked from prod UI — a wiring gap, not dead code. | 2026-07-22 | -| #015 | task | Content-first fallback regression tests | Added `tests/registry-record-loader.dom.test.tsx` (8) + `tests/medication-record-page.dom.test.tsx` (6) covering content-first fallback paint, live swap-in, spinner/skeleton, error + not-found/unauthorized states, and the invariant that no authoritative verification badge shows before live governance reconciles (registry fixture-flag neutralization + medication governance-drop-on-error). | 2026-07-22 | -| #004 | rec | Rescope provider-gated RAG safety ideas | Closed obsolete — rescue source (754-line RAG-safety worktree) unrecoverable/pruned across all refs; answer-quality thresholds + deep-health already shipped on `main` (#585/#587); only cost-cap preflight was genuinely missing and, per session decision, dropped rather than re-filed. | 2026-07-22 | +| ID | Type | Summary | Outcome | Resolved | +| ---- | ---- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| #003 | task | Staging tenancy release evidence outstanding | Ran GitHub Action and validated isolation | 2026-07-21 | +| #002 | task | Process-ownership fix not yet isolated on `main` | Fixed process isolation using child.pid termination | 2026-07-21 | +| #008 | rec | Dead href builders in `document-flow-routes.ts` | Not dead code (false positive): `documentReaderHref`/`documentEvidenceHref` are live via the mock wrappers in `src/components/document-search-mockups.tsx` + `src/components/master-document-flow-mockups.tsx` (rendered under `src/app/mockups/document-search/`) and covered by `tests/document-flow-routes.test.ts`; removing breaks the build. Only the production non-mock hrefs are unlinked from prod UI — a wiring gap, not dead code. | 2026-07-22 | +| #015 | task | Content-first fallback regression tests | Added `tests/registry-record-loader.dom.test.tsx` (8) + `tests/medication-record-page.dom.test.tsx` (6) covering content-first fallback paint, live swap-in, spinner/skeleton, error + not-found/unauthorized states, and the invariant that no authoritative verification badge shows before live governance reconciles (registry fixture-flag neutralization + medication governance-drop-on-error). | 2026-07-22 | +| #004 | rec | Rescope provider-gated RAG safety ideas | Closed obsolete — rescue source (754-line RAG-safety worktree) unrecoverable/pruned across all refs; answer-quality thresholds + deep-health already shipped on `main` (#585/#587); only cost-cap preflight was genuinely missing and, per session decision, dropped rather than re-filed. | 2026-07-22 | +| #019 | task | Comparison-class coverage: admission doc never in top-5 | **Not reproducible (2026-07-22, $0).** The run #61 **and** #57 eval-canary artifacts both place `Admission of Community Patients (AKG).pdf` at ranks 4-5 with `missingDocumentSubstrings: []`, `documentRecallAt5: 1`, `contentRecallAt5: 1`, and zero failing cases across the 36-case retrieval suite. The claim is false at both endpoints of the stated #57-#61 range, so no retrieval canary was spent. | 2026-07-22 | From 5d153f0bb51ba85f77af81fb3431fdcea8ff608f Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:22:56 +0800 Subject: [PATCH 2/6] docs(governance): #022 prioritized refresh worklist; capture #025 reporting gap Generated read-only at $0 from the run #61 + #57 eval-canary artifacts. - New docs/source-governance-refresh-worklist-2026-07-22.md (successor to the 2026-07-02 priority list): disambiguates the two governance rates (0.8409 is per-case over the 44-case answer suite; 0.5976 is per-slot over 676 retrieval top-result slots), gives the burn-down curve, and lists the top 25 documents by surfaced slots. - Key reframing: 59 of 124 surfaced documents are review-required, but 38 (64%) are the BMJ published-reference tier all at clinical_validation_status "unverified" -> ONE attestation-policy decision, not 38 clinical reviews. The remaining 21 are genuine local WA health-service reviews. Top-10 documents clear 44% of flagged slots; top-20 clear 66%. - #025 (new): the canary's own "## Source Governance" table prints all zeros (Top results | 0) even though topResults carry full governance metadata, so the operator-facing log does not surface this at all. Docs-only. No provider calls, no code change. Co-Authored-By: Claude Opus 4.8 --- docs/outstanding-issues.md | 43 ++++---- ...-governance-refresh-worklist-2026-07-22.md | 101 ++++++++++++++++++ 2 files changed, 123 insertions(+), 21 deletions(-) create mode 100644 docs/source-governance-refresh-worklist-2026-07-22.md diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index 60d5a9ad..ab22477f 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -27,30 +27,31 @@ Durable, cross-session memory of everything still outstanding for this repo: ope - Resolving an item moves its row to **Resolved / archive** with the date and a one-line outcome — rows are archived, not deleted, so the history stays auditable. - + ## Open items -| ID | Pri | Type | Summary | Detail / next action | Source | Added | -| ---- | --- | ----- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -| #001 | P2 | task | Semantic reranking still gated off | `RAG_SEMANTIC_RERANK_ENABLED=false` from PR #901. Do not enable until the provider-backed 36/36 retrieval-quality gate **and** an ambiguity-focused canary are explicitly approved and recorded. | `docs/process-hardening.md` (Semantic reranking rollout debt); PR #901 | 2026-07-21 | -| #005 | P3 | rec | `finalScore` saturates at clamp ceiling | Base + ~40 stacked boosts routinely exceed 1.0, so strong matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. If ranking is ever revisited, break ties by the **pre-clamp** score rather than raising the `[0,1]` ceiling (downstream gates assume `[0,1]`). Ordering already sorts by the unbounded pre-clamp `rankScore` (`clinical-search.ts:1735,1927,1950-1955`), so the clamp confines only the reported confidence value, not result order. Not a defect on the current golden set; any change here is a protected RAG surface (canary required). | `docs/rag-hybrid-findings-and-todo.md` P1 item 4; `src/lib/clinical-search.ts:1735` | 2026-07-21 | -| #006 | P2 | issue | Globe "Language & region" button had no handler | Fixed in this PR to the disabled "Coming soon" placeholder convention (`aria-disabled` + sr-only note). Wire to a real language/region settings screen when one exists, then drop the placeholder state. | `src/components/clinical-dashboard/master-search-header.tsx:1829`; session 2026-07-21 | 2026-07-21 | -| #007 | P3 | rec | `/tools` vs `/?mode=tools` parallel Tools entry points | `/tools` (standalone `ApplicationsLauncherPage`) has no inbound in-app link; the sidebar Tools item uses `/?mode=tools`. Decide the canonical entry point and wire nav consistently, or drop the standalone `/tools` page + `/applications` redirect. Currently allowlisted in `tests/route-reachability.test.ts`. | `src/app/tools/page.tsx`; `src/app/applications/route.ts` | 2026-07-21 | -| #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | -| #010 | P3 | task | Un-built "Coming soon" controls across forms/favourites | ~10 disabled placeholders (forms refine/reset, favourites sort/add/new-set, move-to-set, remove-favourite). Correctly flagged (`aria-disabled` + "Coming soon"), not defects — wire when the underlying features land. | `forms-search-results-page.tsx`; `favourites-hub.tsx`; `favourites-command-library-page.tsx` | 2026-07-21 | -| #011 | P3 | task | Auth DB-connection allocation is operator-only | Supabase Auth (GoTrue) is capped at ~10 absolute DB connections (Supabase perf advisor). Switch to **percentage-based** allocation in the Supabase **dashboard** before the first compute scale-up — **not settable via SQL/MCP** (operator-owned). Verify via a staging soak + an approval-gated read-only advisor re-check. | `docs/auth-connection-cap-runbook.md`; `docs/process-hardening.md` (Known follow-up debts) | 2026-07-21 | -| #012 | P3 | rec | Slim the lazy cross-mode differentials chunk | `cross-mode-differentials.ts` is dynamically imported (correctly code-split **out** of the initial/dashboard bundle — verified), but it pulls the full ~860 KB differentials snapshot (~125 KB gzip lazy chunk) just to build a tiny `{slug,title,clinicalHinge}` + presentations + aliases catalog. A precomputed lightweight index (generator + drift check, like the `specifiers-content` split / medications `fields=index`) would cut that lazy chunk ~5–10×. Not a bundle leak — an M-effort slim. | `src/lib/cross-mode-differentials.ts`; `src/components/clinical-dashboard/cross-mode-links.tsx:150`; session 2026-07-21 (build:analyze) | 2026-07-21 | -| #013 | P3 | rec | Route-chunk + mockup catalogue JSON weight | `build:analyze`: `/specifiers` ships `specifiers-search-index.json` (~180 KB parsed), `/forms` ships `forms-catalog.json` (~132 KB), `/formulation` ships `formulation-content.json` (~52 KB, client-side local search — needs index/full split or a search endpoint, architectural). All route-scoped (not initial bundle). Also `*-mockups.tsx` (~100 KB across chunks) build though `/mockups` 404s in prod — exclude from the prod artifact. | session 2026-07-21 (build:analyze) | 2026-07-21 | -| #014 | P3 | rec | Realize the `next/image` win on signed previews | `next.config` `images` (AVIF + `*.supabase.co` `remotePatterns` pinned to the project host, from #1024) is currently inert — signed document/image previews still render as raw ``. Route them through `next/image` to actually get AVIF + lazy optimization. | #1024; `src/components/clinical-dashboard/signed-image.tsx`; session 2026-07-21 | 2026-07-21 | -| #016 | P3 | rec | "Big but not easy" structural + motion perf | Deferred larger levers: (a) nonce-CSP forces every product route to `ƒ Dynamic` (zero static generation) — evaluate Partial Prerendering / static shells for the static clinical catalogues (DSM/differentials/therapy/specifiers/formulation); (b) sidebar expand/collapse animates `grid-template-columns` (biggest smoothness cost, motion-gated — needs a transform-overlay rethink); (c) Therapy Compass fetches 692 KB / 2.5 MB JSON client-side (defer until interaction + confirm brotli); (d) settings/setup/admin dialogs static-imported into the home chunk (`next/dynamic` them). | session 2026-07-21 (build route table + design audit) | 2026-07-21 | -| #017 | P3 | task | Field Web-Vitals baseline via live Lighthouse | In-sandbox runtime vitals were blocked (prod server hard-requires Supabase secrets; dev-mode CLS measured excellent at 0.00–0.04, content-first pages 0.000). Run Lighthouse against `psychiatry.tools` for real LCP/INP/CLS to prioritize #012–#016 by measured impact rather than reasoning. | session 2026-07-21 (measurement pass) | 2026-07-21 | -| #018 | P2 | task | Monitoring targeting misses are generation, not retrieval | **Re-diagnosed 2026-07-22 from the run #61 + #57 eval-canary artifacts (no spend).** NOT retrieval-depth: all three cases are grounded with 4-5 citations, and the 36-case retrieval suite is 100 pct doc/content recall in both runs (content_mrr_at_10 0.9344, zero failing cases). Cause is answer generation: lithium-monitoring-range and adhd-medication-monitoring return the `source_backed_review_fallback` boilerplate ("a full written answer could not be completed just now"), which cannot carry a schedule; antipsychotic-metabolic emits degenerate extractive text ("The guidance for metabolic is that compliance, monitoring and evaluation.") and is classified `document_lookup` when the case expects `medication_dose_risk`. Suite-wide, not monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are route=extractive across dose/monitoring/pathway intents - substantially the same phenomenon as #021. **No retrieval canary warranted.** Next: treat jointly with #021 as one generation-quality item; the cheap standalone fix is the antipsychotic query_class drift. | run #61/#57 eval-canary artifacts; session 2026-07-22 | 2026-07-21 | -| #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | -| #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | -| #022 | P2 | task | Source-governance metadata refresh (operator) | Governance warning rate ~0.84 across canaries: stale/review-required/unknown source metadata on most top results. Operator work (document review-status attestation in the app), not a code defect. Next action: generate the prioritized refresh worklist ($0 read-only) and schedule a metadata pass. | runs #57-#61 Source Governance tables; docs/observability-slos.md | 2026-07-21 | -| #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | -| #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | +| ID | Pri | Type | Summary | Detail / next action | Source | Added | +| ---- | --- | ----- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | +| #001 | P2 | task | Semantic reranking still gated off | `RAG_SEMANTIC_RERANK_ENABLED=false` from PR #901. Do not enable until the provider-backed 36/36 retrieval-quality gate **and** an ambiguity-focused canary are explicitly approved and recorded. | `docs/process-hardening.md` (Semantic reranking rollout debt); PR #901 | 2026-07-21 | +| #005 | P3 | rec | `finalScore` saturates at clamp ceiling | Base + ~40 stacked boosts routinely exceed 1.0, so strong matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. If ranking is ever revisited, break ties by the **pre-clamp** score rather than raising the `[0,1]` ceiling (downstream gates assume `[0,1]`). Ordering already sorts by the unbounded pre-clamp `rankScore` (`clinical-search.ts:1735,1927,1950-1955`), so the clamp confines only the reported confidence value, not result order. Not a defect on the current golden set; any change here is a protected RAG surface (canary required). | `docs/rag-hybrid-findings-and-todo.md` P1 item 4; `src/lib/clinical-search.ts:1735` | 2026-07-21 | +| #006 | P2 | issue | Globe "Language & region" button had no handler | Fixed in this PR to the disabled "Coming soon" placeholder convention (`aria-disabled` + sr-only note). Wire to a real language/region settings screen when one exists, then drop the placeholder state. | `src/components/clinical-dashboard/master-search-header.tsx:1829`; session 2026-07-21 | 2026-07-21 | +| #007 | P3 | rec | `/tools` vs `/?mode=tools` parallel Tools entry points | `/tools` (standalone `ApplicationsLauncherPage`) has no inbound in-app link; the sidebar Tools item uses `/?mode=tools`. Decide the canonical entry point and wire nav consistently, or drop the standalone `/tools` page + `/applications` redirect. Currently allowlisted in `tests/route-reachability.test.ts`. | `src/app/tools/page.tsx`; `src/app/applications/route.ts` | 2026-07-21 | +| #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | +| #010 | P3 | task | Un-built "Coming soon" controls across forms/favourites | ~10 disabled placeholders (forms refine/reset, favourites sort/add/new-set, move-to-set, remove-favourite). Correctly flagged (`aria-disabled` + "Coming soon"), not defects — wire when the underlying features land. | `forms-search-results-page.tsx`; `favourites-hub.tsx`; `favourites-command-library-page.tsx` | 2026-07-21 | +| #011 | P3 | task | Auth DB-connection allocation is operator-only | Supabase Auth (GoTrue) is capped at ~10 absolute DB connections (Supabase perf advisor). Switch to **percentage-based** allocation in the Supabase **dashboard** before the first compute scale-up — **not settable via SQL/MCP** (operator-owned). Verify via a staging soak + an approval-gated read-only advisor re-check. | `docs/auth-connection-cap-runbook.md`; `docs/process-hardening.md` (Known follow-up debts) | 2026-07-21 | +| #012 | P3 | rec | Slim the lazy cross-mode differentials chunk | `cross-mode-differentials.ts` is dynamically imported (correctly code-split **out** of the initial/dashboard bundle — verified), but it pulls the full ~860 KB differentials snapshot (~125 KB gzip lazy chunk) just to build a tiny `{slug,title,clinicalHinge}` + presentations + aliases catalog. A precomputed lightweight index (generator + drift check, like the `specifiers-content` split / medications `fields=index`) would cut that lazy chunk ~5–10×. Not a bundle leak — an M-effort slim. | `src/lib/cross-mode-differentials.ts`; `src/components/clinical-dashboard/cross-mode-links.tsx:150`; session 2026-07-21 (build:analyze) | 2026-07-21 | +| #013 | P3 | rec | Route-chunk + mockup catalogue JSON weight | `build:analyze`: `/specifiers` ships `specifiers-search-index.json` (~180 KB parsed), `/forms` ships `forms-catalog.json` (~132 KB), `/formulation` ships `formulation-content.json` (~52 KB, client-side local search — needs index/full split or a search endpoint, architectural). All route-scoped (not initial bundle). Also `*-mockups.tsx` (~100 KB across chunks) build though `/mockups` 404s in prod — exclude from the prod artifact. | session 2026-07-21 (build:analyze) | 2026-07-21 | +| #014 | P3 | rec | Realize the `next/image` win on signed previews | `next.config` `images` (AVIF + `*.supabase.co` `remotePatterns` pinned to the project host, from #1024) is currently inert — signed document/image previews still render as raw ``. Route them through `next/image` to actually get AVIF + lazy optimization. | #1024; `src/components/clinical-dashboard/signed-image.tsx`; session 2026-07-21 | 2026-07-21 | +| #016 | P3 | rec | "Big but not easy" structural + motion perf | Deferred larger levers: (a) nonce-CSP forces every product route to `ƒ Dynamic` (zero static generation) — evaluate Partial Prerendering / static shells for the static clinical catalogues (DSM/differentials/therapy/specifiers/formulation); (b) sidebar expand/collapse animates `grid-template-columns` (biggest smoothness cost, motion-gated — needs a transform-overlay rethink); (c) Therapy Compass fetches 692 KB / 2.5 MB JSON client-side (defer until interaction + confirm brotli); (d) settings/setup/admin dialogs static-imported into the home chunk (`next/dynamic` them). | session 2026-07-21 (build route table + design audit) | 2026-07-21 | +| #017 | P3 | task | Field Web-Vitals baseline via live Lighthouse | In-sandbox runtime vitals were blocked (prod server hard-requires Supabase secrets; dev-mode CLS measured excellent at 0.00–0.04, content-first pages 0.000). Run Lighthouse against `psychiatry.tools` for real LCP/INP/CLS to prioritize #012–#016 by measured impact rather than reasoning. | session 2026-07-21 (measurement pass) | 2026-07-21 | +| #018 | P2 | task | Monitoring targeting misses are generation, not retrieval | **Re-diagnosed 2026-07-22 from the run #61 + #57 eval-canary artifacts (no spend).** NOT retrieval-depth: all three cases are grounded with 4-5 citations, and the 36-case retrieval suite is 100 pct doc/content recall in both runs (content_mrr_at_10 0.9344, zero failing cases). Cause is answer generation: lithium-monitoring-range and adhd-medication-monitoring return the `source_backed_review_fallback` boilerplate ("a full written answer could not be completed just now"), which cannot carry a schedule; antipsychotic-metabolic emits degenerate extractive text ("The guidance for metabolic is that compliance, monitoring and evaluation.") and is classified `document_lookup` when the case expects `medication_dose_risk`. Suite-wide, not monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are route=extractive across dose/monitoring/pathway intents - substantially the same phenomenon as #021. **No retrieval canary warranted.** Next: treat jointly with #021 as one generation-quality item; the cheap standalone fix is the antipsychotic query_class drift. | run #61/#57 eval-canary artifacts; session 2026-07-22 | 2026-07-21 | +| #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | +| #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | +| #022 | P2 | task | Source-governance metadata refresh (operator) | **Worklist generated 2026-07-22 ($0, read-only): `docs/source-governance-refresh-worklist-2026-07-22.md`.** Reframed - this is NOT 59 clinical reviews. Of the 124 documents surfacing in canary top results, 59 are review-required, and **38 (64 pct) are the BMJ published-reference tier all sitting at `clinical_validation_status: unverified`** - one attestation-policy decision, not 38 reviews. The remaining 21 are genuine local WA health-service reviews (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1), mostly `document_status: review_due`. Burn-down: top-10 documents clear 44 pct of flagged slots, top-20 clear 66 pct. Next: decide the BMJ attestation policy, then attest local docs by visibility (start `Clozapine Management by GP (NMHS)`, 22 slots at rank 1). | runs #61/#57 Source Governance data; `docs/source-governance-refresh-worklist-2026-07-22.md` | 2026-07-21 | +| #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | +| #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | +| #025 | P3 | issue | Canary Source Governance table reports all zeros | Run #61 `answer-quality.log` prints `## Source Governance` with `Top results | 0`and every rate 0, even though the run's`golden-retrieval.json` `topResults` carry full per-result governance metadata (`document_status`, `clinical_validation_status`, `extraction_quality`). The operator-facing table is therefore not populated and the #022 worklist had to be derived from raw JSON. Fix the table's data wiring so governance is visible from the log itself. | run #61 eval-canary artifacts; session 2026-07-22 | 2026-07-22 | ## Resolved / archive diff --git a/docs/source-governance-refresh-worklist-2026-07-22.md b/docs/source-governance-refresh-worklist-2026-07-22.md new file mode 100644 index 00000000..f70e1d90 --- /dev/null +++ b/docs/source-governance-refresh-worklist-2026-07-22.md @@ -0,0 +1,101 @@ +# Source-governance refresh worklist — 2026-07-22 + +Successor to [`source-review-priority-2026-07-02.md`](source-review-priority-2026-07-02.md), regenerated +from live canary artifacts. Ledger item: **#022**. Produced read-only at **$0** — no provider calls, no +live queries; everything below is derived from the Eval Canary artifacts for runs **#61** and **#57**. + +## What the two governance numbers actually mean + +They are different denominators and are often conflated: + +| Number | Source | Denominator | Meaning | +| -------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------- | ----------------------------------------------------------- | +| **0.8409** | run #61 `answer-quality.log` → Answer Metrics → "Source governance warning rate" | **44 answer-quality cases** | ~37 of 44 cases raised at least one governance warning | +| **0.5976** (404/676) | runs #61 + #57 `golden-retrieval.json` → `topResults` | **676 individual top-result slots** (36 retrieval cases × 2 runs) | 60% of surfaced result slots carry review-required metadata | + +Policy (verbatim from the canary log): _"unknown, unverified, review_due, outdated, unknown extraction, +and poor extraction metadata are treated as review-required; do not silently default them to current or +approved."_ + +> **Reporting gap worth noting:** run #61's own `## Source Governance` table reports `Top results | 0` and +> all-zero rates, even though the underlying `topResults` records carry full governance metadata. The +> operator-facing table in the log is therefore **not** populated — this worklist had to be derived from +> the raw JSON. Worth fixing so the canary log surfaces this directly. + +## The reframing: this is not 59 document reviews + +**59 of the 124 distinct documents** appearing in top results are review-required. But they fall into two +very different classes: + +| Class | Docs | Flag | What it actually needs | +| ------------------------------------- | -------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **BMJ published reference tier** | **38 (64%)** | `clinical_validation_status: unverified` | **One policy decision**, not 38 clinical reviews — decide how third-party BMJ Best Practice content is attested (bulk attestation / a "third-party published" validation state / an ingestion default), then apply it | +| **Local WA health-service documents** | **21** (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1) | mostly `document_status: review_due` | **Genuine periodic review** — real attestation work, but a tractable ~21 documents | + +By flag combination: `val:unverified` 30 · `doc:review_due` 19 · both 8 · `doc:unknown` 1 · +`val:unverified + doc:unknown` 1. + +## Burn-down math + +Ranked by number of review-required top-result slots (both runs combined; 404 flagged slots total): + +| Top N documents | Slots cleared | Share of all flagged slots | +| --------------- | ------------- | -------------------------- | +| 5 | 110 / 404 | **27%** | +| 10 | 176 / 404 | **44%** | +| 15 | 226 / 404 | **56%** | +| 20 | 266 / 404 | **66%** | +| 30 | 332 / 404 | **82%** | + +Because the BMJ tier dominates, resolving the BMJ attestation policy alone would clear the large majority +of these slots in one action. + +## Prioritized worklist (top 25 by surfaced slots) + +`Slots` = review-required top-result appearances across runs #61 + #57. `Best` = highest rank achieved +(rank 1 = most user-visible). `Cum.` = cumulative share of all 404 flagged slots. + +| # | Document | Slots | Best | Publisher | Needs | Cum. | +| --: | ---------------------------------------------------------------------- | ----: | ---: | --------- | ----------------------------------------------------- | ---: | +| 1 | Bipolar disorder in adults.pdf | 32 | 1 | BMJ | validation (unverified) | 8% | +| 2 | Clozapine Management by GP (NMHS).pdf | 22 | 1 | NMHS | document status (review_due) | 13% | +| 3 | Alcohol withdrawal.pdf | 22 | 1 | BMJ | document status (review_due); validation (unverified) | 19% | +| 4 | Opioid use disorder.pdf | 18 | 1 | BMJ | validation (unverified) | 23% | +| 5 | Postnatal depression.pdf | 16 | 1 | BMJ | validation (unverified) | 27% | +| 6 | Anorexia nervosa.pdf | 16 | 1 | BMJ | validation (unverified) | 31% | +| 7 | Generalised anxiety disorder.pdf | 14 | 2 | BMJ | validation (unverified) | 35% | +| 8 | Alcohol and Other Drugs - Addiction, Toxicity and Withdrawal (FSH).pdf | 12 | 1 | FSH | document status (review_due) | 38% | +| 9 | Attention deficit hyperactivity disorder in adults.pdf | 12 | 1 | BMJ | validation (unverified) | 41% | +| 10 | Alcohol use disorder.pdf | 12 | 5 | BMJ | validation (unverified) | 44% | +| 11 | Insomnia.pdf | 10 | 1 | BMJ | validation (unverified) | 46% | +| 12 | Schizophrenia.pdf | 10 | 1 | BMJ | validation (unverified) | 49% | +| 13 | Panic disorders.pdf | 10 | 1 | BMJ | validation (unverified) | 51% | +| 14 | Depression in adults.pdf | 10 | 2 | BMJ | validation (unverified) | 53% | +| 15 | Attention deficit hyperactivity disorder in children.pdf | 10 | 4 | BMJ | validation (unverified) | 56% | +| 16 | Clozapine Coordinator and Clozapine Clinic (NMHS).pdf | 8 | 1 | NMHS | document status (review_due) | 58% | +| 17 | Suicide risk mitigation.pdf | 8 | 1 | BMJ | validation (unverified) | 60% | +| 18 | Post-traumatic stress disorder.pdf | 8 | 1 | BMJ | validation (unverified) | 62% | +| 19 | Obsessive-compulsive disorder.pdf | 8 | 1 | BMJ | validation (unverified) | 64% | +| 20 | Tourette's syndrome.pdf | 8 | 1 | BMJ | document status (review_due); validation (unverified) | 66% | +| 21 | Functional neurological and somatic symptom disorders.pdf | 8 | 3 | BMJ | validation (unverified) | 68% | +| 22 | MHATT Assessment and Treatment Process (AKG).pdf | 8 | 4 | AKG | document status (review_due) | 70% | +| 23 | Social anxiety disorder.pdf | 8 | 5 | BMJ | validation (unverified) | 72% | +| 24 | Personality disorders.pdf | 8 | 6 | BMJ | validation (unverified) | 74% | +| 25 | Depression in children.pdf | 6 | 1 | BMJ | validation (unverified) | 75% | + +## Suggested order of work + +1. **Decide the BMJ attestation policy** (clears ~64% of review-required documents in one action). +2. **Attest the local documents by visibility** — start with `Clozapine Management by GP (NMHS)` (22 slots, + rank 1), then the FSH addiction/withdrawal document, then the remaining NMHS/AKG/CAMHS/KEMH items. +3. **Re-read the warning rate** on the next canary to confirm the burn-down. + +## Scope and limits + +- This is **operator work** (document review-status attestation in the app), not a code defect. No code + change is proposed here. +- The list reflects **only documents that surfaced in golden-case top results**, so it is a + visibility-weighted worklist, not a full corpus audit. The 2026-07-02 predecessor notes the live corpus + is far larger (~2,065 indexed documents at that time). +- Counts combine two runs (#61, #57); a document appearing in both runs counts twice, which is intentional + — it weights persistently-surfaced documents higher. From 5c2bb8d3cff5f0d989866f469df3e6500a9a0b62 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Wed, 22 Jul 2026 08:39:36 +0000 Subject: [PATCH 3/6] =?UTF-8?q?merge(main):=20resolve=20outstanding-issues?= =?UTF-8?q?.md=20conflict=20=E2=80=94=20carry=20#022=20worklist,=20add=20#?= =?UTF-8?q?031,=20next-id=3D032?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/outstanding-issues.md | 48 ++++++++------------------------------ 1 file changed, 10 insertions(+), 38 deletions(-) diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index b0aabaeb..915dc9b4 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -27,34 +27,7 @@ Durable, cross-session memory of everything still outstanding for this repo: ope - Resolving an item moves its row to **Resolved / archive** with the date and a one-line outcome — rows are archived, not deleted, so the history stays auditable. -<<<<<<< HEAD - - -## Open items - -| ID | Pri | Type | Summary | Detail / next action | Source | Added | -| ---- | --- | ----- | --------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | -| #001 | P2 | task | Semantic reranking still gated off | `RAG_SEMANTIC_RERANK_ENABLED=false` from PR #901. Do not enable until the provider-backed 36/36 retrieval-quality gate **and** an ambiguity-focused canary are explicitly approved and recorded. | `docs/process-hardening.md` (Semantic reranking rollout debt); PR #901 | 2026-07-21 | -| #005 | P3 | rec | `finalScore` saturates at clamp ceiling | Base + ~40 stacked boosts routinely exceed 1.0, so strong matches tie at 1.0 and order by an arbitrary `document_id` tiebreak. If ranking is ever revisited, break ties by the **pre-clamp** score rather than raising the `[0,1]` ceiling (downstream gates assume `[0,1]`). Ordering already sorts by the unbounded pre-clamp `rankScore` (`clinical-search.ts:1735,1927,1950-1955`), so the clamp confines only the reported confidence value, not result order. Not a defect on the current golden set; any change here is a protected RAG surface (canary required). | `docs/rag-hybrid-findings-and-todo.md` P1 item 4; `src/lib/clinical-search.ts:1735` | 2026-07-21 | -| #006 | P2 | issue | Globe "Language & region" button had no handler | Fixed in this PR to the disabled "Coming soon" placeholder convention (`aria-disabled` + sr-only note). Wire to a real language/region settings screen when one exists, then drop the placeholder state. | `src/components/clinical-dashboard/master-search-header.tsx:1829`; session 2026-07-21 | 2026-07-21 | -| #007 | P3 | rec | `/tools` vs `/?mode=tools` parallel Tools entry points | `/tools` (standalone `ApplicationsLauncherPage`) has no inbound in-app link; the sidebar Tools item uses `/?mode=tools`. Decide the canonical entry point and wire nav consistently, or drop the standalone `/tools` page + `/applications` redirect. Currently allowlisted in `tests/route-reachability.test.ts`. | `src/app/tools/page.tsx`; `src/app/applications/route.ts` | 2026-07-21 | -| #009 | P3 | rec | Confirm `/api/jobs` is intentionally server/ops-only | No client `fetch()` reaches `/api/jobs` (only tests import it). Confirm it is a deliberate ops/manual surface; if abandoned, remove it. | `src/app/api/jobs/route.ts` | 2026-07-21 | -| #010 | P3 | task | Un-built "Coming soon" controls across forms/favourites | ~10 disabled placeholders (forms refine/reset, favourites sort/add/new-set, move-to-set, remove-favourite). Correctly flagged (`aria-disabled` + "Coming soon"), not defects — wire when the underlying features land. | `forms-search-results-page.tsx`; `favourites-hub.tsx`; `favourites-command-library-page.tsx` | 2026-07-21 | -| #011 | P3 | task | Auth DB-connection allocation is operator-only | Supabase Auth (GoTrue) is capped at ~10 absolute DB connections (Supabase perf advisor). Switch to **percentage-based** allocation in the Supabase **dashboard** before the first compute scale-up — **not settable via SQL/MCP** (operator-owned). Verify via a staging soak + an approval-gated read-only advisor re-check. | `docs/auth-connection-cap-runbook.md`; `docs/process-hardening.md` (Known follow-up debts) | 2026-07-21 | -| #012 | P3 | rec | Slim the lazy cross-mode differentials chunk | `cross-mode-differentials.ts` is dynamically imported (correctly code-split **out** of the initial/dashboard bundle — verified), but it pulls the full ~860 KB differentials snapshot (~125 KB gzip lazy chunk) just to build a tiny `{slug,title,clinicalHinge}` + presentations + aliases catalog. A precomputed lightweight index (generator + drift check, like the `specifiers-content` split / medications `fields=index`) would cut that lazy chunk ~5–10×. Not a bundle leak — an M-effort slim. | `src/lib/cross-mode-differentials.ts`; `src/components/clinical-dashboard/cross-mode-links.tsx:150`; session 2026-07-21 (build:analyze) | 2026-07-21 | -| #013 | P3 | rec | Route-chunk + mockup catalogue JSON weight | `build:analyze`: `/specifiers` ships `specifiers-search-index.json` (~180 KB parsed), `/forms` ships `forms-catalog.json` (~132 KB), `/formulation` ships `formulation-content.json` (~52 KB, client-side local search — needs index/full split or a search endpoint, architectural). All route-scoped (not initial bundle). Also `*-mockups.tsx` (~100 KB across chunks) build though `/mockups` 404s in prod — exclude from the prod artifact. | session 2026-07-21 (build:analyze) | 2026-07-21 | -| #014 | P3 | rec | Realize the `next/image` win on signed previews | `next.config` `images` (AVIF + `*.supabase.co` `remotePatterns` pinned to the project host, from #1024) is currently inert — signed document/image previews still render as raw ``. Route them through `next/image` to actually get AVIF + lazy optimization. | #1024; `src/components/clinical-dashboard/signed-image.tsx`; session 2026-07-21 | 2026-07-21 | -| #016 | P3 | rec | "Big but not easy" structural + motion perf | Deferred larger levers: (a) nonce-CSP forces every product route to `ƒ Dynamic` (zero static generation) — evaluate Partial Prerendering / static shells for the static clinical catalogues (DSM/differentials/therapy/specifiers/formulation); (b) sidebar expand/collapse animates `grid-template-columns` (biggest smoothness cost, motion-gated — needs a transform-overlay rethink); (c) Therapy Compass fetches 692 KB / 2.5 MB JSON client-side (defer until interaction + confirm brotli); (d) settings/setup/admin dialogs static-imported into the home chunk (`next/dynamic` them). | session 2026-07-21 (build route table + design audit) | 2026-07-21 | -| #017 | P3 | task | Field Web-Vitals baseline via live Lighthouse | In-sandbox runtime vitals were blocked (prod server hard-requires Supabase secrets; dev-mode CLS measured excellent at 0.00–0.04, content-first pages 0.000). Run Lighthouse against `psychiatry.tools` for real LCP/INP/CLS to prioritize #012–#016 by measured impact rather than reasoning. | session 2026-07-21 (measurement pass) | 2026-07-21 | -| #018 | P2 | task | Monitoring targeting misses are generation, not retrieval | **Re-diagnosed 2026-07-22 from the run #61 + #57 eval-canary artifacts (no spend).** NOT retrieval-depth: all three cases are grounded with 4-5 citations, and the 36-case retrieval suite is 100 pct doc/content recall in both runs (content_mrr_at_10 0.9344, zero failing cases). Cause is answer generation: lithium-monitoring-range and adhd-medication-monitoring return the `source_backed_review_fallback` boilerplate ("a full written answer could not be completed just now"), which cannot carry a schedule; antipsychotic-metabolic emits degenerate extractive text ("The guidance for metabolic is that compliance, monitoring and evaluation.") and is classified `document_lookup` when the case expects `medication_dose_risk`. Suite-wide, not monitoring-specific: 15/30 cases hit the fallback and all 7 targeting misses are route=extractive across dose/monitoring/pathway intents - substantially the same phenomenon as #021. **No retrieval canary warranted.** Next: treat jointly with #021 as one generation-quality item; the cheap standalone fix is the antipsychotic query_class drift. | run #61/#57 eval-canary artifacts; session 2026-07-22 | 2026-07-21 | -| #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | -| #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | -| #022 | P2 | task | Source-governance metadata refresh (operator) | **Worklist generated 2026-07-22 ($0, read-only): `docs/source-governance-refresh-worklist-2026-07-22.md`.** Reframed - this is NOT 59 clinical reviews. Of the 124 documents surfacing in canary top results, 59 are review-required, and **38 (64 pct) are the BMJ published-reference tier all sitting at `clinical_validation_status: unverified`** - one attestation-policy decision, not 38 reviews. The remaining 21 are genuine local WA health-service reviews (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1), mostly `document_status: review_due`. Burn-down: top-10 documents clear 44 pct of flagged slots, top-20 clear 66 pct. Next: decide the BMJ attestation policy, then attest local docs by visibility (start `Clozapine Management by GP (NMHS)`, 22 slots at rank 1). | runs #61/#57 Source Governance data; `docs/source-governance-refresh-worklist-2026-07-22.md` | 2026-07-21 | -| #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | -| #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | -| #025 | P3 | issue | Canary Source Governance table reports all zeros | Run #61 `answer-quality.log` prints `## Source Governance` with `Top results | 0`and every rate 0, even though the run's`golden-retrieval.json` `topResults` carry full per-result governance metadata (`document_status`, `clinical_validation_status`, `extraction_quality`). The operator-facing table is therefore not populated and the #022 worklist had to be derived from raw JSON. Fix the table's data wiring so governance is visible from the log itself. | run #61 eval-canary artifacts; session 2026-07-22 | 2026-07-22 | -======= - + ## Open items @@ -76,7 +49,7 @@ Durable, cross-session memory of everything still outstanding for this repo: ope | #019 | P2 | task | Admission doc dropped by answer-stage re-rank, not retrieval | DIAGNOSED 2026-07-22 from run #61 artifacts: retrieval coverage is PERFECT — golden documentRecallAt5=1, reciprocalRankAt10=1, missingDocumentSubstrings=[], with "Admission of Community Patients (AKG).pdf" at raw rank 4-5 (matches the strict alias AdmissionCommunityPts). The answer stage then DROPS it from answer.sources top-5 and promotes "Falls Prevention and Management (AKG).pdf" (raw rank 8, off-topic) in its place. So the defect is post-retrieval evidence balancing (cross-document-synthesis.ts balanceCrossDocumentResults / rankAnswerEvidence), NOT comparison-class retrieval coverage. Separately: nothing guarantees BOTH sides of a comparison are represented — comparison_multi_document_gate (rag.ts:2043) is satisfied by any 2 distinct docs. Fix = protected surface (canary pair + ~$2-4 approval). | run #61 artifacts; session 2026-07-22 | 2026-07-21 | | #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | | #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | -| #022 | P2 | task | Source-governance metadata refresh (operator) | Governance warning rate ~0.84 across canaries: stale/review-required/unknown source metadata on most top results. Operator work (document review-status attestation in the app), not a code defect. Next action: generate the prioritized refresh worklist ($0 read-only) and schedule a metadata pass. | runs #57-#61 Source Governance tables; docs/observability-slos.md | 2026-07-21 | +| #022 | P2 | task | Source-governance metadata refresh (operator) | **Worklist generated 2026-07-22 ($0, read-only): `docs/source-governance-refresh-worklist-2026-07-22.md`.** Reframed - this is NOT 59 clinical reviews. Of the 124 documents surfacing in canary top results, 59 are review-required, and **38 (64 pct) are the BMJ published-reference tier all sitting at `clinical_validation_status: unverified`** - one attestation-policy decision, not 38 reviews. The remaining 21 are genuine local WA health-service reviews (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1), mostly `document_status: review_due`. Burn-down: top-10 documents clear 44 pct of flagged slots, top-20 clear 66 pct. Next: decide the BMJ attestation policy, then attest local docs by visibility (start `Clozapine Management by GP (NMHS)`, 22 slots at rank 1). | runs #61/#57 Source Governance data; `docs/source-governance-refresh-worklist-2026-07-22.md` | 2026-07-21 | | #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | | #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | | #025 | P2 | task | Activate the three webhooks (operator secrets) | Merged (#968) + deployed but inert — verified live: `POST /api/webhooks/railway` returns `503 webhook_not_configured`. To turn on: (1) Railway → set `RAILWAY_WEBHOOK_SECRET` + add the `?token=…` webhook URL; (2) the chat URLs `SLACK_WEBHOOK_URL`/`DISCORD_WEBHOOK_URL` must be set in BOTH places — the Railway **app/server env** (the receiver forwards deploy alerts via `postChatNotification`, which reads server env, so repo-secret-only leaves the Railway webhook authenticated but returning `delivered:false`) AND as **GitHub repo secrets** (the CI-failure workflow reads `secrets.*`); (3) `SUPABASE_INGESTION_WEBHOOK_SECRET`. Each fails closed until set, so this is pure ops. See docs/webhooks.md. | session 2026-07-22; PR #968; docs/webhooks.md | 2026-07-22 | @@ -85,17 +58,16 @@ Durable, cross-session memory of everything still outstanding for this repo: ope | #028 | P3 | rec | Runtime error tracking (Sentry or similar) | No error tracking in the repo — production exceptions on `psychiatry.tools`, including how often `RAG_PROVIDER_MODE=auto` silently degrades to source-only, are invisible. Weigh adding `@sentry/nextjs` (dependency + DSN secret + instrumentation) vs cost; alert → chat/issue. Provider-backed; needs explicit sign-off before adding the dependency. | session 2026-07-22 webhook review | 2026-07-22 | | #029 | P2 | issue | 12 of 30 answer-quality cases return the fallback stub | run #61 --dump-answers: 12/30 quality cases emit the source_backed_review_fallback boilerplate with answer_sections: [], all grounded with 4-6 citations. Some still PASS targeting because the stub echoes query keywords (the contraindication/document_lookup matchers need only a keyword), so the targeting metric MASKS the problem for those intents. Superset of #018 — fix in the extractive composer, validate with the provider-backed answer eval. | run #61 dump artifact; session 2026-07-22 | 2026-07-22 | | #030 | P3 | issue | Wide-tier alias lets one doc satisfy both comparison slots | In src/lib/eval-document-matching.ts, "Admission to Discharge for Mental Health Inpatients" appears in BOTH the AdmissionCommunityPts and Discharge alias lists, so a single document can satisfy both expectedFiles slots and make allHit true — a latent false-pass on admission-discharge cases. Not firing today (that doc is not in the failing top-5) but it would mask a real miss. Tighten the tables so one doc cannot fill both sides. | src/lib/eval-document-matching.ts:32-65; session 2026-07-22 | 2026-07-22 | ->>>>>>> origin/main +| #031 | P3 | issue | Canary Source Governance table reports all zeros | Run #61 `answer-quality.log` prints `## Source Governance` with `Top results \| 0` and every rate 0, even though the run's `golden-retrieval.json` `topResults` carry full per-result governance metadata (`document_status`, `clinical_validation_status`, `extraction_quality`). The operator-facing table is therefore not populated and the #022 worklist had to be derived from raw JSON. Fix the table's data wiring so governance is visible from the log itself. | run #61 eval-canary artifacts; session 2026-07-22 | 2026-07-22 | ## Resolved / archive Move resolved rows here with the resolution date and a one-line outcome. Keep them — do not delete. -| ID | Type | Summary | Outcome | Resolved | -| ---- | ---- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -| #003 | task | Staging tenancy release evidence outstanding | Ran GitHub Action and validated isolation | 2026-07-21 | -| #002 | task | Process-ownership fix not yet isolated on `main` | Fixed process isolation using child.pid termination | 2026-07-21 | -| #008 | rec | Dead href builders in `document-flow-routes.ts` | Not dead code (false positive): `documentReaderHref`/`documentEvidenceHref` are live via the mock wrappers in `src/components/document-search-mockups.tsx` + `src/components/master-document-flow-mockups.tsx` (rendered under `src/app/mockups/document-search/`) and covered by `tests/document-flow-routes.test.ts`; removing breaks the build. Only the production non-mock hrefs are unlinked from prod UI — a wiring gap, not dead code. | 2026-07-22 | -| #015 | task | Content-first fallback regression tests | Added `tests/registry-record-loader.dom.test.tsx` (8) + `tests/medication-record-page.dom.test.tsx` (6) covering content-first fallback paint, live swap-in, spinner/skeleton, error + not-found/unauthorized states, and the invariant that no authoritative verification badge shows before live governance reconciles (registry fixture-flag neutralization + medication governance-drop-on-error). | 2026-07-22 | -| #004 | rec | Rescope provider-gated RAG safety ideas | Closed obsolete — rescue source (754-line RAG-safety worktree) unrecoverable/pruned across all refs; answer-quality thresholds + deep-health already shipped on `main` (#585/#587); only cost-cap preflight was genuinely missing and, per session decision, dropped rather than re-filed. | 2026-07-22 | -| #019 | task | Comparison-class coverage: admission doc never in top-5 | **Not reproducible (2026-07-22, $0).** The run #61 **and** #57 eval-canary artifacts both place `Admission of Community Patients (AKG).pdf` at ranks 4-5 with `missingDocumentSubstrings: []`, `documentRecallAt5: 1`, `contentRecallAt5: 1`, and zero failing cases across the 36-case retrieval suite. The claim is false at both endpoints of the stated #57-#61 range, so no retrieval canary was spent. | 2026-07-22 | +| ID | Type | Summary | Outcome | Resolved | +| ---- | ---- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| #003 | task | Staging tenancy release evidence outstanding | Ran GitHub Action and validated isolation | 2026-07-21 | +| #002 | task | Process-ownership fix not yet isolated on `main` | Fixed process isolation using child.pid termination | 2026-07-21 | +| #008 | rec | Dead href builders in `document-flow-routes.ts` | Not dead code (false positive): `documentReaderHref`/`documentEvidenceHref` are live via the mock wrappers in `src/components/document-search-mockups.tsx` + `src/components/master-document-flow-mockups.tsx` (rendered under `src/app/mockups/document-search/`) and covered by `tests/document-flow-routes.test.ts`; removing breaks the build. Only the production non-mock hrefs are unlinked from prod UI — a wiring gap, not dead code. | 2026-07-22 | +| #015 | task | Content-first fallback regression tests | Added `tests/registry-record-loader.dom.test.tsx` (8) + `tests/medication-record-page.dom.test.tsx` (6) covering content-first fallback paint, live swap-in, spinner/skeleton, error + not-found/unauthorized states, and the invariant that no authoritative verification badge shows before live governance reconciles (registry fixture-flag neutralization + medication governance-drop-on-error). | 2026-07-22 | +| #004 | rec | Rescope provider-gated RAG safety ideas | Closed obsolete — rescue source (754-line RAG-safety worktree) unrecoverable/pruned across all refs; answer-quality thresholds + deep-health already shipped on `main` (#585/#587); only cost-cap preflight was genuinely missing and, per session decision, dropped rather than re-filed. | 2026-07-22 | From 715d6489d7a31d237b3ab1e6aa68f054cd11ad53 Mon Sep 17 00:00:00 2001 From: "coderabbitai[bot]" <136622811+coderabbitai[bot]@users.noreply.github.com> Date: Wed, 22 Jul 2026 08:49:32 +0000 Subject: [PATCH 4/6] fix: apply CodeRabbit auto-fixes Fixed 1 file(s) based on 1 unresolved review comment. Co-authored-by: CodeRabbit --- docs/source-governance-refresh-worklist-2026-07-22.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source-governance-refresh-worklist-2026-07-22.md b/docs/source-governance-refresh-worklist-2026-07-22.md index f70e1d90..2e70a639 100644 --- a/docs/source-governance-refresh-worklist-2026-07-22.md +++ b/docs/source-governance-refresh-worklist-2026-07-22.md @@ -29,7 +29,7 @@ very different classes: | Class | Docs | Flag | What it actually needs | | ------------------------------------- | -------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **BMJ published reference tier** | **38 (64%)** | `clinical_validation_status: unverified` | **One policy decision**, not 38 clinical reviews — decide how third-party BMJ Best Practice content is attested (bulk attestation / a "third-party published" validation state / an ingestion default), then apply it | +| **BMJ published reference tier** | **38 (64%)** | `clinical_validation_status: unverified` | **One policy decision**, not 38 clinical reviews — decide how third-party BMJ Best Practice content is attested (bulk attestation / a "third-party published" validation state / an explicit metadata update that preserves the third-party/unverified distinction), then apply it | | **Local WA health-service documents** | **21** (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1) | mostly `document_status: review_due` | **Genuine periodic review** — real attestation work, but a tractable ~21 documents | By flag combination: `val:unverified` 30 · `doc:review_due` 19 · both 8 · `doc:unknown` 1 · @@ -85,7 +85,7 @@ of these slots in one action. ## Suggested order of work -1. **Decide the BMJ attestation policy** (clears ~64% of review-required documents in one action). +1. **Decide the BMJ attestation policy** (clears ~64% of review-required documents in one action) — any chosen option must be an explicit, auditable metadata update that never silently defaults to current/approved and preserves the third-party/unverified distinction; this remains open debt (ledger #022) until the approved policy is implemented, not merely decided. 2. **Attest the local documents by visibility** — start with `Clozapine Management by GP (NMHS)` (22 slots, rank 1), then the FSH addiction/withdrawal document, then the remaining NMHS/AKG/CAMHS/KEMH items. 3. **Re-read the warning rate** on the next canary to confirm the burn-down. From 0ee7f4a7c9a11704421a3d7b90d2f26544673d8f Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 22 Jul 2026 16:52:29 +0800 Subject: [PATCH 5/6] docs(governance): constrain BMJ policy options to auditable metadata updates Clarify that BMJ attestation must preserve third-party/unverified status and never silently default ingestion metadata to current or approved. Co-authored-by: Cursor --- docs/source-governance-refresh-worklist-2026-07-22.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source-governance-refresh-worklist-2026-07-22.md b/docs/source-governance-refresh-worklist-2026-07-22.md index f70e1d90..703714f2 100644 --- a/docs/source-governance-refresh-worklist-2026-07-22.md +++ b/docs/source-governance-refresh-worklist-2026-07-22.md @@ -29,7 +29,7 @@ very different classes: | Class | Docs | Flag | What it actually needs | | ------------------------------------- | -------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **BMJ published reference tier** | **38 (64%)** | `clinical_validation_status: unverified` | **One policy decision**, not 38 clinical reviews — decide how third-party BMJ Best Practice content is attested (bulk attestation / a "third-party published" validation state / an ingestion default), then apply it | +| **BMJ published reference tier** | **38 (64%)** | `clinical_validation_status: unverified` | **One policy decision**, not 38 clinical reviews — decide how third-party BMJ Best Practice content is attested (bulk attestation via an explicit, auditable metadata update / a dedicated "third-party published" validation state). Do **not** silently default ingestion metadata to `current` or `approved`; preserve the third-party/unverified distinction until the approved policy is applied. | | **Local WA health-service documents** | **21** (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1) | mostly `document_status: review_due` | **Genuine periodic review** — real attestation work, but a tractable ~21 documents | By flag combination: `val:unverified` 30 · `doc:review_due` 19 · both 8 · `doc:unknown` 1 · @@ -85,7 +85,7 @@ of these slots in one action. ## Suggested order of work -1. **Decide the BMJ attestation policy** (clears ~64% of review-required documents in one action). +1. **Decide the BMJ attestation policy** (clears ~64% of review-required documents in one action). Any chosen option must use an explicit, auditable metadata update and must preserve the third-party/unverified distinction until applied — never silently default ingestion metadata to `current` or `approved`. 2. **Attest the local documents by visibility** — start with `Clozapine Management by GP (NMHS)` (22 slots, rank 1), then the FSH addiction/withdrawal document, then the remaining NMHS/AKG/CAMHS/KEMH items. 3. **Re-read the warning rate** on the next canary to confirm the burn-down. From 69b0fa5618ca47463d3cdabd042c2fd1bc734223 Mon Sep 17 00:00:00 2001 From: BigSimmo <87357024+BigSimmo@users.noreply.github.com> Date: Wed, 22 Jul 2026 17:06:58 +0800 Subject: [PATCH 6/6] style(docs): prettier-format governance worklist and issues ledger Co-authored-by: Cursor --- docs/outstanding-issues.md | 4 ++-- docs/source-governance-refresh-worklist-2026-07-22.md | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index 915dc9b4..aa24ac52 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -49,7 +49,7 @@ Durable, cross-session memory of everything still outstanding for this repo: ope | #019 | P2 | task | Admission doc dropped by answer-stage re-rank, not retrieval | DIAGNOSED 2026-07-22 from run #61 artifacts: retrieval coverage is PERFECT — golden documentRecallAt5=1, reciprocalRankAt10=1, missingDocumentSubstrings=[], with "Admission of Community Patients (AKG).pdf" at raw rank 4-5 (matches the strict alias AdmissionCommunityPts). The answer stage then DROPS it from answer.sources top-5 and promotes "Falls Prevention and Management (AKG).pdf" (raw rank 8, off-topic) in its place. So the defect is post-retrieval evidence balancing (cross-document-synthesis.ts balanceCrossDocumentResults / rankAnswerEvidence), NOT comparison-class retrieval coverage. Separately: nothing guarantees BOTH sides of a comparison are represented — comparison_multi_document_gate (rag.ts:2043) is satisfied by any 2 distinct docs. Fix = protected surface (canary pair + ~$2-4 approval). | run #61 artifacts; session 2026-07-22 | 2026-07-21 | | #020 | P3 | task | Validate eval:quality cost readout post-fix | The estimator fix landed in PR #1050 (zero-usage cases count $0 with rates configured; provider-attempted cases with missing usage stay n/a). Remaining action: confirm the next canary run prints a real dollar figure in Answer Metrics, then archive this row. | PR #1050; run #60/#61 Answer Metrics tables | 2026-07-21 | | #021 | P3 | rec | E-3d H2 residual: strong/comparison generation discards | approx. 6 generation attempts per full 44-case run still fail the final quality gate and fall to extractive on strong-route comparison/complex shapes (the designed-conservative outcome). PARKED: weakest cost/benefit on the queue — a wave (approx. $2-4 pair + reviewer cycle) to shave seconds off a few hard cases. Revisit only if latency/waste complaints or a cheaper lever appears. | E-3c design record; runs #59-#61 diagnostics | 2026-07-21 | -| #022 | P2 | task | Source-governance metadata refresh (operator) | **Worklist generated 2026-07-22 ($0, read-only): `docs/source-governance-refresh-worklist-2026-07-22.md`.** Reframed - this is NOT 59 clinical reviews. Of the 124 documents surfacing in canary top results, 59 are review-required, and **38 (64 pct) are the BMJ published-reference tier all sitting at `clinical_validation_status: unverified`** - one attestation-policy decision, not 38 reviews. The remaining 21 are genuine local WA health-service reviews (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1), mostly `document_status: review_due`. Burn-down: top-10 documents clear 44 pct of flagged slots, top-20 clear 66 pct. Next: decide the BMJ attestation policy, then attest local docs by visibility (start `Clozapine Management by GP (NMHS)`, 22 slots at rank 1). | runs #61/#57 Source Governance data; `docs/source-governance-refresh-worklist-2026-07-22.md` | 2026-07-21 | +| #022 | P2 | task | Source-governance metadata refresh (operator) | **Worklist generated 2026-07-22 ($0, read-only): `docs/source-governance-refresh-worklist-2026-07-22.md`.** Reframed - this is NOT 59 clinical reviews. Of the 124 documents surfacing in canary top results, 59 are review-required, and **38 (64 pct) are the BMJ published-reference tier all sitting at `clinical_validation_status: unverified`** - one attestation-policy decision, not 38 reviews. The remaining 21 are genuine local WA health-service reviews (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1), mostly `document_status: review_due`. Burn-down: top-10 documents clear 44 pct of flagged slots, top-20 clear 66 pct. Next: decide the BMJ attestation policy, then attest local docs by visibility (start `Clozapine Management by GP (NMHS)`, 22 slots at rank 1). | runs #61/#57 Source Governance data; `docs/source-governance-refresh-worklist-2026-07-22.md` | 2026-07-21 | | #023 | P3 | task | Read Sunday 2026-07-26 scheduled-run artifacts | The 18:00 UTC scheduled runs deliver three free datapoints at once: first full-44 weekly canary (validates the #1044 ANSWER_CASE_LIMIT raise), browser-matrix flake second datapoint (webkit ui-route-coverage now reproduced + root-caused 2026-07-22 → see #024; firefox ui-formulation:91 still awaits a datapoint), and the irrelevant@10 labeling-audit artifact (§3.1 human-decision class). Read all three, then disposition. | sessions 2026-07-20/21; branch-review-ledger convergence notes | 2026-07-21 | | #024 | P3 | issue | WebKit e2e `_rsc`-prefetch access-control-checks errors | verify:release:offline on `main` ce32fe170 (2026-07-22) reproduced #023's webkit clause: **6/6 deterministic** failures in `tests/ui-route-coverage.spec.ts` (Therapy Compass; DSM home/comparison; Specifier comparison/map; Differential stream), each a `pageerror … ?_rsc=… due to access control checks` on Next.js RSC prefetch — Chromium + Firefox clean. Not merge-blocking (required gate `test:e2e:pr` is chromium-only; the full webkit matrix is advisory/release-time). Most likely a Playwright route-interception × WebKit interaction, not a Safari user defect. Next: decide (a) allow/mock the `_rsc` routes for the `webkit` e2e project, or (b) confirm real Safari impact — before trusting the full-matrix webkit gate at release. NB the 2 other webkit fails (`ui-stress:412`, `ui-universal-search:210`) passed on isolated re-run = true flake. | session 2026-07-22 (verify:release:offline, `main` ce32fe170); refines #023 | 2026-07-22 | | #025 | P2 | task | Activate the three webhooks (operator secrets) | Merged (#968) + deployed but inert — verified live: `POST /api/webhooks/railway` returns `503 webhook_not_configured`. To turn on: (1) Railway → set `RAILWAY_WEBHOOK_SECRET` + add the `?token=…` webhook URL; (2) the chat URLs `SLACK_WEBHOOK_URL`/`DISCORD_WEBHOOK_URL` must be set in BOTH places — the Railway **app/server env** (the receiver forwards deploy alerts via `postChatNotification`, which reads server env, so repo-secret-only leaves the Railway webhook authenticated but returning `delivered:false`) AND as **GitHub repo secrets** (the CI-failure workflow reads `secrets.*`); (3) `SUPABASE_INGESTION_WEBHOOK_SECRET`. Each fails closed until set, so this is pure ops. See docs/webhooks.md. | session 2026-07-22; PR #968; docs/webhooks.md | 2026-07-22 | @@ -58,7 +58,7 @@ Durable, cross-session memory of everything still outstanding for this repo: ope | #028 | P3 | rec | Runtime error tracking (Sentry or similar) | No error tracking in the repo — production exceptions on `psychiatry.tools`, including how often `RAG_PROVIDER_MODE=auto` silently degrades to source-only, are invisible. Weigh adding `@sentry/nextjs` (dependency + DSN secret + instrumentation) vs cost; alert → chat/issue. Provider-backed; needs explicit sign-off before adding the dependency. | session 2026-07-22 webhook review | 2026-07-22 | | #029 | P2 | issue | 12 of 30 answer-quality cases return the fallback stub | run #61 --dump-answers: 12/30 quality cases emit the source_backed_review_fallback boilerplate with answer_sections: [], all grounded with 4-6 citations. Some still PASS targeting because the stub echoes query keywords (the contraindication/document_lookup matchers need only a keyword), so the targeting metric MASKS the problem for those intents. Superset of #018 — fix in the extractive composer, validate with the provider-backed answer eval. | run #61 dump artifact; session 2026-07-22 | 2026-07-22 | | #030 | P3 | issue | Wide-tier alias lets one doc satisfy both comparison slots | In src/lib/eval-document-matching.ts, "Admission to Discharge for Mental Health Inpatients" appears in BOTH the AdmissionCommunityPts and Discharge alias lists, so a single document can satisfy both expectedFiles slots and make allHit true — a latent false-pass on admission-discharge cases. Not firing today (that doc is not in the failing top-5) but it would mask a real miss. Tighten the tables so one doc cannot fill both sides. | src/lib/eval-document-matching.ts:32-65; session 2026-07-22 | 2026-07-22 | -| #031 | P3 | issue | Canary Source Governance table reports all zeros | Run #61 `answer-quality.log` prints `## Source Governance` with `Top results \| 0` and every rate 0, even though the run's `golden-retrieval.json` `topResults` carry full per-result governance metadata (`document_status`, `clinical_validation_status`, `extraction_quality`). The operator-facing table is therefore not populated and the #022 worklist had to be derived from raw JSON. Fix the table's data wiring so governance is visible from the log itself. | run #61 eval-canary artifacts; session 2026-07-22 | 2026-07-22 | +| #031 | P3 | issue | Canary Source Governance table reports all zeros | Run #61 `answer-quality.log` prints `## Source Governance` with `Top results \| 0` and every rate 0, even though the run's `golden-retrieval.json` `topResults` carry full per-result governance metadata (`document_status`, `clinical_validation_status`, `extraction_quality`). The operator-facing table is therefore not populated and the #022 worklist had to be derived from raw JSON. Fix the table's data wiring so governance is visible from the log itself. | run #61 eval-canary artifacts; session 2026-07-22 | 2026-07-22 | ## Resolved / archive diff --git a/docs/source-governance-refresh-worklist-2026-07-22.md b/docs/source-governance-refresh-worklist-2026-07-22.md index d10b0487..811c8032 100644 --- a/docs/source-governance-refresh-worklist-2026-07-22.md +++ b/docs/source-governance-refresh-worklist-2026-07-22.md @@ -27,10 +27,10 @@ approved."_ **59 of the 124 distinct documents** appearing in top results are review-required. But they fall into two very different classes: -| Class | Docs | Flag | What it actually needs | -| ------------------------------------- | -------------------------------------------------------------- | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Class | Docs | Flag | What it actually needs | +| ------------------------------------- | -------------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **BMJ published reference tier** | **38 (64%)** | `clinical_validation_status: unverified` | **One policy decision**, not 38 clinical reviews — decide how third-party BMJ Best Practice content is attested (bulk attestation via an explicit, auditable metadata update / a dedicated "third-party published" validation state). Do **not** silently default ingestion metadata to `current` or `approved`; preserve the third-party/unverified distinction until the approved policy is applied. | -| **Local WA health-service documents** | **21** (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1) | mostly `document_status: review_due` | **Genuine periodic review** — real attestation work, but a tractable ~21 documents | +| **Local WA health-service documents** | **21** (FSH 7, NMHS 4, CAMHS 3, AKG 2, KEMH 2, RPBG 2, RKPG 1) | mostly `document_status: review_due` | **Genuine periodic review** — real attestation work, but a tractable ~21 documents | By flag combination: `val:unverified` 30 · `doc:review_due` 19 · both 8 · `doc:unknown` 1 · `val:unverified + doc:unknown` 1.