diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 168be5104..53fb766b0 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -786,8 +786,11 @@ This file is append-only. Never rewrite or delete an existing review record; app | 2026-07-24 | execute-audit-code-remediation (PR #1162) | 3cb7c977 | Conflict fix + Bugbot + local review | Before: CONFLICTING (21 files). After: mergeable. Restored atomic upload RPC; aligned private-access tests (133/133). Bugbot 2 medium left open. | private-access-routes 133/133; no provider-backed checks | | 2026-07-25 | codex/document-clinical-summary-20260725 (PR #1169) | 605a47b551a03774fab41416bf980dfbc9610221 | Open-PR maintenance: malformed persisted profile guard | Before: one actionable thread showed non-array or malformed persisted summary groups could throw during render. After: every priority group is normalized through an array/item guard and malformed values are ignored while valid items still render. | Focused Vitest 7/7 pass; Prettier and diff checks pass; no provider-backed checks run. | | 2026-07-25 | cursor/search-performance-review-4ee9 (PR #1134) | 692834a86e612cc8b311dc6895e007f182f5c5b8 | Open-PR maintenance: superseded docs-link thread and clean main sync | Before: branch was behind current main with one outdated docs-link thread; its product tree already matched main. After: merged current main cleanly and verified the route-group-aware docs-link fix now covers legacy route references. RAG impact: no retrieval behaviour change — history sync and docs tooling verification only. | `node scripts/check-docs-links.mjs` pass (1154 references); clean merge-tree; no live RAG canary or provider-backed check run. | +| 2026-07-24 | PR #1176 / `cursor/pdf-crop-malformed-repro-9b3e` | `5391bf185cd5dffd00a31eb1d282ccfc93277a73` | #076 page-edge table crop geometry fix + fixture regression | APPROVE. No P0-P1. Fix is narrowly scoped to post-find_tables candidate extension from contiguous cell drawings; title/footer inflation avoided by ignoring text during geometry growth; incompleteness warning retained when content continues past the page. Highest residual risk: text-grid tables without cell drawings still will not edge-extend; left/right/top paths are symmetric but fixture-proven only for bottom. Broad PR #1129 retention/padding/storage changes remain out of scope. | Python page-edge + budget 6/6; Vitest pdf-extractor 3 passed / 1 skipped; offline only. | +| 2026-07-24 | PR #1176 / `cursor/pdf-crop-malformed-repro-9b3e` | `28c1e2352a7cabb9c73f52d5084ee69f47895c5e` | #076 page-edge table crop geometry fix (post-merge tip) | APPROVE (supersedes prior #1176 row for 5391bf18 after remote merge). No P0-P1. Same scoped drawing-based edge extension; residual: text-grid tables without drawings, and left/right/top only unit-symmetric. PR #1129 retention/padding still out of scope. | Re-ran Python page-edge + budget 6/6 and Vitest pdf-extractor 3/1 skipped after merge; offline only. | | 2026-07-25 | codex/hydration-fixes (PR #1131) | 555213fcf4dec82c6dbb445630e59e0d5465149a | Open-PR maintenance: persisted-state hydration coverage | Before: the browser guard covered only an empty-storage dashboard load. After: it seeds theme localStorage plus cookie, sidebar state, and document-viewer PDF mode before navigation, while retaining the default case. | Repository Playwright runner built the isolated production app and passed 3/3 Chromium hydration scenarios; Prettier and diff checks pass; no provider-backed checks run. | | 2026-07-24 | open-PR conflict sync (20 PRs) | multi-head | Conflict resolution pass | Before: 8 PRs behind/dirty (#1124 #1131 #1162 #1169 #1174-1177). After: merged origin/main into all; all 20 open PRs MERGEABLE behind=0 (BLOCKED only by CI/reviews). | merge origin/main per branch; no provider-backed checks run | | 2026-07-24 | open-PR conflict sync (22 PRs) | multi-head | Conflict resolution pass | Before: all 22 open PRs behind/dirty vs main (several CONFLICTING/DIRTY). After: merged origin/main into every open head; all pushes OK; merge-tree classified 22/22 clean. | merge origin/main per branch; check:branch-review-ledger on #1172; no provider-backed checks run | | 2026-07-24 | cursor/pr-queue-hygiene-72ec | pending-push | PR queue hygiene | Add pr-branch-sync workflow + sync:pr-branches helper; bump postcss to clear npm audit high; document anti-churn guidance in AGENTS/process-hardening/pr-babysit/run-pr. | check:github-actions PASS; docs:check-scripts/index PASS; vitest sync-open-pr-branches 3/3; npm audit high clean; no provider-backed checks run | | 2026-07-24 | codex/apply-phone-layout-to-all-home-pages (PR #1124) | pending | Babysit: ledger dedupe + merge readiness | Before: Static PR failed on exact duplicate ledger rows after main sync. After: removed duplicate rows; squash auto-merge armed. | check:branch-review-ledger PASS; no provider-backed checks run | +| 2026-07-25 | cursor/pdf-crop-malformed-repro-9b3e (PR #1176) | c31543f5b5862dbe6911079029f28500a6af2a59 | Run PR sweep: CI fix + threads + drift | Before: GitHub reported DIRTY and Static PR checks failed formatting `docs/outstanding-issues.md` and `tests/pdf-extractor.test.ts`; 0 unresolved threads. After: merged current `origin/main` cleanly and formatted both failing files. | Prettier check pass; `git diff --check` pass; no provider-backed checks run. | diff --git a/docs/outstanding-issues.md b/docs/outstanding-issues.md index 5c7002f7b..40b940058 100644 --- a/docs/outstanding-issues.md +++ b/docs/outstanding-issues.md @@ -142,7 +142,6 @@ removed after current-main verification; it is not missing recommended work. | #040 | P3 | rec | Add targeted visual-regression baselines | Keep a small approved baseline set for high-value desktop/mobile surfaces and accessibility modes instead of screenshotting every route. Start with account/settings, document viewer, mode homes and bottom-composer interactions; define an intentional-update workflow before enabling blocking comparisons. | design audit reconciliation; session 2026-07-22 | 2026-07-22 | | #041 | P3 | rec | Extend the existing Factsheets reading model | Do not add a second patient-facing Factsheets mode. Future patient-content work should extend the existing Easy Read/Standard presentation and its accessibility/content contracts. Revisit only with a concrete user need and source-governance plan. | design audit reconciliation; session 2026-07-22 | 2026-07-22 | | #075 | P2 | issue | Search-scope label enumeration can truncate after 1,000 rows | **Outcome:** every owner-visible source label can participate in scoped search without silently dropping rows at the Supabase response cap. **Next:** reproduce on current `main` with more than 1,000 distinct labels, then add bounded deterministic pagination and a focused multi-page regression. **Verify:** search-scope unit/API contracts, offline RAG contracts, `verify:cheap`, and production-readiness. **Stop:** do not replay mixed PR #1132, widen aliases, change ranking, or run a live canary unless the isolated fix demonstrably changes protected retrieval behaviour. | PR #1132 (`src/lib/search-scope.ts`, `tests/search-scope.test.ts`); primary reconciliation 2026-07-24 | 2026-07-24 | -| #076 | P3 | task | Reproduce malformed fallback PDF image/table crops | **Outcome:** a real current-main fixture proves whether fallback images or table crops are malformed before retention/storage behaviour changes. **Next:** capture one minimal authorised fixture, identify the exact extraction stage and expected geometry, and add a red Python/DOM regression. **Success:** the smallest fix preserves valid assets within existing count/byte budgets and signed-image privacy boundaries. **Stop:** do not merge broad PR #1129, arbitrary crop thresholds/padding, storage expansion, or timeout increases without the reproducer and budget evidence. | PR #1129; `worker/python/extract_pdf_assets.py`; `src/lib/image-filtering.ts`; primary reconciliation 2026-07-24 | 2026-07-24 | | #077 | P2 | issue | Concurrent tasks can re-dirty the canonical primary checkout | **Outcome:** `C:\Dev\Apps\Database` remains a clean synchronization target while write work happens in task-owned worktrees. **Next:** add a cooperative owner/lease check to task-start and lifecycle transitions; before a primary write, branch switch, fast-forward, or cleanup, report owner, dirty state, and Git operation markers and fail closed on an active owner. Include stale-lease recovery. **Success:** a focused concurrency test refuses a second primary writer while read-only commands and separate worktrees continue normally. **Stop:** do not add an OS-wide lock, kill processes, discard existing dirty files, or serialize independent feature worktrees. | primary re-dirtied by another active task immediately after reconciliation proof; session 2026-07-24 | 2026-07-24 | | #078 | P3 | task | Generate a deterministic reconciliation evidence pack | **Outcome:** one report-only command produces the final local evidence now assembled manually. **Next:** extend the reconciliation lifecycle with an explicit output path and include the frozen base/HEAD, per-worktree dispositions, operation markers resolved through Git, archive refs, bundle path/size/SHA-256/verify result, retained worktree count, and local/base tree equality. Accept remote PR state only as explicit approved input. **Success:** fixture tests prove deterministic output and redaction; an interrupted run leaves no false completion record. **Stop:** never fetch, call GitHub/providers, inspect secret values, mutate refs, or delete work implicitly. | `scripts/reconciliation-preflight.mjs`; `docs/reconciliation-playbook.md`; session 2026-07-24 | 2026-07-24 | | #079 | P3 | task | Disposition retained worktrees in bounded cleanup batches | **Outcome:** the retained reconciliation tail is gradually classified without another disruptive all-worktree sweep. **Next:** process no more than ten worktrees per explicitly scheduled pass using current owner/process metadata, open-PR state, exact review-ledger coverage, ancestry, and cherry-pick-aware content proof. **Success:** remove only clean, inactive, bundled worktrees whose content is merged or explicitly rejected; record every disposition and retain recovery evidence. **Stop:** preserve dirty, active, secret-bearing, post-freeze, paused, or ambiguous work and never use reset, force deletion, broad clean, or process killing. | final reconciliation inventory retained 104 independent worktrees; session 2026-07-24 | 2026-07-24 | @@ -151,35 +150,36 @@ removed after current-main verification; it is not missing recommended work. Move resolved rows here with the resolution date and a one-line outcome. Keep them — do not delete. -| ID | Type | Summary | Outcome | Resolved | -| ---- | ----- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | -| #070 | issue | Presentation mobile tabs misroute Overview/Map/Related | Fixed in PR #1135: Overview/Map/Related deep-link to diagnosis `?tab=` sections; Compare stays on the presentation page. Regression in `tests/mobile-interaction-regressions.test.ts`. (Provisional PR-branch IDs `#068`–`#072` were renumbered after `main` claimed `#068`/`#069`.) | 2026-07-24 | -| #071 | issue | Evidence/Clinical Notes Add fakes success without persistence | Fixed in PR #1135: sticky Add controls use the focusable coming-soon placeholder pattern instead of optimistic `setAdded(true)`. | 2026-07-24 | -| #072 | issue | Tools hub exposes false Sort/More affordances | Fixed in PR #1135: Sort is a status label, More filter targets coordination/saved without a fake menu chevron, and the favourites shortcut is labelled Saved/Favourites. | 2026-07-24 | -| #073 | issue | Presentation compare dock CTA is a self-link no-op | Fixed in PR #1135: dock shows non-link "Comparing (N)" status while already comparing. | 2026-07-24 | -| #074 | issue | Mode-action popup hard-reloads internal clinical routes | Fixed in PR #1135: `master-search-header` uses `router.push` for DSM/Specifiers/Formulation actions and mode href fallback. | 2026-07-24 | -| #068 | task | Regenerate full drift-manifest snapshot after schema hygiene | Full Docker `npm run drift:manifest` replay succeeded on a Docker-capable host; `supabase/drift-manifest.json` now carries live `def_hash` values for the plpgsql table-facts body (offline generator_note removed). | 2026-07-24 | -| #052 | issue | Reindex can overlap a fresh agent-enrichment pass | PR #1143 retained the friendly full/retry preflight and closed its check-then-enqueue race with an owner-scoped transactional RPC. Reindex enqueue and the agent claim path serialize on the document row; disposable PostgreSQL proved both interleavings, and exact-head migration replay/unit/build/Chromium/policy/security checks passed. | 2026-07-24 | -| #062 | issue | Upload crash can strand a queued document without a job | Aged owner-scoped `queued`-without-open-job rows are detected by `reindex:health`; the six-hour autopilot raises a durable alert, and guarded recovery uses PR #1143's transactional RPC so enqueue is owner-scoped, idempotent and atomic. `recover:ingestion --include-stranded-queued` remains dry-run/confirmation-first; scheduled production mutation is not enabled. | 2026-07-24 | -| #060 | issue | Safety Plan Generator contradicted the privacy contract | PR #1119 removed patient identifier entry, leaves the post-export name line blank, and aligned tool, privacy and PIA copy. DOM/privacy tests and Chromium copy/print/network coverage prove working content remains in React memory with no fetch/XHR; hosted Production UI, build, unit, policy, safety, static-analysis and secret checks passed. Support-contact details remain classified as sensitive local-only working content. | 2026-07-24 | -| #061 | issue | Missing answer relevance metadata was treated as source-backed | PR #1125 now requires explicit source-backed relevance for trusted/grounded presentation and prevents visual tables, clinical-note sections and quotes, and comparison metadata from bypassing the render model. Three actionable P2 review paths were fixed; focused policy/DOM tests, offline RAG, production-readiness, build, unit, static, security, and Production UI gates passed. No retrieval, ranking, generation, provider, or data behavior changed. | 2026-07-24 | -| #034 | issue | Answer cache can serve stale governance metadata | Current-source verification found direct route coverage already asserts RAG-cache invalidation on document PATCH, source review, label, bulk, and reindex mutation paths. The residual test recommendation is already met; changing the protected cache key is unnecessary. | 2026-07-24 | -| #014 | rec | Realize the `next/image` win on signed previews | Superseded: `SignedImage` uses `next/image` for layout and sizing but deliberately sets `unoptimized`, preventing bearer signed URLs from entering the unauthenticated optimizer cache where cached content could outlive the token. No optimization task remains unless private-image delivery changes. | 2026-07-24 | -| #026 | task | Wire the Supabase document-change trigger | PR #1100 merged after disposable PostgreSQL replay and hosted migration replay. Production migration history and read-only catalog proof confirm the enabled metadata trigger, security-definer function, pinned search path and denied anonymous/authenticated execution; `npm run check:drift` reports no unexpected live drift. Delivery remains intentionally inert until the operator inputs tracked in #025 are configured. | 2026-07-24 | -| #031 | issue | Populate canary Source Governance table | The answer-quality step now consumes the preceding `golden-retrieval.json` only for source-governance reporting. Offline replay of run `30018289898` populated 338 top results, including 202 review-required entries, while retaining zero retrieval cases and no additional threshold failures. Retrieval and ranking behavior are unchanged. | 2026-07-24 | -| #020 | task | Validate eval:quality cost readout post-fix | Confirmed on merged-main canary run `30018289898`: Answer Metrics reported 9 nonzero-cost cases and an estimated answer cost of `$0.234736`; the structured report retained the same value. The PR #1050 estimator fix is operationally proven. | 2026-07-23 | -| #003 | task | Staging tenancy release evidence outstanding | Ran GitHub Action and validated isolation | 2026-07-21 | -| #002 | task | Process-ownership fix not yet isolated on `main` | Fixed process isolation using child.pid termination | 2026-07-21 | -| #008 | rec | Dead href builders in `document-flow-routes.ts` | Not dead code (false positive): `documentReaderHref`/`documentEvidenceHref` are live via the mock wrappers in `src/components/document-search-mockups.tsx` + `src/components/master-document-flow-mockups.tsx` (rendered under `src/app/mockups/document-search/`) and covered by `tests/document-flow-routes.test.ts`; removing breaks the build. Only the production non-mock hrefs are unlinked from prod UI — a wiring gap, not dead code. | 2026-07-22 | -| #015 | task | Content-first fallback regression tests | Added `tests/registry-record-loader.dom.test.tsx` (8) + `tests/medication-record-page.dom.test.tsx` (6) covering content-first fallback paint, live swap-in, spinner/skeleton, error + not-found/unauthorized states, and the invariant that no authoritative verification badge shows before live governance reconciles (registry fixture-flag neutralization + medication governance-drop-on-error). | 2026-07-22 | -| #004 | rec | Rescope provider-gated RAG safety ideas | Closed obsolete — rescue source (754-line RAG-safety worktree) unrecoverable/pruned across all refs; answer-quality thresholds + deep-health already shipped on `main` (#585/#587); only cost-cap preflight was genuinely missing and, per session decision, dropped rather than re-filed. | 2026-07-22 | -| #006 | issue | Globe "Language & region" button had no handler | Resolved on main with the repository's disabled "Coming soon" placeholder convention and button-wiring coverage. Future language/region work remains a feature request, not an inert-control defect. | 2026-07-22 | -| #042 | issue | Invalid optional credentials fell into anonymous access | PRs #1078/#1079 introduced `absent \| valid \| invalid`, return 401 for presented invalid credentials, preserve authoritative header precedence and prefer the current-project session cookie. The archived anonymous-upload metadata patch was rejected as stale because uploads are already administrator-only before duplicate lookup. | 2026-07-22 | -| #043 | issue | Readiness could report healthy or throw on Supabase errors | PR #1080 now fails readiness closed for returned and thrown dependency failures, preserves recognized actionable messages, and prevents raw dependency-error disclosure. | 2026-07-22 | -| #044 | issue | Publication approval was not bound to immutable reviewed state | PR #1081 added a canonical reviewed-state digest, row locks, active-job rejection and a new forward migration with replay/schema/type/drift evidence. | 2026-07-22 | -| #045 | issue | Bulk reindex discarded partial-success results | PR #1084 reserves preflight conflicts for non-2xx responses; completed mixed batches return per-item success/failure/missing results, and the UI refreshes successful work. | 2026-07-22 | -| #046 | issue | DOCX extraction lacked explicit resource budgets | PR #1085 added pre-inflate declared-size checks and post-read fail-safes for artifact count, per-artifact bytes, aggregate media, Word XML and extracted UTF-8 text. | 2026-07-22 | -| #047 | issue | XLSX extraction could construct unbounded results | PR #1086 bounds worksheets, non-empty rows, rendered cells and UTF-8 output while preserving sparse-column rendering. | 2026-07-22 | -| #048 | issue | Account copy overstated sync/privacy and enabled unavailable SSO | PR #1087 now maps copy to actual favourites/preferences persistence, identifies browser-session recents, removes the contradictory "never shared" claim and clearly disables unavailable providers using the accessible placeholder contract. | 2026-07-22 | -| #049 | issue | Process diagnostic exposed a Cursor worker API key | The exact worker was stopped, the key was revoked server-side, both local encrypted worker-secret records were removed, and authorized repository/backup scans found no plaintext copy. Follow-up guardrails now prevent repository process inventory from serializing command lines and redact heavyweight-lock command text before persistence or errors. | 2026-07-23 | -| #050 | issue | Next.js 16.2.10 remained in a high-severity security range | Upgraded `next` and `@next/env` to 16.2.11, regenerated the npm lockfile, confirmed the production dependency audit is clean, and passed focused framework checks, `verify:cheap`, and the full Chromium UI gate. | 2026-07-23 | +| ID | Type | Summary | Outcome | Resolved | +| ---- | ----- | ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | +| #076 | task | Reproduce malformed fallback PDF image/table crops | Reproduced truncated page-edge `table_crop`s on current-main with `worker/python/fixtures/malformed-table-crop-page-edge.pdf`. Root cause: `pymupdf_find_tables` stops at the last fully detected row; fix extends the candidate from contiguous cell drawings, recovers the on-page score-5 remnant, and emits `table_crop_edge_incomplete` / `crop_completeness=0.9` when content continues past the page. PR #1176. Broad PR #1129 retention/padding changes not merged. | 2026-07-24 | +| #070 | issue | Presentation mobile tabs misroute Overview/Map/Related | Fixed in PR #1135: Overview/Map/Related deep-link to diagnosis `?tab=` sections; Compare stays on the presentation page. Regression in `tests/mobile-interaction-regressions.test.ts`. (Provisional PR-branch IDs `#068`–`#072` were renumbered after `main` claimed `#068`/`#069`.) | 2026-07-24 | +| #071 | issue | Evidence/Clinical Notes Add fakes success without persistence | Fixed in PR #1135: sticky Add controls use the focusable coming-soon placeholder pattern instead of optimistic `setAdded(true)`. | 2026-07-24 | +| #072 | issue | Tools hub exposes false Sort/More affordances | Fixed in PR #1135: Sort is a status label, More filter targets coordination/saved without a fake menu chevron, and the favourites shortcut is labelled Saved/Favourites. | 2026-07-24 | +| #073 | issue | Presentation compare dock CTA is a self-link no-op | Fixed in PR #1135: dock shows non-link "Comparing (N)" status while already comparing. | 2026-07-24 | +| #074 | issue | Mode-action popup hard-reloads internal clinical routes | Fixed in PR #1135: `master-search-header` uses `router.push` for DSM/Specifiers/Formulation actions and mode href fallback. | 2026-07-24 | +| #068 | task | Regenerate full drift-manifest snapshot after schema hygiene | Full Docker `npm run drift:manifest` replay succeeded on a Docker-capable host; `supabase/drift-manifest.json` now carries live `def_hash` values for the plpgsql table-facts body (offline generator_note removed). | 2026-07-24 | +| #052 | issue | Reindex can overlap a fresh agent-enrichment pass | PR #1143 retained the friendly full/retry preflight and closed its check-then-enqueue race with an owner-scoped transactional RPC. Reindex enqueue and the agent claim path serialize on the document row; disposable PostgreSQL proved both interleavings, and exact-head migration replay/unit/build/Chromium/policy/security checks passed. | 2026-07-24 | +| #062 | issue | Upload crash can strand a queued document without a job | Aged owner-scoped `queued`-without-open-job rows are detected by `reindex:health`; the six-hour autopilot raises a durable alert, and guarded recovery uses PR #1143's transactional RPC so enqueue is owner-scoped, idempotent and atomic. `recover:ingestion --include-stranded-queued` remains dry-run/confirmation-first; scheduled production mutation is not enabled. | 2026-07-24 | +| #060 | issue | Safety Plan Generator contradicted the privacy contract | PR #1119 removed patient identifier entry, leaves the post-export name line blank, and aligned tool, privacy and PIA copy. DOM/privacy tests and Chromium copy/print/network coverage prove working content remains in React memory with no fetch/XHR; hosted Production UI, build, unit, policy, safety, static-analysis and secret checks passed. Support-contact details remain classified as sensitive local-only working content. | 2026-07-24 | +| #061 | issue | Missing answer relevance metadata was treated as source-backed | PR #1125 now requires explicit source-backed relevance for trusted/grounded presentation and prevents visual tables, clinical-note sections and quotes, and comparison metadata from bypassing the render model. Three actionable P2 review paths were fixed; focused policy/DOM tests, offline RAG, production-readiness, build, unit, static, security, and Production UI gates passed. No retrieval, ranking, generation, provider, or data behavior changed. | 2026-07-24 | +| #034 | issue | Answer cache can serve stale governance metadata | Current-source verification found direct route coverage already asserts RAG-cache invalidation on document PATCH, source review, label, bulk, and reindex mutation paths. The residual test recommendation is already met; changing the protected cache key is unnecessary. | 2026-07-24 | +| #014 | rec | Realize the `next/image` win on signed previews | Superseded: `SignedImage` uses `next/image` for layout and sizing but deliberately sets `unoptimized`, preventing bearer signed URLs from entering the unauthenticated optimizer cache where cached content could outlive the token. No optimization task remains unless private-image delivery changes. | 2026-07-24 | +| #026 | task | Wire the Supabase document-change trigger | PR #1100 merged after disposable PostgreSQL replay and hosted migration replay. Production migration history and read-only catalog proof confirm the enabled metadata trigger, security-definer function, pinned search path and denied anonymous/authenticated execution; `npm run check:drift` reports no unexpected live drift. Delivery remains intentionally inert until the operator inputs tracked in #025 are configured. | 2026-07-24 | +| #031 | issue | Populate canary Source Governance table | The answer-quality step now consumes the preceding `golden-retrieval.json` only for source-governance reporting. Offline replay of run `30018289898` populated 338 top results, including 202 review-required entries, while retaining zero retrieval cases and no additional threshold failures. Retrieval and ranking behavior are unchanged. | 2026-07-24 | +| #020 | task | Validate eval:quality cost readout post-fix | Confirmed on merged-main canary run `30018289898`: Answer Metrics reported 9 nonzero-cost cases and an estimated answer cost of `$0.234736`; the structured report retained the same value. The PR #1050 estimator fix is operationally proven. | 2026-07-23 | +| #003 | task | Staging tenancy release evidence outstanding | Ran GitHub Action and validated isolation | 2026-07-21 | +| #002 | task | Process-ownership fix not yet isolated on `main` | Fixed process isolation using child.pid termination | 2026-07-21 | +| #008 | rec | Dead href builders in `document-flow-routes.ts` | Not dead code (false positive): `documentReaderHref`/`documentEvidenceHref` are live via the mock wrappers in `src/components/document-search-mockups.tsx` + `src/components/master-document-flow-mockups.tsx` (rendered under `src/app/mockups/document-search/`) and covered by `tests/document-flow-routes.test.ts`; removing breaks the build. Only the production non-mock hrefs are unlinked from prod UI — a wiring gap, not dead code. | 2026-07-22 | +| #015 | task | Content-first fallback regression tests | Added `tests/registry-record-loader.dom.test.tsx` (8) + `tests/medication-record-page.dom.test.tsx` (6) covering content-first fallback paint, live swap-in, spinner/skeleton, error + not-found/unauthorized states, and the invariant that no authoritative verification badge shows before live governance reconciles (registry fixture-flag neutralization + medication governance-drop-on-error). | 2026-07-22 | +| #004 | rec | Rescope provider-gated RAG safety ideas | Closed obsolete — rescue source (754-line RAG-safety worktree) unrecoverable/pruned across all refs; answer-quality thresholds + deep-health already shipped on `main` (#585/#587); only cost-cap preflight was genuinely missing and, per session decision, dropped rather than re-filed. | 2026-07-22 | +| #006 | issue | Globe "Language & region" button had no handler | Resolved on main with the repository's disabled "Coming soon" placeholder convention and button-wiring coverage. Future language/region work remains a feature request, not an inert-control defect. | 2026-07-22 | +| #042 | issue | Invalid optional credentials fell into anonymous access | PRs #1078/#1079 introduced `absent \| valid \| invalid`, return 401 for presented invalid credentials, preserve authoritative header precedence and prefer the current-project session cookie. The archived anonymous-upload metadata patch was rejected as stale because uploads are already administrator-only before duplicate lookup. | 2026-07-22 | +| #043 | issue | Readiness could report healthy or throw on Supabase errors | PR #1080 now fails readiness closed for returned and thrown dependency failures, preserves recognized actionable messages, and prevents raw dependency-error disclosure. | 2026-07-22 | +| #044 | issue | Publication approval was not bound to immutable reviewed state | PR #1081 added a canonical reviewed-state digest, row locks, active-job rejection and a new forward migration with replay/schema/type/drift evidence. | 2026-07-22 | +| #045 | issue | Bulk reindex discarded partial-success results | PR #1084 reserves preflight conflicts for non-2xx responses; completed mixed batches return per-item success/failure/missing results, and the UI refreshes successful work. | 2026-07-22 | +| #046 | issue | DOCX extraction lacked explicit resource budgets | PR #1085 added pre-inflate declared-size checks and post-read fail-safes for artifact count, per-artifact bytes, aggregate media, Word XML and extracted UTF-8 text. | 2026-07-22 | +| #047 | issue | XLSX extraction could construct unbounded results | PR #1086 bounds worksheets, non-empty rows, rendered cells and UTF-8 output while preserving sparse-column rendering. | 2026-07-22 | +| #048 | issue | Account copy overstated sync/privacy and enabled unavailable SSO | PR #1087 now maps copy to actual favourites/preferences persistence, identifies browser-session recents, removes the contradictory "never shared" claim and clearly disables unavailable providers using the accessible placeholder contract. | 2026-07-22 | +| #049 | issue | Process diagnostic exposed a Cursor worker API key | The exact worker was stopped, the key was revoked server-side, both local encrypted worker-secret records were removed, and authorized repository/backup scans found no plaintext copy. Follow-up guardrails now prevent repository process inventory from serializing command lines and redact heavyweight-lock command text before persistence or errors. | 2026-07-23 | +| #050 | issue | Next.js 16.2.10 remained in a high-severity security range | Upgraded `next` and `@next/env` to 16.2.11, regenerated the npm lockfile, confirmed the production dependency audit is clean, and passed focused framework checks, `verify:cheap`, and the full Chromium UI gate. | 2026-07-23 | diff --git a/tests/pdf-extractor.test.ts b/tests/pdf-extractor.test.ts index e5c964a70..4b39d7d99 100644 --- a/tests/pdf-extractor.test.ts +++ b/tests/pdf-extractor.test.ts @@ -154,4 +154,49 @@ describe.runIf(hasPyMuPDF)("Python PDF table extraction", () => { expect(tableCrop?.metadata?.table_role).toBe("admin"); expect(tableCrop?.metadata?.accessible_table_markdown).toContain("Published date"); }); + + // #076: page-edge table crops must keep the on-page remnant of a straddling + // final row after find_tables truncates the candidate bbox. + it("includes on-page remnant geometry for a table row that straddles the page bottom (#076)", async () => { + const root = await mkdtemp(path.join(tmpdir(), "clinical-kb-page-edge-crop-")); + const imageDir = path.join(root, "images"); + const jsonPath = path.join(root, "extract.json"); + await mkdir(imageDir, { recursive: true }); + + const fixturePath = path.join(process.cwd(), "worker", "python", "fixtures", "malformed-table-crop-page-edge.pdf"); + const result = spawnSync( + pythonBin, + [path.join(process.cwd(), "worker", "python", "extract_pdf_assets.py"), fixturePath, imageDir, jsonPath], + { cwd: process.cwd(), encoding: "utf8" }, + ); + + expect(result.status).toBe(0); + const payload = JSON.parse(await readFile(jsonPath, "utf8")) as { + images: Array<{ + sourceKind?: string; + bbox?: number[]; + metadata?: Record; + }>; + warnings?: string[]; + }; + const tableCrop = payload.images.find((image) => image.sourceKind === "table_crop"); + expect(tableCrop).toBeTruthy(); + + const pageHeight = Number(tableCrop?.metadata?.page_height); + const clip = (tableCrop?.bbox ?? tableCrop?.metadata?.clip_bbox) as number[] | undefined; + expect(clip).toEqual(expect.any(Array)); + // Stage: pymupdf_find_tables → extend_table_rect_for_edge_content → save_page_crop. + // Expected geometry: clip bottom reaches the page edge so the on-page remnant is retained. + expect(Number(clip?.[3])).toBeGreaterThanOrEqual(pageHeight - 1); + + const rows = (tableCrop?.metadata?.table_rows as string[][] | undefined) ?? []; + const hasScoreRow = rows.some((row) => String(row?.[0] ?? "").trim() === "5"); + const cropCompleteness = Number(tableCrop?.metadata?.crop_completeness); + const incomplete = + Number.isFinite(cropCompleteness) && cropCompleteness < 0.99 + ? true + : (payload.warnings ?? []).some((warning) => warning.includes("table_crop_edge_incomplete")); + expect(hasScoreRow || incomplete).toBe(true); + expect(tableCrop?.metadata?.edge_content_extended).toBe(true); + }); }); diff --git a/worker/python/extract_pdf_assets.py b/worker/python/extract_pdf_assets.py index f504de76e..cf6198a75 100644 --- a/worker/python/extract_pdf_assets.py +++ b/worker/python/extract_pdf_assets.py @@ -293,6 +293,150 @@ def expanded_rect(rect, page_rect, x_padding=4, y_padding=4): ) +# Max gap between a find_tables bbox edge and continuing cell drawings that +# still belong to the same table. Keeps footers/unrelated blocks from merging in. +TABLE_EDGE_CONTENT_GAP = 24.0 + + +def _rects_overlap_horizontally(a, b, tolerance=2.0): + return not (a.x1 < b.x0 - tolerance or a.x0 > b.x1 + tolerance) + + +def _rects_overlap_vertically(a, b, tolerance=2.0): + return not (a.y1 < b.y0 - tolerance or a.y0 > b.y1 + tolerance) + + +def page_drawing_rects(page): + rects = [] + for drawing in page.get_drawings() or []: + rect = drawing.get("rect") + if not rect: + continue + rect = fitz.Rect(rect) + # Ignore hairline page rules / tiny marks. + if rect.width < 8 or rect.height < 8: + continue + rects.append(rect) + return rects + + +def extend_table_rect_for_edge_content(page, rect, max_gap=TABLE_EDGE_CONTENT_GAP): + """Grow a find_tables bbox to keep on-page remnants of straddling table rows/cols. + + `page.find_tables()` often stops at the last fully detected row when the final + row straddles the page edge. Cell drawings for that remnant still exist on the + page; fold them into the crop rectangle (clamped to the page). Text blocks are + intentionally ignored here so titles/footers do not inflate the crop. + """ + page_rect = page.rect + content_rects = page_drawing_rects(page) + x0, y0, x1, y1 = float(rect.x0), float(rect.y0), float(rect.x1), float(rect.y1) + edges = [] + clipped_by_page = False + + def absorb(axis): + nonlocal x0, y0, x1, y1, clipped_by_page + changed = False + current = fitz.Rect(x0, y0, x1, y1) + for content in content_rects: + if axis in ("bottom", "top") and not _rects_overlap_horizontally(current, content): + continue + if axis in ("left", "right") and not _rects_overlap_vertically(current, content): + continue + if axis == "bottom": + if content.y1 <= y1 + 0.5 or content.y0 > y1 + max_gap: + continue + if content.y1 > page_rect.y1 + 0.5: + clipped_by_page = True + next_y1 = min(max(y1, float(content.y1)), float(page_rect.y1)) + if next_y1 > y1 + 0.5: + y1 = next_y1 + changed = True + elif axis == "top": + if content.y0 >= y0 - 0.5 or content.y1 < y0 - max_gap: + continue + if content.y0 < page_rect.y0 - 0.5: + clipped_by_page = True + next_y0 = max(min(y0, float(content.y0)), float(page_rect.y0)) + if next_y0 < y0 - 0.5: + y0 = next_y0 + changed = True + elif axis == "right": + if content.x1 <= x1 + 0.5 or content.x0 > x1 + max_gap: + continue + if content.x1 > page_rect.x1 + 0.5: + clipped_by_page = True + next_x1 = min(max(x1, float(content.x1)), float(page_rect.x1)) + if next_x1 > x1 + 0.5: + x1 = next_x1 + changed = True + elif axis == "left": + if content.x0 >= x0 - 0.5 or content.x1 < x0 - max_gap: + continue + if content.x0 < page_rect.x0 - 0.5: + clipped_by_page = True + next_x0 = max(min(x0, float(content.x0)), float(page_rect.x0)) + if next_x0 < x0 - 0.5: + x0 = next_x0 + changed = True + return changed + + for axis in ("bottom", "top", "right", "left"): + guard = 0 + while absorb(axis) and guard < 8: + if axis not in edges: + edges.append(axis) + guard += 1 + + extended = fitz.Rect(x0, y0, x1, y1) & page_rect + return extended, { + "extended": bool(edges), + "clipped_by_page": clipped_by_page, + "edges": edges, + } + + +def recover_table_rows_from_edge_strip(page, original_rect, extended_rect, existing_rows, column_count=0): + """Recover structured rows from text that continues past a truncated find_tables bbox.""" + existing_keys = { + tuple(clean_cell(cell) for cell in row) + for row in existing_rows or [] + if any(clean_cell(cell) for cell in row) + } + recovered = [] + for block in page.get_text("blocks") or []: + if len(block) < 5: + continue + block_rect = fitz.Rect(block[:4]) + intersection = extended_rect & block_rect + if intersection.is_empty: + continue + # Only consider text that sits in the strip beyond the original detection. + outside_bottom = block_rect.y1 > original_rect.y1 + 0.5 and block_rect.y0 >= original_rect.y1 - 2.0 + outside_top = block_rect.y0 < original_rect.y0 - 0.5 and block_rect.y1 <= original_rect.y0 + 2.0 + outside_right = block_rect.x1 > original_rect.x1 + 0.5 and block_rect.x0 >= original_rect.x1 - 2.0 + outside_left = block_rect.x0 < original_rect.x0 - 0.5 and block_rect.x1 <= original_rect.x0 + 2.0 + if not (outside_bottom or outside_top or outside_right or outside_left): + continue + lines = [clean_cell(line) for line in str(block[4] or "").splitlines() if clean_cell(line)] + if not lines: + continue + # Prefer blocks that match the known column count (one cell per line). + if column_count and len(lines) == column_count: + row = lines + elif column_count and len(lines) > 1 and len(lines) < column_count: + # Partial remnant still useful for search/OCR text. + row = lines + [""] * (column_count - len(lines)) + else: + continue + key = tuple(row) + if key in existing_keys: + continue + existing_keys.add(key) + recovered.append(row) + return recovered + + def rect_intersection_ratio(a, b): intersection = a & b if intersection.is_empty: @@ -937,33 +1081,68 @@ def extract(pdf_path, output_dir, budget=None): table_candidates.extend(fallback_table_candidates(page, [candidate["rect"] for candidate in table_candidates])) table_candidates = merge_related_table_candidates(table_candidates, page.rect) for table_candidate in table_candidates: - table_rect = expanded_rect(table_candidate["rect"], page.rect, 4, 4) + detected_rect = table_candidate["rect"] + edge_rect, edge_info = extend_table_rect_for_edge_content(page, detected_rect) + table_rows = list(table_candidate.get("table_rows") or []) + column_count = int(table_candidate.get("column_count") or (len(table_rows[0]) if table_rows else 0) or 0) + recovered_rows = [] + if edge_info["extended"]: + recovered_rows = recover_table_rows_from_edge_strip( + page, + detected_rect, + edge_rect, + table_rows, + column_count, + ) + if recovered_rows: + table_rows = (table_rows + recovered_rows)[:MAX_TABLE_ROWS] + rows_truncated = bool(table_candidate.get("rows_truncated")) or bool(edge_info["clipped_by_page"]) + if edge_info["clipped_by_page"]: + warnings.append( + f"table_crop_edge_incomplete: page {page_number} table continues past the page edge; " + "retained the on-page remnant in the crop" + ) + table_rect = expanded_rect(edge_rect, page.rect, 4, 4) + table_text = (table_candidate.get("table_text") or "")[:MAX_TABLE_TEXT_CHARS] + accessible = ( + table_candidate.get("accessible_table_markdown") or table_candidate.get("table_text") or "" + )[:MAX_TABLE_TEXT_CHARS] + if recovered_rows: + accessible = (table_rows_to_markdown(table_rows) or accessible)[:MAX_TABLE_TEXT_CHARS] + table_text = accessible + crop_metadata = { + "pageNumber": page_number, + "source_kind": "table_crop", + "candidate_type": "table", + "table_label": table_candidate.get("table_label"), + "table_title": table_candidate.get("table_title"), + "table_text": table_text, + "table_role": table_candidate.get("table_role"), + "table_confidence": table_candidate.get("table_confidence"), + "table_rows": table_rows, + "table_columns": table_candidate.get("table_columns") or [], + "accessible_table_markdown": accessible, + "row_count": len(table_rows) if table_rows else table_candidate.get("row_count"), + "rows_truncated": rows_truncated, + "column_count": column_count or table_candidate.get("column_count"), + "heading_text": table_candidate.get("heading_text"), + "bbox": rect_payload(table_rect), + "extraction_method": table_candidate.get("extraction_method"), + "table_index": table_candidate.get("table_index"), + } + if edge_info["extended"]: + crop_metadata["edge_content_extended"] = True + crop_metadata["edge_content_edges"] = edge_info["edges"] + if edge_info["clipped_by_page"]: + # Partial last row still visible on-page, but structured extraction cannot be complete. + crop_metadata["crop_completeness"] = 0.9 crop = save_page_crop( page, table_rect, output_dir, f"page-{page_number}-table-crop-{crop_index}.png", "table_crop", - { - "pageNumber": page_number, - "source_kind": "table_crop", - "candidate_type": "table", - "table_label": table_candidate.get("table_label"), - "table_title": table_candidate.get("table_title"), - "table_text": (table_candidate.get("table_text") or "")[:MAX_TABLE_TEXT_CHARS], - "table_role": table_candidate.get("table_role"), - "table_confidence": table_candidate.get("table_confidence"), - "table_rows": table_candidate.get("table_rows") or [], - "table_columns": table_candidate.get("table_columns") or [], - "accessible_table_markdown": (table_candidate.get("accessible_table_markdown") or table_candidate.get("table_text") or "")[:MAX_TABLE_TEXT_CHARS], - "row_count": table_candidate.get("row_count"), - "rows_truncated": bool(table_candidate.get("rows_truncated")), - "column_count": table_candidate.get("column_count"), - "heading_text": table_candidate.get("heading_text"), - "bbox": rect_payload(table_rect), - "extraction_method": table_candidate.get("extraction_method"), - "table_index": table_candidate.get("table_index"), - }, + crop_metadata, budget, warnings, ) diff --git a/worker/python/fixtures/README.md b/worker/python/fixtures/README.md new file mode 100644 index 000000000..35b6c67a9 --- /dev/null +++ b/worker/python/fixtures/README.md @@ -0,0 +1,26 @@ +# PDF extraction fixtures + +## `malformed-table-crop-page-edge.pdf` + +Minimal authorised **synthetic** clinical-table PDF for outstanding issue `#076`. + +- No patient identifiers, no third-party copyrighted content. +- A4 page (`595.28 × 841.89`), one titled 4-row table whose final row straddles the page bottom (`y0=824` … `y1=860`; page ends at `841.89`). +- Regenerated with PyMuPDF via the geometry constants pinned in `test_extract_pdf_assets_page_edge_crop.py`. + +### Failing stage (pre-fix) + +`page.find_tables()` (`extraction_method=pymupdf_find_tables`) returns a candidate bbox that ends at the last **fully** detected row (`y1=824`). Without extension, `expanded_rect(..., 4, 4)` yielded crop clip `[46, 712, 564, 828]`, so: + +- ~14 px of still-on-page content for the straddling score-`5` row was omitted from the `table_crop` pixmap +- structured `table_rows` silently dropped that row +- no incompleteness warning was emitted + +### Fix stage + +`extend_table_rect_for_edge_content` grows the candidate using contiguous **cell drawings** toward the page edge, then `save_page_crop` renders the clamped remnant. Structured text in the extended strip is recovered when possible; page-clipped continuation sets `rows_truncated`, `crop_completeness=0.9`, and a `table_crop_edge_incomplete` warning. + +### Expected geometry after the fix + +- `table_crop.bbox[3]` (clip bottom) ≥ `page_height - 1` +- structured rows include the on-page remnant of score `5`, or incompleteness is marked diff --git a/worker/python/fixtures/malformed-table-crop-page-edge.pdf b/worker/python/fixtures/malformed-table-crop-page-edge.pdf new file mode 100644 index 000000000..df36d10be Binary files /dev/null and b/worker/python/fixtures/malformed-table-crop-page-edge.pdf differ diff --git a/worker/python/test_extract_pdf_assets_page_edge_crop.py b/worker/python/test_extract_pdf_assets_page_edge_crop.py new file mode 100644 index 000000000..7163cc866 --- /dev/null +++ b/worker/python/test_extract_pdf_assets_page_edge_crop.py @@ -0,0 +1,105 @@ +"""Regression for #076: page-edge table crops keep on-page remnant geometry. + +Failing stage (pre-fix): pymupdf_find_tables candidate detection stopped at the last +fully detected row. Fix stage: extend_table_rect_for_edge_content + save_page_crop. +Expected geometry: table_crop clip bottom reaches the page bottom and the straddling +score-5 remnant is retained (structured row and/or incompleteness signal). +""" + +from __future__ import annotations + +import os +import sys +import tempfile +import unittest + +sys.path.insert(0, os.path.dirname(__file__)) +import extract_pdf_assets as extractor + +FIXTURE_DIR = os.path.join(os.path.dirname(__file__), "fixtures") +FIXTURE_PDF = os.path.join(FIXTURE_DIR, "malformed-table-crop-page-edge.pdf") + +# Geometry pinned by the checked-in synthetic fixture (A4). +PAGE_WIDTH = 595.28 +PAGE_HEIGHT = 841.89 +STRADDLING_ROW_Y0 = 824.0 +EXPECTED_SCORE_ROW = "5" + + +@unittest.skipUnless(os.path.isfile(FIXTURE_PDF), "page-edge crop fixture missing") +class PageEdgeTableCropGeometryTests(unittest.TestCase): + def test_fixture_find_tables_stage_still_drops_straddling_row_bbox(self): + """Upstream diagnosis: find_tables alone still truncates; the fix extends afterward.""" + document = extractor.fitz.open(FIXTURE_PDF) + try: + page = document[0] + self.assertAlmostEqual(float(page.rect.width), PAGE_WIDTH, places=2) + self.assertAlmostEqual(float(page.rect.height), PAGE_HEIGHT, places=2) + + tables = page.find_tables() + self.assertGreaterEqual(len(tables.tables), 1, "fixture must expose a find_tables candidate") + candidate = extractor.fitz.Rect(tables.tables[0].bbox) + self.assertLess( + candidate.y1, + PAGE_HEIGHT - 1, + "fixture no longer stresses find_tables truncation at the page edge", + ) + self.assertAlmostEqual(candidate.y1, STRADDLING_ROW_Y0, places=1) + + extended, info = extractor.extend_table_rect_for_edge_content(page, candidate) + self.assertTrue(info["extended"]) + self.assertIn("bottom", info["edges"]) + self.assertTrue(info["clipped_by_page"]) + self.assertGreaterEqual(float(extended.y1), PAGE_HEIGHT - 1) + finally: + document.close() + + def test_table_crop_includes_on_page_remnant_of_straddling_row(self): + """ + Stage: pymupdf_find_tables → extend_table_rect_for_edge_content → save_page_crop + Expected geometry: clip_bbox[3] >= page_height - 1 (on-page remnant retained) + Expected structure: table_rows includes the score-5 clinical row (or incompleteness is marked) + """ + with tempfile.TemporaryDirectory(prefix="page-edge-crop-") as output_dir: + result = extractor.extract(FIXTURE_PDF, output_dir) + crops = [image for image in result["images"] if image.get("sourceKind") == "table_crop"] + self.assertEqual(len(crops), 1, result) + crop = crops[0] + metadata = crop.get("metadata") or {} + clip = crop.get("bbox") or metadata.get("clip_bbox") + self.assertIsInstance(clip, list) + self.assertGreaterEqual(len(clip), 4) + + clip_bottom = float(clip[3]) + page_height = float(metadata.get("page_height") or PAGE_HEIGHT) + self.assertGreaterEqual( + clip_bottom, + page_height - 1.0, + ( + "stage=pymupdf_find_tables→extend_table_rect_for_edge_content→save_page_crop; " + f"expected clip bottom ≥ {page_height - 1.0}, got {clip_bottom}; " + f"candidate method={metadata.get('extraction_method')}" + ), + ) + + rows = metadata.get("table_rows") or [] + has_score_row = any(str(row[0]).strip() == EXPECTED_SCORE_ROW for row in rows if row) + crop_completeness = metadata.get("crop_completeness") + incomplete = ( + isinstance(crop_completeness, (int, float)) and float(crop_completeness) < 0.99 + ) or any( + "table_crop_edge_incomplete" in str(warning) for warning in result.get("warnings") or [] + ) + self.assertTrue( + has_score_row or incomplete, + ( + "expected score-5 row in table_rows or an explicit incompleteness signal; " + f"rows={rows!r} crop_completeness={crop_completeness!r} warnings={result.get('warnings')!r}" + ), + ) + self.assertTrue(metadata.get("edge_content_extended")) + self.assertIn("bottom", metadata.get("edge_content_edges") or []) + + +if __name__ == "__main__": + unittest.main()