feat(eval): snapshot regeneration from live artifact + alias tiering docs + fixture-length pin (ADDENDUM 4 A-PR-2)

[BigSimmo]

Summary

Phase A completion (ADDENDUM 4, PR 2 of 2) — all parts landed; eval ground truth + measurement scaffolding only, no runtime ranking behavior change.

• Two-tier alias documentation (drift guard): cross-referencing headers in scripts/lib/clinical-aliases.ts (strict golden tier) and src/lib/eval-document-matching.ts (deliberately wider captured-case tier). Investigation of the A-PR-1 governance finding showed bulk-merging would silently loosen golden ground truth (e.g. "Clozapine GP Shared Care" satisfying the pinned prescribing-guideline expectation) — so the tiers stay separate, now explicitly. Comments only (reviewer-verified byte-identical tables via pure-additive diffs).
• Fixture-length snapshot pin: snapshot case count locked to the live golden fixture length (was hardcoded 36) with regeneration instructions in the failure message — fixture and snapshot can only evolve in lockstep.
• First provenance-stamped snapshot regeneration: rag-ranking-candidate-snapshot.v1.json rebuilt from canary run 29763761133's artifact (sha256 5af5b802… verified byte-identical after transfer; 36/36 green, doc_recall 1.0, content_recall 1.0 — first perfect content recall, CIWA alias confirmed live). The alias-aware builder proves itself on real data: agitation-im-po-options goes 0→5 graded positives (EMHS guideline recognized); flowchart-next-step confirmed as the sole zero-positive case.
• generatedAt now validator-required (was optional): every builder output is stamped, and an absent value would silently disable the now-active 30-day freshness gate. Closes the governance review's hand-edit follow-up.
• Lithium document gate from live evidence: lithium-therapy-monitoring was the only golden case with no document expectation — its rr@10 was a hardcoded 0.00 (measurement noise). It gains expectedDocumentSubstrings: ["Lithium"], deliberately broad because the corpus carries multiple legitimate lithium guidelines (EMHS clinical Bump actions/checkout from 4 to 6 #1, FSH initiation Bump actions/setup-node from 4 to 6 #2, CAMHS Bump @types/node from 20.19.41 to 25.9.1 #3 live) — the gate asserts subject precision, not a single title that would flake between equally-correct documents. Measured mrr@10 rises ~+0.028 from de-noising alone. Snapshot rebuilt in lockstep from the same artifact. ⚕️ Clinical sign-off requested on this one fixture change (marking this PR ready = sign-off).
• Artifact-grounded punctuation audit: 7 joined-token occurrences in live top-5 previews — 3 are the alias-covered CIWA-Ar class (including a line-broken ciwa- ar variant), 4 are ordinary prose punctuation (treatment, mood, (opioid ptsd.[35]) invisible to the whitespace-delimited matcher. Per the approved plan, this systemic class is proposed as its own reviewed follow-up (word-boundary matching), not bundled here.

Deferred with reasons: new-query fixture cases (saturated-tie shapes, captured rag_query_misses) need live validation before they may gate — unlocked by Phase D-1 (branch-eval dispatch input) or a dedicated validation dispatch. Real Phase B ordering headroom, now cleanly measured: flowchart 0.20, alcohol-ciwa 0.25, patient-safety 0.33, opioid 0.33 rr@10 — all text_fast_path.

Verification

• Targeted vitest 76/76 (ranking-tuning, eval-retrieval, eval-quality, rag-eval-cases) — re-run green after each stage
• npm run test — 3019 passed / 1 failed: the known container-only pdf-extraction-budget artifact (baselined on clean main)
• npm run lint + npm run typecheck — clean (typecheck covers the generatedAt type change across all snapshot consumers)
• Prettier clean (fixtures + touched files)
• Hosted CI pr-required: success on final head 99e8be5 (run 4088)
• Freshness gate: active and green on the regenerated snapshot
• npm run eval:retrieval:quality not run locally (no live keys in sandbox): the source artifact IS the live 36/36 evidence, and the next scheduled canary re-validates the gated lithium case live (it ranks Bump actions/checkout from 4 to 6 #1–4 in the current artifact, so the gate is satisfied with margin)

Risk and rollout

• Risk: eval ground truth only (reviewer-verified: no src/ runtime consumer of any touched module/fixture). The lithium gate could theoretically fail a future canary if the corpus lost all lithium-titled documents from the top-5 — which would be a real retrieval regression worth failing on. Validator hardening cannot brick other consumers (typecheck + reviewer grep of all three consumers).
• Rollback: plain revert per commit; reverting the regeneration restores the pre-provenance snapshot (validator change reverts with it — same commit, atomic).
• Provider or production effects: None in this diff. Source artifact came from the user-authorized canary dispatch (~$1–2; ADDENDUM 4 spend ~$1–2 of ≤$10).

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — answer path untouched
• No patient-identifiable document workflow introduced — snapshot keeps sha256 identity + exclusion sanitization on the new data (reviewer: 189/189 hashed candidates, forbidden-content scans clean)
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy)
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata and conservative-behavior invariants unchanged; alias tables NOT merged (documented tiering instead); the single fixture-expectation change is flagged above for clinical sign-off
• Deployment classification/TGA SaMD impact: no clinical decision-support behavior changed

Notes

• Clinical-governance reviewer verdict: APPROVE (no P0/P1/P2 findings). All five verification targets independently confirmed: byte-identical alias tables (pure-additive numstat), sanitization contract on the new snapshot data, lithium gate clinically sound + strictly tightening + matching the fixture's broad-substring convention, generatedAt requirement safe for all three consumers, and no safety gate weakened by the pin updates. Two informational P3s recorded: (a) treat 2026-07-20 as a baseline step when trending absolute mrr@10 (the lithium de-noising is a one-time +~0.028 that gates nothing); (b) the now-active freshness gate will fail the offline suite ~2026-08-19 unless the snapshot is regenerated from a fresh canary artifact — the intended fail-closed anti-drift mechanism, with a recurring ~$1–2 regeneration cost (or $0 when piggybacked on the weekly scheduled run).
• Ledger row appended (content commits e58c827, 5b664f8, 3d8f798).
• Phase A of ADDENDUM 4 is functionally complete on merge; Phase B (offline tuner → staged RAG_RANKING_CONFIG → live canary pair) starts from this snapshot.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

Summary by CodeRabbit

• Improvements

  • Refreshed search relevance evaluation data and ranking measurements using current results.
  • Improved validation of ranking snapshots by requiring complete timestamps and matching the current golden test set.
  • Updated lithium therapy monitoring retrieval checks to verify relevant content is returned.
  • Adjusted evaluation scoring and coverage expectations for more accurate relevance assessment.

• Documentation

  • Clarified clinical alias tiering guidance and evidence requirements.
  • Added provenance and freshness details to review records.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d43e9a4d-8493-42c3-bc61-f41a0f5a7084

📥 Commits

Reviewing files that changed from the base of the PR and between 69fc18b and 99e8be5.

📒 Files selected for processing (7)

• docs/branch-review-ledger.md
• scripts/fixtures/rag-ranking-candidate-snapshot.v1.json
• scripts/fixtures/rag-retrieval-golden.json
• scripts/lib/clinical-aliases.ts
• scripts/lib/ranking-tuning.ts
• src/lib/eval-document-matching.ts
• tests/ranking-tuning.test.ts

---

📝 Walkthrough

Walkthrough

The PR regenerates ranking snapshots with provenance metadata, requires valid snapshot timestamps, updates retrieval and alias-tier evidence, aligns tests with the golden fixture count, and records measurement-floor verification results.

Changes

Ranking measurement-floor refresh

Layer / File(s)	Summary
Snapshot contract and regenerated evidence scripts/lib/ranking-tuning.ts, scripts/fixtures/rag-ranking-candidate-snapshot.v1.json	Ranking snapshots require generatedAt, while candidate hashes, relevance fields, and scoring features are regenerated with provenance metadata.
Alias and retrieval evidence gates scripts/lib/clinical-aliases.ts, src/lib/eval-document-matching.ts, scripts/fixtures/rag-retrieval-golden.json	Alias tier boundaries are documented, and the lithium retrieval case now requires a document containing Lithium.
Validation assertions and measurement record tests/ranking-tuning.test.ts, docs/branch-review-ledger.md	Tests derive snapshot counts from the golden fixture, validate missing timestamps, update tuner invariants, and record the measurement-floor completion results.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

• BigSimmo/Database#901: Updates the same ranking-tuning snapshot and validation path.
• BigSimmo/Database#982: Updates related lithium retrieval and ranking fixtures.
• BigSimmo/Database#999: Adds related ranking-snapshot provenance and freshness expectations.

Suggested labels: codex

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.