feat(eval): canary retrieval artifact + alias-aware ranking-snapshot builder with freshness provenance (ADDENDUM 4 A-PR-1)

[BigSimmo]

Summary

Measurement floor for the approved evidence-gated ranking-tuning phase — no ranking behavior change anywhere in this diff. Three gaps closed:

• Canary artifact emission ($0 regeneration path): the eval-canary golden step now passes --json-out and a pinned actions/upload-artifact step uploads .local/eval-canary/ on success and failure (30-day retention = the freshness window). --json-out is decoupled from --json in scripts/eval-retrieval.ts so the artifact file writes independently while stdout keeps the human-readable per-case lines and summary that the tee'd log — and the failure-issue analyzeFailureText step — depend on. Ranking-snapshot regeneration stops costing a paid dispatch.
• Alias-aware snapshot grading: clinicalDocumentAliases/clinicalContentAliases move verbatim from scripts/eval-retrieval.ts into shared scripts/lib/clinical-aliases.ts; the snapshot builder now grades documentMatch/contentMatch through the same sanctioned expansions the live gates use (documentLabelMatchesWithAliases/contentLabelMatchesWithAliases in scripts/lib/ranking-snapshot-builder.ts). Previously alias-satisfied hits (e.g. the EMHS agitation guideline, spelled-out "absolute neutrophil count") graded as misses, feeding the offline tuner mislabeled ground truth. Discriminating tests pin that raw labelMatches fails both cases and the alias-aware matchers pass them.
• Snapshot provenance + freshness: the builder stamps generatedAt (+ optional --source-run-id); validateRankingSnapshot accepts/validates both and relaxes exactly-36 to at-least-36 (floor still rejects truncated artifacts; sourceCaseCount match and per-case candidate minimums unchanged) ahead of the planned fixture expansion. A 30-day freshness test in tests/ranking-tuning.test.ts activates on first regeneration and then blocks silent corpus drift, with the regeneration command in its failure message. New build:ranking-snapshot npm script.

Static hyphen audit of all 36 golden cases' content terms (CIWA-Ar class): no currently-blocked term (canary #52 is 36/36). Residual risk classes documented for the artifact-grounded A-PR-2 pass: punctuation-joined corpus tokens (IM/PO, schizo-affective, post-natal) and inert stem entries (obsess/compuls/hyperactiv/impuls can never match under the whole-token matcher; those cases currently pass via whole-word OR-alternates).

Builder verified end-to-end on a synthetic 36-case artifact: alias grading, provenance stamping, and validator acceptance all confirmed.

Verification

• npm run verify:pr-local — Verification not run locally: hosted pr-required covers build/bundle-scan/RAG-fixture validation; locally ran the full unit suite + the workflow-specific gates below inside the pre-18:00Z merge window
• Targeted vitest 52/52 (tests/ranking-tuning.test.ts, tests/eval-retrieval.test.ts, tests/eval-quality.test.ts), re-run green on the main-synced head
• npm run test — 3019 passed / 1 failed: the known container-only pdf-extraction-budget child-process artifact (baselined on clean main)
• npm run lint + npm run typecheck — clean
• npm run check:github-actions + npm run check:ci-scope — PASS (workflow diff)
• npm run check:production-readiness — expected missing-secret FAILs only (demo-mode container, no live keys)
• npm run eval:retrieval:quality not run: no retrieval/ranking/selection/scoring behavior changed (eval harness scaffolding + workflow only; live gates consume the identical alias maps). Live confirmation is free: tonight's 18:00 UTC scheduled canary runs the unchanged gates and emits the first artifact.

Risk and rollout

• Risk: operational (workflow) + eval-harness scaffolding. Worst credible failures: a malformed upload step breaks the canary job (mitigated: if: always() + if-no-files-found: ignore; pin verified against the repo allowlist), or --json-out decoupling alters eval stdout (mitigated: per-case lines and human summary asserted unchanged by existing eval tests; JSON dump only ever printed under explicit --json). Snapshot-validator relaxation cannot loosen ground truth: ≥36 floor, sourceCaseCount equality, and per-case candidate minimums all still enforced.
• Rollback: plain revert — no cache semantics, no data, no ranking behavior involved.
• Provider or production effects: None from this diff itself. The artifact upload adds no provider calls; it exposes the same content class the tee'd step logs already expose (titles/file names, score telemetry, 220-char previews of the all-public clinical corpus) — no env values or secrets are written to .local/eval-canary/.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — answer path untouched
• No patient-identifiable document workflow was introduced or expanded — corpus is all-public guidelines; artifact carries the same fields the step logs already print
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — workflow env unchanged
• Service-role keys and private document access remain server-only — no new secret exposure; upload path contains eval output only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — alias tables moved verbatim, zero entries added/removed/reworded
• Deployment classification/TGA SaMD impact: no clinical decision-support behavior changed

Notes

• ADDENDUM 4 Phase A, PR 1 of 2. A-PR-2 (snapshot regeneration + artifact-grounded fixture expansion 36→~46) starts from the first emitted artifact — tonight's scheduled canary if this merges before 18:00 UTC.
• Ledger row appended (docs/branch-review-ledger.md, content commit 81ab969).
• Clinical-governance reviewer verdict: APPROVE-WITH-NITS (P3 only). Independently confirmed: alias tables moved byte-identical; live gate call sites unchanged; artifact payload adds no data class beyond the already-public tee'd step logs; truncation/floor/per-case validator checks intact; freshness gate fail-safe (dormant until first regeneration, malformed values rejected); no new provider calls.
• Tracking for A-PR-2 (from the review): (a) a third, pre-existing and divergent clinicalDocumentAliases in src/lib/eval-document-matching.ts (consumed by src/lib/rag/rag-eval-cases.ts) should be reconciled with the shared module; (b) once the fixture grows past 36, pin the validator floor to the live fixture length instead of the coarse ≥36; (c) after first regeneration stamps generatedAt, consider making it required so a hand-edit cannot silently re-disable the freshness gate.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

Summary by CodeRabbit

• New Features

  • Added a command to build ranking snapshots from evaluation artifacts.
  • Ranking snapshots now include generation timestamps and optional source-run identifiers.
  • Evaluation runs can produce downloadable JSON artifacts while retaining readable console results.
  • Improved clinical document and content matching using recognized aliases.

• Validation

  • Snapshots now support 36 or more cases and validate freshness and provenance metadata.
  • Added safeguards for incomplete or corrupted evaluation artifacts.

• Documentation

  • Updated review records to document the enhanced evaluation and snapshot workflow.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a6d4fd76-fea7-4b53-a53b-788818fb5be3

📥 Commits

Reviewing files that changed from the base of the PR and between b57acf2 and f28f95c.

📒 Files selected for processing (9)

• .github/workflows/eval-canary.yml
• docs/branch-review-ledger.md
• package.json
• scripts/build-ranking-snapshot.ts
• scripts/eval-retrieval.ts
• scripts/lib/clinical-aliases.ts
• scripts/lib/ranking-snapshot-builder.ts
• scripts/lib/ranking-tuning.ts
• tests/ranking-tuning.test.ts

---

📝 Walkthrough

Walkthrough

The change centralizes clinical aliases, updates retrieval output handling, preserves canary evaluation artifacts, and adds provenance metadata plus relaxed case-count validation for ranking snapshots. Tests cover alias matching, freshness, provenance, and truncated artifacts.

Changes

Ranking snapshot evaluation

Layer / File(s)	Summary
Shared alias matching scripts/lib/clinical-aliases.ts, scripts/lib/ranking-snapshot-builder.ts, scripts/eval-retrieval.ts	Clinical document and content aliases are centralized and used by retrieval evaluation and snapshot candidate grading.
Evaluation artifact emission scripts/eval-retrieval.ts, .github/workflows/eval-canary.yml	--json-out writes JSON without suppressing human-readable output, and the canary uploads evaluation logs and JSON for 30 days.
Snapshot provenance and validation scripts/build-ranking-snapshot.ts, scripts/lib/ranking-tuning.ts, package.json	Snapshot generation accepts at least 36 cases, records generatedAt and optional sourceRunId, validates both fields, and exposes a build script.
Validation coverage and ledger tests/ranking-tuning.test.ts, docs/branch-review-ledger.md	Tests cover alias matching, freshness, provenance, and truncated snapshots; the review ledger records the measurement update.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant EvalCanary
  participant EvalRetrieval
  participant ArtifactStorage
  EvalCanary->>EvalRetrieval: run evaluation with --json-out
  EvalRetrieval->>ArtifactStorage: write JSON and human-readable log
  EvalCanary->>ArtifactStorage: upload evaluation directory

Loading

sequenceDiagram
  participant SnapshotBuilder
  participant AliasMatchers
  participant SnapshotValidator
  SnapshotBuilder->>AliasMatchers: match document and content labels
  AliasMatchers-->>SnapshotBuilder: return alias-aware matches
  SnapshotBuilder->>SnapshotValidator: validate generated snapshot
  SnapshotValidator-->>SnapshotBuilder: accept snapshot metadata and cases

Loading

Possibly related PRs

• BigSimmo/Database#901: Introduced the ranking snapshot and tuner code extended by these alias-aware matching, validation, and provenance changes.

Suggested reviewers: claude

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.