feat(ci): eval-canary dispatch inputs — staged ranking-config override + branch evals (ADDENDUM 4 B-PR-1 / D-1)

[BigSimmo]

Summary

Two optional workflow_dispatch inputs on eval-canary.yml — the staging rails for Phase B and the pre-merge eval gap-closer from Phase D-1. Scheduled runs are completely unaffected (empty inputs resolve to today's exact behavior).

• rag_ranking_config — stages a RAG_RANKING_CONFIG JSON override for a single live eval run, the documented rollout path for tuned ranking weights (no code defaults touched; unset = production weights). A new validation step fails the run loudly on malformed JSON: resolveRankingConfig (src/lib/ranking-config.ts:183-191) silently falls back to production defaults on a parse error, which would turn a staged-weights eval into an accidental baseline and corrupt the before/after pair comparison. The step also echoes the active override into the log for auditability.
• ref — runs the eval from a branch/SHA (Phase D-1), closing the PR-template gap "CI cannot run the golden eval — needs live keys": ranking PRs can now gather live proof pre-merge via one dispatch.

First consumer (next step, separate dispatch): the tuned 3-class config from the offline tuner on the fresh A-PR-2 snapshot — document_lookup titleSectionRelevance→0.9, table_threshold clinicalEvidence→0.95, comparison hybridRelevance→0.95 (proxy rank quality 0.833→1.0) — validated live against the re-gated #53 baseline (mrr@10 ≈0.892) with targets mrr@10 ≥0.90, irrelevant@10 ≤0.08, zero case regressions.

Verification

• npm run check:github-actions — pin check passed (no new actions; checkout pin unchanged)
• npm run check:ci-scope — passed
• Prettier clean
• No unit-test surface (workflow-only diff); behavior proof is the next dispatch itself: a run with an override logs RAG_RANKING_CONFIG override active: {...}, a malformed override fails at the validation step, and the weekly schedule continues running with empty inputs
• Verification not run: verify:pr-local — workflow-only change; the gates above are the repo's designated checks for .github/workflows/ diffs

Risk and rollout

• Risk: operational (workflow inputs). Two considered hazards: (1) silent-default corruption — mitigated by the loud validation step (the precise failure mode this PR exists to prevent); (2) ref runs branch code with the eval job's live secrets — bounded because workflow_dispatch requires repository write access, and this repo has a single trusted collaborator; the input only widens which commit an already-trusted dispatcher can evaluate. No new secrets, no permission changes, concurrency/issue-reporting behavior untouched.
• Rollback: plain revert — inputs disappear, schedule cadence unaffected throughout.
• Provider or production effects: none from this diff itself; it enables explicitly-dispatched staged evals (~$1–2 each, inside the user-raised ≤$20 ADDENDUM 4 cap, ~$1–2 spent).

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — no answer-path change
• No patient-identifiable document workflow introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — env block unchanged apart from the pass-through override variable
• Service-role keys and private document access remain server-only — no new secret exposure
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative — ranking defaults untouched; staged overrides are per-run and revert by omission
• Deployment classification/TGA SaMD impact: no production behavior change (eval tooling only)

Notes

• ADDENDUM 4 Phase B step 2 rails + Phase D-1 in one operational diff. On merge, the tuned-weights dispatch (task in flight) produces the live half of the pair.
• Ledger row appended (content commit f7f0b18).

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

---

Generated by Claude Code

Summary by CodeRabbit

• New Features

  • Evaluation workflows now support optional ranking configuration overrides and branch or commit selection.
  • Invalid ranking configuration JSON is detected before evaluation runs.

• Documentation

  • Added ledger documentation covering the new workflow inputs, validation behavior, and trusted-access considerations.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 83ba3a25-8697-4f3b-bc11-b0766f47f661

📥 Commits

Reviewing files that changed from the base of the PR and between adefc0e and cdb03db.

📒 Files selected for processing (2)

• .github/workflows/eval-canary.yml
• docs/branch-review-ledger.md

---

📝 Walkthrough

Walkthrough

The Eval Canary workflow now accepts optional ranking configuration and branch/SHA inputs, validates non-empty ranking overrides as JSON objects, and checks out the selected reference. The branch review ledger documents these inputs and dispatch constraints.

Changes

Eval Canary overrides

Layer / File(s)	Summary
Dispatch override inputs and validation .github/workflows/eval-canary.yml	Adds optional rag_ranking_config and ref inputs, wires the ranking override into the workflow environment, and rejects non-object JSON values.
Reference checkout and dispatch documentation .github/workflows/eval-canary.yml, docs/branch-review-ledger.md	Uses the optional reference during checkout and documents scheduled-run behavior and dispatch authorization constraints.

Estimated code review effort: 2 (Simple) | ~10 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Dispatcher
  participant Eval_Canary
  participant Ranking_Config_Validator
  participant actions_checkout
  Dispatcher->>Eval_Canary: submit optional rag_ranking_config and ref
  Eval_Canary->>Ranking_Config_Validator: validate non-empty ranking override
  Ranking_Config_Validator-->>Eval_Canary: accept object or fail
  Eval_Canary->>actions_checkout: checkout selected ref or default

Loading

Suggested reviewers: claude

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.