feat(ci): opt-in answer-quality/targeting eval on canary dispatch (Phase E instrument)

[BigSimmo]

Summary

First Phase E delivery (user-approved with budget): give the canary an opt-in answer_quality_eval dispatch input that runs the 30-case eval:answer-quality fixture — five answer-quality metrics (relevance, readability, artifact leaks, intent coverage, fail-closed) plus the per-intent structural targeting metric (dose→figure, red-result→withhold action, monitoring→schedule). E-1 recon found this instrument has never run in CI, and the canary's answer gate samples only 8 of the 44 eval:quality cases — so the Phase E baseline needs this step.

Design constraints honored:

• Dispatch-only and default-off: scheduled runs and every existing dispatch shape are byte-identical (if: workflow_dispatch && input == 'true'); follows the feat(ci): eval-canary dispatch inputs — staged ranking-config override + branch evals (ADDENDUM 4 B-PR-1 / D-1) #1003 input-only precedent.
• Informational by construction: the script exits 0 without --targeting-floor, so this step can never fail a canary — quality gates stay owned by the eval:quality step.
• The tee'd log lands in .local/eval-canary/ and rides the existing pinned artifact upload; 45-minute job timeout comfortably absorbs the ~10-minute step.

Next (after merge): the E-2 baseline dispatch — answer_case_limit=44 + answer_quality_eval=true — one run giving the full answer-quality baseline on both instruments plus the golden retrieval eval as the built-in no-regression net. Estimated $3–8 of the user-authorized ≤$20 Phase E envelope; reported before/after dispatch.

RAG impact: no retrieval behaviour change — dispatch-input-only workflow addition; retrieval steps, schedule, and defaults untouched.

Verification

• npm run check:github-actions — pass (no new actions; existing pins untouched)
• npm run check:ci-scope — pass
• npm run check:gate-manifest — pass
• npx prettier --check on both changed files — clean
• No provider calls in this PR (workflow-definition change only); the paid dispatch happens separately under the approved envelope

Risk and rollout

• Risk: low — a conditioned, default-off step in a dispatch-only path; scheduled behaviour provably unchanged (condition requires workflow_dispatch AND explicit 'true'). The step is informational and cannot fail the run.
• Rollback: revert.
• Provider or production effects: none from this diff. When the input is enabled on a dispatch, the step spends 30 live generation cases ($1–3) — always under an explicit user-approved dispatch.

Clinical Governance Preflight

• Source-backed claims still require linked source verification before clinical use — no behaviour change
• No patient-identifiable document workflow was introduced or expanded
• Supabase target remains Clinical KB Database (sjrfecxgysukkwxsowpy) — same env block, no new secret exposure
• Service-role keys and private document access remain server-only
• Demo/synthetic content remains clearly separated from real clinical sources
• Source metadata, review status, and outdated/unknown-source behavior remain conservative
• Deployment classification/TGA SaMD impact was checked: no clinical decision-support behaviour changed — measurement only

🤖 Generated with Claude Code

https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9

---

Generated by Claude Code

Summary by CodeRabbit

• New Features

  • Added an optional dispatch-only workflow input to run an additional 30-case answer-quality/targeting evaluation.
  • When enabled, the workflow produces a dedicated log artifact containing the targeting evaluation output.

• Documentation

  • Updated the Branch Review Ledger with the Phase E kickoff details, including the default-off behavior, canary sampling notes, and baseline follow-up parameters.

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: c747c86e-975a-4951-84c8-c9d411cba307

📥 Commits

Reviewing files that changed from the base of the PR and between f2fada0 and 22d9262.

📒 Files selected for processing (1)

• docs/branch-review-ledger.md

🚧 Files skipped from review as they are similar to previous changes (1)

• docs/branch-review-ledger.md

---

📝 Walkthrough

Walkthrough

The canary workflow adds an optional dispatch input for running the answer-quality evaluation and writing a dedicated log. Scheduled and default dispatch behavior remains unchanged, and the review ledger documents the update.

Changes

Answer-quality canary evaluation

Layer / File(s)	Summary
Add optional answer-quality evaluation .github/workflows/eval-canary.yml, docs/branch-review-ledger.md	The workflow adds the default-off answer_quality_eval dispatch input and conditionally runs npm run eval:answer-quality; the ledger records the dispatch-only behavior and evaluation details.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

• BigSimmo/Database#624: Updates the answer-quality evaluation’s latency context and thresholds in the related evaluation flow.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the opt-in canary answer-quality eval addition.
Description check	✅ Passed	The description covers the required summary, verification, risk, and governance sections and is mostly complete.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/clinical-kb-pwa-review-asi3wb

---

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.