From f2fada0828ca46e977796ec2a12f10c9d5b5b40d Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 22:44:01 +0000 Subject: [PATCH] feat(ci): opt-in answer-quality/targeting eval on canary dispatch (Phase E instrument) Adds an answer_quality_eval dispatch input to eval-canary that runs the 30-case eval:answer-quality fixture (five quality metrics + per-intent structural targeting) after the eval:quality gate subset. The instrument existed but was never run in CI. Dispatch-only and default-off: scheduled runs and existing dispatch shapes are byte-identical, and the step is informational by design (exit 0 without --targeting-floor) so quality gates stay owned by eval:quality. The tee'd log rides the existing artifact upload. Follows the #1003 input-only precedent; retrieval steps untouched. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- .github/workflows/eval-canary.yml | 21 +++++++++++++++++++++ docs/branch-review-ledger.md | 1 + 2 files changed, 22 insertions(+) diff --git a/.github/workflows/eval-canary.yml b/.github/workflows/eval-canary.yml index 7b0b2faa0..3b085f9ab 100644 --- a/.github/workflows/eval-canary.yml +++ b/.github/workflows/eval-canary.yml @@ -33,6 +33,14 @@ on: description: "Optional git ref (branch/SHA) to evaluate; empty = default branch" required: false default: "" + # Phase E: opt-in run of the 30-case answer-quality/targeting eval (eval:answer-quality) + # alongside the gate subset. Dispatch-only and default-off, so scheduled runs and + # existing dispatch shapes are byte-identical. Informational: the script exits 0 + # without a --targeting-floor, so it can never fail the canary. + answer_quality_eval: + description: "Also run the 30-case answer-quality/targeting eval (eval:answer-quality); 'true' to enable" + required: false + default: "" schedule: # Sunday 18:00 UTC = Monday 02:00 Australia/Perth — off-peak for clinicians. - cron: "0 18 * * 0" @@ -155,6 +163,19 @@ jobs: set -o pipefail npm run eval:quality -- --rag-only --limit "$ANSWER_CASE_LIMIT" --fail-on-threshold 2>&1 | tee .local/eval-canary/answer-quality.log + # Phase E baseline/paired-run instrument: the five answer-quality metrics (relevance, + # readability, artifact leaks, intent coverage, fail-closed) + per-intent structural + # targeting over the 30-case fixture. Informational by design — no --targeting-floor, + # exit 0 — so quality gates stay owned by the eval:quality step above; the tee'd log + # rides the existing artifact upload. + - name: Answer-quality targeting eval (optional, dispatch-only) + id: answer_targeting + if: github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true' + run: | + mkdir -p .local/eval-canary + set -o pipefail + npm run eval:answer-quality 2>&1 | tee .local/eval-canary/answer-targeting.log + # Uploaded on success AND failure: the JSON artifact regenerates the offline ranking # snapshot (download it, then `npm run build:ranking-snapshot -- --input --output # scripts/fixtures/rag-ranking-candidate-snapshot.v1.json --source-run-id `), and diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 7b51dd458..ebec073a8 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -661,3 +661,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: matcher + artifact follow-ups) | ab145f6 | Remaining documented improvements implemented: word-boundary textContainsClinicalTerm + top-10 canary artifact rows | Matcher: boundaries + internal separators widened to any non-alphanumeric run — PROVEN strict superset by artifact replay on canary #53 (1,126 term×alias×result comparisons, 0 lost matches, 7 gained = exactly the previously-documented punctuation-joined occurrences: treatment,/mood,/(opioid/ptsd.[35]/ciwa-ar ×3). More-tolerant measurement cannot fail a passing case → weekly scheduled canary = free live confirmation. Exported + 3 direct unit-test groups (superset preservation, audit classes incl. line-broken 'ciwa- ar' and 'full-blood-count', substring-inside-word rejections). Artifact: topResultSummary 5→10 rows so rr@10/irrelevant@10 metrics' actual inputs are captured — unblocks the offline irrelevant@10 labeling audit next artifact. docs/rag-behaviour updated to implemented state. Phase E remains gated on separate approval. | Targeted vitest 59/59; npm run test 3028 passed / 1 known container pdf-budget artifact; lint+typecheck+prettier clean; audit script run recorded above; no provider calls | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: DB easy wins) | see PR head | User-requested database + process easy-wins pass: read-only Supabase advisor sweep (user-authorized) → canary-liveness probe + advisor-disposition docs; live mutations withheld for per-item confirmation | Advisors (live, read-only): security = 1 INFO (document_title_words RLS-no-policy = the deliberate fail-closed pattern — now comment-documented at the schema block so it is never 'fixed'); performance = ~33 unused-index INFOs + auth connection-strategy note → docs/db-maintenance.md TRIAGE list with retrieval-surface trgm indexes flagged RAG-protected (dropping = full canary protocol), owner-scoped indexes retained for multi-tenant design, operational candidates deferred (negligible benefit at corpus size). Implemented: ci.yml static-pr warn-only eval-canary staleness probe (actions:read, github-script pinned, >8 days → warning; never fails) — needed because #923 (2026-07-19) moved the canary cadence from daily to weekly Sunday 18:00 UTC, where a dropped fire would go unnoticed for a week (the failure-issue step only reacts to runs that happen). CORRECTION (CodeRabbit review on this PR): the initially recorded "2026-07-20 dropped Sunday fire" incident did not occur — 2026-07-20 is a Monday; the Sunday 2026-07-19 slot fired as scheduled run #48 (19:03 UTC, success), and under the weekly cron no 2026-07-20 slot existed. Probe stands as proactive hardening, staleness now measured from updated_at with a finite-timestamp guard. Presented for confirmation (NOT implemented): scheduled telemetry retention (purge:query-logs is owner-scoped + unscheduled; needs owner/window/policy decision), auth percentage connection strategy (dashboard config), any index drops. | check:github-actions PASS; check:ci-scope PASS; check:function-grants 28/28 PASS; docs:check-links 1034 PASS; supabase-schema vitest 66/66; lint+typecheck+prettier clean; Supabase access read-only only | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from e6cd6cb; PR: telemetry-retention correction) | 01368ff (+0144e69 main merge-in) | User-directed "implement your recommendation for all decisions" close-out: the scheduled-retention recommendation is RETRACTED as founded on a false premise — telemetry retention is ALREADY ACTIVE inside the database via pg_cron; docs corrected, nothing built, no live mutations | Fresh read-only cron.job verification (2026-07-20) matches docs/privacy-impact-assessment.md §6 exactly: jobid 11 purge-expired-rag-queries daily 03:30 (30d), jobid 12 purge-rag-retrieval-logs daily 03:00 (90d), jobid 13 purge-rag-query-misses daily 03:45 (90d), jobid 16 purge-rag-response-cache hourly (bounded 1000); v3 worker jobs present-inactive under backlog auto-toggle (jobid 10); obsolete unbounded cache job absent; audit_logs indefinite by design. db-maintenance.md "open decision" section replaced with the resolved state; purge:query-logs clarified as the MANUAL owner-scoped tool (not the retention mechanism). A GitHub-side weekly deleter would have duplicated pg_cron with window drift (a 90d rag_queries sweep can never out-delete the live 30d job). Remaining decisions stand as documented no-action: auth percentage connection strategy deferred to next instance resize; operational index drops not recommended. | prettier + docs:check-links PASS; Supabase access read-only (single cron.job SELECT, user-authorized read-only envelope); no workflow/schema/config changes | +| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from f33890a; PR: Phase E instrument input) | see PR head + follow-up SHA pin | Phase E kickoff (user-approved with budget): eval-canary gains opt-in `answer_quality_eval` dispatch input running the 30-case eval:answer-quality fixture (5 quality metrics + per-intent targeting) — the instrument was previously never run in CI. Dispatch-only, default-off, informational (exit 0, no --targeting-floor): scheduled runs and existing dispatch shapes byte-identical; gates stay owned by eval:quality. Follows the #1003 input-only precedent. | E-1 recon (all $0/read-only) recorded here: canary answer gate samples only 8 of 44 eval:quality cases (ANSWER_CASE_LIMIT default); eval:quality --provider-mode offline is a real $0 harness (provider deleted, deterministic source-only path) usable as an E-3 regression guard; live 30-day answer telemetry (34 answer-path rows): 25 full answers, misses = evidence_gap ×4 (avg 26s spent before gap), provider_incomplete_max_output_tokens ×2 (avg 82s wasted then discarded), provider_generation_failed ×1, retrieval_gap_or_conflict ×1, source_only_no_api ×1; success latency avg 13.1s / p90 25.2s. Headroom classes for E-3: truncation waste (rag.ts:4265 self-heal insufficient live), late evidence-gap detection, p90 latency, thin CI answer coverage. Next: E-2 baseline dispatch (answer_case_limit=44 + answer_quality_eval=true, est $3-8 of user-authorized ≤$20 Phase E envelope) after this merges. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; no provider calls this PR (workflow change only); RAG impact: none — retrieval steps untouched |