feat(ci): weekly canary guards the full 44-case answer set (I9)

.github/workflows/eval-canary.yml

@@ -17,7 +17,7 @@ on:
       answer_case_limit:
         description: "Number of answer-quality cases to run (--limit)"
         required: false
-        default: "8"
+        default: "44"
       # Staged ranking-weight evaluation (ADDENDUM 4 Phase B): a RAG_RANKING_CONFIG JSON
       # override evaluated live WITHOUT touching code defaults. Empty = production weights.
       # Malformed JSON fails the run loudly (validation step below) instead of silently
@@ -158,7 +158,7 @@ jobs:
       - name: Answer-quality subset (live generation)
         id: answer_quality
         env:
-          ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '8' }}
+          ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '44' }}
           # Canary-only latency allowance: cross-region runner -> Sydney Supabase
           # + OpenAI. Release/local eval:quality keeps the strict default gates.
           EVAL_LATENCY_CONTEXT: cross-region-runner

docs/branch-review-ledger.md

@@ -678,3 +678,5 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (parity commit review) | 1aebf02 (fix landed a3b9a54) | rag-retrieval-reviewer on the monitoring evidence-gate parity commit: REQUEST-CHANGES (soft) — P2 reproduced: inflected monitoring kind tokens (monitor\w*/annual(?:ly)?/blood tests?/ecgs?/lfts?) steal sole-dose-value sentences from the dose arm; dose-intent answers then reject the monitoring-kind fact ("Quetiapine is monitored at a dose of 200 mg daily" flipped grounded true→false, source-gap — fails CLOSED, never a wrong dose). P3: monitoring figure escape lacked the dose escape's multi-drug bare-row guard. Clean: over-admission bounded (broad vocab lives in gate/filter only, promotion still corpus-guarded, claim-support unchanged); regex cost negligible; mismatched-unit test relaxation legitimate (synopsis is corpus-verbatim; weeks pin enforced by atom identity + adjacent_context exclusion from both gate and claim corpora). | BOTH FINDINGS FIXED in a3b9a54: kind arm classifies legacy tokens byte-identically and new-inflection-only sentences fall through to the dose arm when they carry a clinicalDoseValuePattern value (both repro sentences pinned as dose-intent tests); multi-drug bare-row guard extended to monitoring_schedule with a discriminating test — red-proven both directions. Reviewer checks: formatting 38/38, focused 644/644; targeting eval deferred to the wave's live canary pair. |
 | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (parity commit review) | 1aebf02 (fix landed a3b9a54) | clinical-governance-reviewer on the same commit: APPROVE-WITH-NITS. P2 (independently converged with the retrieval reviewer's P3): monitoring figure-escape lacked the dose-path multi-drug cross-entity guard — a bare wrong-drug schedule/level row in a multi-drug chunk could be entity-prefixed for a named-drug monitoring query; downstream gates verify text-vs-source presence, never attribution (worked lithium/valproate LFT path traced through finalize). FIXED in a3b9a54 exactly as its smallest-fix prescribed (guard at the :872-881 site now fires for monitoring_schedule; negative multi-drug test added, red-proven). Clean: unsupported figures impossible (admission-only change; promotion corpus guard + numeric verification + claim support all byte-unchanged); conservative failure intact (figure-bearing-only escape, schedule-free refusal pinned); adjacent-context safety held (sourceEvidenceText excludes adjacent_context; weeks refusal confirmed by probe); no PHI/provider/ranking surface. P3s: RAG impact line (present in the PR body — behaviour-change form, correct since the PR also carries the Option A retrieval change); multi-drug negative test (landed in a3b9a54). | Offline guard-chain trace + targeted vitest probes (named-drug guard, bare-figure admission, conservative gap, adjacent refusal). Provider/release gates deferred per confirmation boundary. |
 | 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (Option A rescue review) | 0abf3c9 | clinical-governance-reviewer on the S3 escalation rescue: APPROVE-WITH-NITS, no P0/P1. P2 = the mandated live canary pair itself (process gate, declared in the PR body; offline-green + review-approved proven insufficient for this surface 2026-07-20). P3s: multi-drug escalation-query recall edge (titled drug + untitled drug — fast-path return can skip the vector leg; recall limitation, not misattribution, mirrors the pre-existing allowlisted-class tradeoff); reviewer probe files must stay uncommitted (relocated to scratchpad). All six clinical concerns verified safe: wrong-document impossible (alias phrases must appear in the query; per-document alias groups, no cross-drug conflation), conservative availability (S3 purely additive via keyed-union merge; sibling retention test-pinned), live expansion acceptable (title-named correct-entity SOP in every firing shape), fail-closed double layer (adversarial short-circuit precedes the predicate; injection-forge case intent=protocol cannot fire — executed), governance metadata unbypassed (same attachDocumentRankingMetadata + status=indexed + access-scope filters), no PHI/provider/schema surface. | Reviewer checks: escalation-rescue suite 8/8, injection-forge intent derivation executed, static trace of the full S3 chain. Live canary pair = the trust gate, dispatched post-merge. |
+| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (Option A wave verdict — no code change; #1040 merged as cde6c5c) | canary run 29827012719 (#61, main cde6c5c) vs banked #60 (29800029819) | OPTION A WAVE ADOPTED — FIRST FULLY-GREEN 44-CASE CANARY IN PROGRAM HISTORY (Blocking failures: None). (1) Option A payoff EXCEEDED: citation_failure_rate 0.0227→0; the neuroleptic-side-effect-escalation case flipped from wrong-doc→failed-generation→1-citation-fallback to **strong route, successful gpt-5.6-sol generation, passed in 15.4s with no fallback marker** — the rescued S3 retrieval fixed generation itself, not just the citation count; expected_source_hit 0.6364→0.6591. (2) Golden held exactly as the blast-radius analysis promised: 36/36 PASS, content_recall 1.0, mrr@10 0.8921 BYTE-IDENTICAL to the pre-wave baseline (zero ordering movement — no golden case fires the predicate), irrelevant@10 0.1083→0.0917 (slightly better). (3) Parity payoff PARTIAL: monitoring targeting 1/5→2/5 (olanzapine-lai flipped — previously called a retrieval-depth residual; quetiapine-dose also flipped on the dose side); lithium-range (232ch) + metabolic (73ch, byte-identical answer to #60) did NOT flip despite offline-proven fixes — their live chunk sets evidently contain no admissible schedule sentence even under the widened gate → reclassified as retrieval-depth/live-content residuals joining adhd; below the ≥3/5 target but strictly improved, no regression anywhere. Dose 2/5 vs 2/4: same passing count, applicable set grew (new quality-metformin-renal-dosing miss = eval-set churn, not regression). (4) No-worse EXCEEDED: relevance 0.5333→0.6 (the two-step watch-item slide FULLY REVERSED to the #58 level), targeting_rate 0.6667→0.6957, fail_closed 0.9 held, readability/artifact_leaks 1.0, route ceilings 0, grounded 1.0, unsupported_correct 1.0, numeric 0, p95 22.8s, red_result 3/3. Adoption per the measured-gain rule: primary goal achieved, three case flips, relevance recovered, zero regressions. Residual queue: monitoring retrieval-depth trio (lithium-range/metabolic/adhd), E-3d H2 discards, weekly ANSWER_CASE_LIMIT 8→44 raise now unblocked (gate would be green), comparison-class coverage. Wave spend +~$2-4 → Phase E + Option A total ~$12-20 of ≤$20. | Evidence: run #61 job log read (Threshold Status: None; Answer Metrics; neuroleptic diagnostics row; targeting metric_rates + 6-miss list; golden 36 PASS lines + summary). Revert drill NOT triggered. |
+| 2026-07-21 | claude/clinical-kb-pwa-review-asi3wb (PR: I9 weekly coverage raise) | see PR head | ADDENDUM 5 post-green item I9 (plan-authorized "after reds fixed"): weekly scheduled canary ANSWER_CASE_LIMIT default 8→44 — the Sunday 18:00 UTC cron now guards the FULL answer-quality case set instead of the first 8 (both #57 blocking reds historically lived OUTSIDE the first 8, leaving the weekly gate blind to them). Unblocked by run #61 proving the citation gate green on the full 44. Cost: est +$1-2/week (user-authorized in the plan). Contract test pin updated in lockstep (eval-canary-workflow.test.ts). Dispatch shapes unchanged (input override still wins); operational-risk diff, plain-revert rollback. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS (20/20); eval-canary-workflow contract 4/4; prettier clean; no provider calls |

tests/eval-canary-workflow.test.ts

@@ -11,7 +11,7 @@ describe("eval canary workflow input", () => {
   });
 
   it("validates the dispatch limit outside shell source and passes it as one quoted argument", () => {
-    expect(workflow).toContain("ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '8' }}");
+    expect(workflow).toContain("ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '44' }}");
     expect(workflow).toContain('[[ ! "$ANSWER_CASE_LIMIT" =~ ^[0-9]+$ ]]');
     expect(workflow).toContain("ANSWER_CASE_LIMIT < 1 || ANSWER_CASE_LIMIT > 100");
     expect(workflow).toContain('--limit "$ANSWER_CASE_LIMIT"');