From 3d1cac1b33a7f3542f4f77d0c057f34311489c29 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 20 Jul 2026 23:46:02 +0000 Subject: [PATCH] fix(ci): run the answer-targeting instrument even when the gate subset fails Dispatch #57 (the Phase E-2 baseline) proved the gap: eval:quality went red on two real gates (citation_failure_rate, route_ceiling_failure_count), and GitHub's failure-skip semantics then skipped the informational targeting step - exactly the run where its data was most needed. The step's if: now includes !cancelled() so baseline and paired runs observe red gates too. Ledger row banks the full E-2 baseline numbers from run 29786560936. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9 --- .github/workflows/eval-canary.yml | 6 ++++-- docs/branch-review-ledger.md | 1 + 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/eval-canary.yml b/.github/workflows/eval-canary.yml index 3b085f9ab..3a00b2303 100644 --- a/.github/workflows/eval-canary.yml +++ b/.github/workflows/eval-canary.yml @@ -167,10 +167,12 @@ jobs: # readability, artifact leaks, intent coverage, fail-closed) + per-intent structural # targeting over the 30-case fixture. Informational by design — no --targeting-floor, # exit 0 — so quality gates stay owned by the eval:quality step above; the tee'd log - # rides the existing artifact upload. + # rides the existing artifact upload. `!cancelled()` keeps the instrument running when + # the gate subset above fails: a baseline/paired run must observe red gates too + # (proven necessary on dispatch #57, where a red eval:quality skipped this step). - name: Answer-quality targeting eval (optional, dispatch-only) id: answer_targeting - if: github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true' + if: ${{ !cancelled() && github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true' }} run: | mkdir -p .local/eval-canary set -o pipefail diff --git a/docs/branch-review-ledger.md b/docs/branch-review-ledger.md index 0b9477ab1..b7cab71fd 100644 --- a/docs/branch-review-ledger.md +++ b/docs/branch-review-ledger.md @@ -663,3 +663,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from e6cd6cb; PR: telemetry-retention correction) | 01368ff (+0144e69 main merge-in) | User-directed "implement your recommendation for all decisions" close-out: the scheduled-retention recommendation is RETRACTED as founded on a false premise — telemetry retention is ALREADY ACTIVE inside the database via pg_cron; docs corrected, nothing built, no live mutations | Fresh read-only cron.job verification (2026-07-20) matches docs/privacy-impact-assessment.md §6 exactly: jobid 11 purge-expired-rag-queries daily 03:30 (30d), jobid 12 purge-rag-retrieval-logs daily 03:00 (90d), jobid 13 purge-rag-query-misses daily 03:45 (90d), jobid 16 purge-rag-response-cache hourly (bounded 1000); v3 worker jobs present-inactive under backlog auto-toggle (jobid 10); obsolete unbounded cache job absent; audit_logs indefinite by design. db-maintenance.md "open decision" section replaced with the resolved state; purge:query-logs clarified as the MANUAL owner-scoped tool (not the retention mechanism). A GitHub-side weekly deleter would have duplicated pg_cron with window drift (a 90d rag_queries sweep can never out-delete the live 30d job). Remaining decisions stand as documented no-action: auth percentage connection strategy deferred to next instance resize; operational index drops not recommended. | prettier + docs:check-links PASS; Supabase access read-only (single cron.job SELECT, user-authorized read-only envelope); no workflow/schema/config changes | | 2026-07-21 | claude/x4-sast-gate (PR #1012) | 0d4985e63 | Maturity X4: blocking SAST gate on the untrusted-document parsing surface | Triage-first per workorder: CI-pinned semgrep/semgrep:1.168.0 over worker/**, src/lib/ingestion*.ts, src/lib/extractors, src/app/api/{ingestion,upload} = 0 ERROR findings (24 TS rules/17 files; 55 Python rules/3 files) — gate starts green with no suppressions. Shipped `semgrep-ingestion-gate` job (no continue-on-error; container digest-pinned to the triage-verified 1.168.0 image) with p/python added for the worker OCR stack; repo-wide advisory job untouched. check-github-action-pins.mjs now enforces both policy halves fail-closed (advisory repo-wide / blocking-and-scoped gate / digest-pinned gate container). Residuals: registry-pack mutability accepted for the narrow surface; making the workflow a branch-protection required check is an operator decision outside this PR. | Exact gate command exit 0 in pinned container; check:github-actions (new assertions verified fail-closed); check:ci-scope; yaml-contract vitest 1/1; verify:cheap 3031 tests green. No provider calls (local Docker only). | | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from f33890a; PR: Phase E instrument input) | see PR head + follow-up SHA pin | Phase E kickoff (user-approved with budget): eval-canary gains opt-in `answer_quality_eval` dispatch input running the 30-case eval:answer-quality fixture (5 quality metrics + per-intent targeting) — the instrument was previously never run in CI. Dispatch-only, default-off, informational (exit 0, no --targeting-floor): scheduled runs and existing dispatch shapes byte-identical; gates stay owned by eval:quality. Follows the #1003 input-only precedent. | E-1 recon (all $0/read-only) recorded here: canary answer gate samples only 8 of 44 eval:quality cases (ANSWER_CASE_LIMIT default); eval:quality --provider-mode offline is a real $0 harness (provider deleted, deterministic source-only path) usable as an E-3 regression guard; live 30-day answer telemetry (34 answer-path rows): 25 full answers, misses = evidence_gap ×4 (avg 26s spent before gap), provider_incomplete_max_output_tokens ×2 (avg 82s wasted then discarded), provider_generation_failed ×1, retrieval_gap_or_conflict ×1, source_only_no_api ×1; success latency avg 13.1s / p90 25.2s. Headroom classes for E-3: truncation waste (rag.ts:4265 self-heal insufficient live), late evidence-gap detection, p90 latency, thin CI answer coverage. Next: E-2 baseline dispatch (answer_case_limit=44 + answer_quality_eval=true, est $3-8 of user-authorized ≤$20 Phase E envelope) after this merges. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; no provider calls this PR (workflow change only); RAG impact: none — retrieval steps untouched | +| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from 5b4098d; PR: E-2 baseline record + instrument fix) | see PR head | ADDENDUM Phase E-2 BASELINE BANKED (canary dispatch #57, run 29786560936, main 5b4098d, answer_case_limit=44): golden retrieval 36/36 green in-run (no-regression net held); eval:quality full-44 RED on exactly two gates — citation_failure_rate 0.0227 (1/44: neuroleptic-side-effect-escalation — expected doc never retrieved, generation quality-failed, extractive fallback with 1 citation) and route_ceiling_failure_count 2 (clozapine-anc-withhold-threshold: 13.3s pure retrieval vs 12s extractive budget, RPC 9.3s, zero generation; agitation-arousal-typo-dosing: 25054ms vs 25000ms after provider_timeout ate 22.6s pre-recovery). Green gates: grounded_supported 1.0, unsupported_correct 1.0 (all 14 refusals incl. both prompt-injection probes at 2ms), numeric grounding failures 0, governance danger 0, p95 17.4s. Non-blocking signals: expected_source_hit 0.6136 (both admission-discharge cases miss MHSP.AdmissionCommunityPts.pdf to sibling NMHS/RKPG policies — labeling-vs-ranking question, §3.1 class), source governance warning rate 0.8182 (metadata debt, waivable class). SYSTEMIC E-3 TARGET: ~9 of 19 generation attempts discarded (fast output fails quality gate → extractive fallback wins; ~7 cases carry fast_quality_retry_strong→extractive reasons) = ~half of generation latency+spend wasted. Instrument defect found+fixed this PR: targeting step was skipped after the red gate (GitHub failure-skip semantics) → if: gains !cancelled() so baselines observe red gates. Spend: est ~$1-2 actual (19 OpenAI-request cases; cost rates unset in CI so report shows n/a) of ≤$20 Phase E envelope. Next: cheap re-dispatch (default limit 8 + answer_quality_eval=true) to bank the skipped 30-case targeting baseline. | Gates this PR: check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean. Baseline evidence: job log run 29786560936 (5 failing-case diagnostics + Answer Metrics table read in full) |