BigSimmo · BigSimmo · Jul 20, 2026 · Jul 20, 2026 · Jul 20, 2026
@@ -33,6 +33,14 @@ on:
         description: "Optional git ref (branch/SHA) to evaluate; empty = default branch"
         required: false
         default: ""
+      # Phase E: opt-in run of the 30-case answer-quality/targeting eval (eval:answer-quality)
+      # alongside the gate subset. Dispatch-only and default-off, so scheduled runs and
+      # existing dispatch shapes are byte-identical. Informational: the script exits 0
+      # without a --targeting-floor, so it can never fail the canary.
+      answer_quality_eval:
+        description: "Also run the 30-case answer-quality/targeting eval (eval:answer-quality); 'true' to enable"
+        required: false
+        default: ""
   schedule:
     # Sunday 18:00 UTC = Monday 02:00 Australia/Perth — off-peak for clinicians.
     - cron: "0 18 * * 0"
@@ -155,6 +163,19 @@ jobs:
           set -o pipefail
           npm run eval:quality -- --rag-only --limit "$ANSWER_CASE_LIMIT" --fail-on-threshold 2>&1 | tee .local/eval-canary/answer-quality.log
 
+      # Phase E baseline/paired-run instrument: the five answer-quality metrics (relevance,
+      # readability, artifact leaks, intent coverage, fail-closed) + per-intent structural
+      # targeting over the 30-case fixture. Informational by design — no --targeting-floor,
+      # exit 0 — so quality gates stay owned by the eval:quality step above; the tee'd log
+      # rides the existing artifact upload.
+      - name: Answer-quality targeting eval (optional, dispatch-only)
+        id: answer_targeting
+        if: github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true'
+        run: |
+          mkdir -p .local/eval-canary
+          set -o pipefail
+          npm run eval:answer-quality 2>&1 | tee .local/eval-canary/answer-targeting.log
+
       # Uploaded on success AND failure: the JSON artifact regenerates the offline ranking
       # snapshot (download it, then `npm run build:ranking-snapshot -- --input <json> --output
       # scripts/fixtures/rag-ranking-candidate-snapshot.v1.json --source-run-id <run-id>`), and

@@ -662,3 +662,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (PR: DB easy wins) | see PR head | User-requested database + process easy-wins pass: read-only Supabase advisor sweep (user-authorized) → canary-liveness probe + advisor-disposition docs; live mutations withheld for per-item confirmation | Advisors (live, read-only): security = 1 INFO (document_title_words RLS-no-policy = the deliberate fail-closed pattern — now comment-documented at the schema block so it is never 'fixed'); performance = ~33 unused-index INFOs + auth connection-strategy note → docs/db-maintenance.md TRIAGE list with retrieval-surface trgm indexes flagged RAG-protected (dropping = full canary protocol), owner-scoped indexes retained for multi-tenant design, operational candidates deferred (negligible benefit at corpus size). Implemented: ci.yml static-pr warn-only eval-canary staleness probe (actions:read, github-script pinned, >8 days → warning; never fails) — needed because #923 (2026-07-19) moved the canary cadence from daily to weekly Sunday 18:00 UTC, where a dropped fire would go unnoticed for a week (the failure-issue step only reacts to runs that happen). CORRECTION (CodeRabbit review on this PR): the initially recorded "2026-07-20 dropped Sunday fire" incident did not occur — 2026-07-20 is a Monday; the Sunday 2026-07-19 slot fired as scheduled run #48 (19:03 UTC, success), and under the weekly cron no 2026-07-20 slot existed. Probe stands as proactive hardening, staleness now measured from updated_at with a finite-timestamp guard. Presented for confirmation (NOT implemented): scheduled telemetry retention (purge:query-logs is owner-scoped + unscheduled; needs owner/window/policy decision), auth percentage connection strategy (dashboard config), any index drops. | check:github-actions PASS; check:ci-scope PASS; check:function-grants 28/28 PASS; docs:check-links 1034 PASS; supabase-schema vitest 66/66; lint+typecheck+prettier clean; Supabase access read-only only |
 | 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from e6cd6cb; PR: telemetry-retention correction) | 01368ff (+0144e69 main merge-in) | User-directed "implement your recommendation for all decisions" close-out: the scheduled-retention recommendation is RETRACTED as founded on a false premise — telemetry retention is ALREADY ACTIVE inside the database via pg_cron; docs corrected, nothing built, no live mutations | Fresh read-only cron.job verification (2026-07-20) matches docs/privacy-impact-assessment.md §6 exactly: jobid 11 purge-expired-rag-queries daily 03:30 (30d), jobid 12 purge-rag-retrieval-logs daily 03:00 (90d), jobid 13 purge-rag-query-misses daily 03:45 (90d), jobid 16 purge-rag-response-cache hourly (bounded 1000); v3 worker jobs present-inactive under backlog auto-toggle (jobid 10); obsolete unbounded cache job absent; audit_logs indefinite by design. db-maintenance.md "open decision" section replaced with the resolved state; purge:query-logs clarified as the MANUAL owner-scoped tool (not the retention mechanism). A GitHub-side weekly deleter would have duplicated pg_cron with window drift (a 90d rag_queries sweep can never out-delete the live 30d job). Remaining decisions stand as documented no-action: auth percentage connection strategy deferred to next instance resize; operational index drops not recommended. | prettier + docs:check-links PASS; Supabase access read-only (single cron.job SELECT, user-authorized read-only envelope); no workflow/schema/config changes |
 | 2026-07-21 | claude/x4-sast-gate (PR #1012) | 0d4985e63 | Maturity X4: blocking SAST gate on the untrusted-document parsing surface | Triage-first per workorder: CI-pinned semgrep/semgrep:1.168.0 over worker/**, src/lib/ingestion*.ts, src/lib/extractors, src/app/api/{ingestion,upload} = 0 ERROR findings (24 TS rules/17 files; 55 Python rules/3 files) — gate starts green with no suppressions. Shipped `semgrep-ingestion-gate` job (no continue-on-error; container digest-pinned to the triage-verified 1.168.0 image) with p/python added for the worker OCR stack; repo-wide advisory job untouched. check-github-action-pins.mjs now enforces both policy halves fail-closed (advisory repo-wide / blocking-and-scoped gate / digest-pinned gate container). Residuals: registry-pack mutability accepted for the narrow surface; making the workflow a branch-protection required check is an operator decision outside this PR. | Exact gate command exit 0 in pinned container; check:github-actions (new assertions verified fail-closed); check:ci-scope; yaml-contract vitest 1/1; verify:cheap 3031 tests green. No provider calls (local Docker only). |
+| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from f33890a; PR: Phase E instrument input) | see PR head + follow-up SHA pin | Phase E kickoff (user-approved with budget): eval-canary gains opt-in `answer_quality_eval` dispatch input running the 30-case eval:answer-quality fixture (5 quality metrics + per-intent targeting) — the instrument was previously never run in CI. Dispatch-only, default-off, informational (exit 0, no --targeting-floor): scheduled runs and existing dispatch shapes byte-identical; gates stay owned by eval:quality. Follows the #1003 input-only precedent. | E-1 recon (all $0/read-only) recorded here: canary answer gate samples only 8 of 44 eval:quality cases (ANSWER_CASE_LIMIT default); eval:quality --provider-mode offline is a real $0 harness (provider deleted, deterministic source-only path) usable as an E-3 regression guard; live 30-day answer telemetry (34 answer-path rows): 25 full answers, misses = evidence_gap ×4 (avg 26s spent before gap), provider_incomplete_max_output_tokens ×2 (avg 82s wasted then discarded), provider_generation_failed ×1, retrieval_gap_or_conflict ×1, source_only_no_api ×1; success latency avg 13.1s / p90 25.2s. Headroom classes for E-3: truncation waste (rag.ts:4265 self-heal insufficient live), late evidence-gap detection, p90 latency, thin CI answer coverage. Next: E-2 baseline dispatch (answer_case_limit=44 + answer_quality_eval=true, est $3-8 of user-authorized ≤$20 Phase E envelope) after this merges. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; no provider calls this PR (workflow change only); RAG impact: none — retrieval steps untouched |