Skip to content

fix(ci): run the answer-targeting instrument even when the gate subset fails (E-2 baseline record) #1016

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 1 commit into from
Jul 20, 2026
+5 −2
Merged

fix(ci): run the answer-targeting instrument even when the gate subset fails (E-2 baseline record) #1016

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/eval-canary.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,10 +167,12 @@ jobs:
# readability, artifact leaks, intent coverage, fail-closed) + per-intent structural
# targeting over the 30-case fixture. Informational by design — no --targeting-floor,
# exit 0 — so quality gates stay owned by the eval:quality step above; the tee'd log
# rides the existing artifact upload.
# rides the existing artifact upload. `!cancelled()` keeps the instrument running when
# the gate subset above fails: a baseline/paired run must observe red gates too
# (proven necessary on dispatch #57, where a red eval:quality skipped this step).
- name: Answer-quality targeting eval (optional, dispatch-only)
id: answer_targeting
if: github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true'
if: ${{ !cancelled() && github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true' }}

@coderabbitai coderabbitai Bot Jul 20, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the relevant workflow section with line numbers.
sed -n '130,220p' .github/workflows/eval-canary.yml | cat -n

# Map the workflow structure to locate the step names and surrounding conditions.
python3 - <<'PY'
from pathlib import Path
p = Path('.github/workflows/eval-canary.yml')
text = p.read_text()
for needle in [
    'answer_quality_eval',
    'golden_retrieval',
    'answer_quality',
    'preflight',
    'install',
    'project_guard',
    'cancelled()',
]:
    print(f"\n## {needle}")
    for i, line in enumerate(text.splitlines(), 1):
        if needle in line:
            print(f"{i}: {line}")
PY

Repository: BigSimmo/Database

Length of output: 7041

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Show the full workflow section around the relevant steps.
sed -n '100,190p' .github/workflows/eval-canary.yml | cat -n

# Show the job-level conditions and any references to the step outcomes later in the file.
sed -n '190,280p' .github/workflows/eval-canary.yml | cat -n

Repository: BigSimmo/Database

Length of output: 9589

Gate the optional targeting run on the setup steps.

!cancelled() lets this dispatch-only eval run after earlier setup failures too, not just after answer_quality fails. Require preflight, setup_node, install, and project_guard to succeed before launching the live targeting step.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/eval-canary.yml at line 175, Update the conditional
guarding the optional targeting run to require successful completion of the
preflight, setup_node, install, and project_guard setup steps, while retaining
the workflow_dispatch and answer_quality_eval checks. Replace the broad
!cancelled() condition so the live targeting step cannot run after any required
setup failure.

run: |
mkdir -p .local/eval-canary
set -o pipefail
Expand Down
1 change: 1 addition & 0 deletions docs/branch-review-ledger.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -663,3 +663,4 @@ Use this ledger to prevent repeated branch and PR reviews when the reviewed HEAD
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from e6cd6cb; PR: telemetry-retention correction) | 01368ff (+0144e69 main merge-in) | User-directed "implement your recommendation for all decisions" close-out: the scheduled-retention recommendation is RETRACTED as founded on a false premise — telemetry retention is ALREADY ACTIVE inside the database via pg_cron; docs corrected, nothing built, no live mutations | Fresh read-only cron.job verification (2026-07-20) matches docs/privacy-impact-assessment.md §6 exactly: jobid 11 purge-expired-rag-queries daily 03:30 (30d), jobid 12 purge-rag-retrieval-logs daily 03:00 (90d), jobid 13 purge-rag-query-misses daily 03:45 (90d), jobid 16 purge-rag-response-cache hourly (bounded 1000); v3 worker jobs present-inactive under backlog auto-toggle (jobid 10); obsolete unbounded cache job absent; audit_logs indefinite by design. db-maintenance.md "open decision" section replaced with the resolved state; purge:query-logs clarified as the MANUAL owner-scoped tool (not the retention mechanism). A GitHub-side weekly deleter would have duplicated pg_cron with window drift (a 90d rag_queries sweep can never out-delete the live 30d job). Remaining decisions stand as documented no-action: auth percentage connection strategy deferred to next instance resize; operational index drops not recommended. | prettier + docs:check-links PASS; Supabase access read-only (single cron.job SELECT, user-authorized read-only envelope); no workflow/schema/config changes |
| 2026-07-21 | claude/x4-sast-gate (PR #1012) | 0d4985e63 | Maturity X4: blocking SAST gate on the untrusted-document parsing surface | Triage-first per workorder: CI-pinned semgrep/semgrep:1.168.0 over worker/**, src/lib/ingestion*.ts, src/lib/extractors, src/app/api/{ingestion,upload} = 0 ERROR findings (24 TS rules/17 files; 55 Python rules/3 files) — gate starts green with no suppressions. Shipped `semgrep-ingestion-gate` job (no continue-on-error; container digest-pinned to the triage-verified 1.168.0 image) with p/python added for the worker OCR stack; repo-wide advisory job untouched. check-github-action-pins.mjs now enforces both policy halves fail-closed (advisory repo-wide / blocking-and-scoped gate / digest-pinned gate container). Residuals: registry-pack mutability accepted for the narrow surface; making the workflow a branch-protection required check is an operator decision outside this PR. | Exact gate command exit 0 in pinned container; check:github-actions (new assertions verified fail-closed); check:ci-scope; yaml-contract vitest 1/1; verify:cheap 3031 tests green. No provider calls (local Docker only). |
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from f33890a; PR: Phase E instrument input) | see PR head + follow-up SHA pin | Phase E kickoff (user-approved with budget): eval-canary gains opt-in `answer_quality_eval` dispatch input running the 30-case eval:answer-quality fixture (5 quality metrics + per-intent targeting) — the instrument was previously never run in CI. Dispatch-only, default-off, informational (exit 0, no --targeting-floor): scheduled runs and existing dispatch shapes byte-identical; gates stay owned by eval:quality. Follows the #1003 input-only precedent. | E-1 recon (all $0/read-only) recorded here: canary answer gate samples only 8 of 44 eval:quality cases (ANSWER_CASE_LIMIT default); eval:quality --provider-mode offline is a real $0 harness (provider deleted, deterministic source-only path) usable as an E-3 regression guard; live 30-day answer telemetry (34 answer-path rows): 25 full answers, misses = evidence_gap ×4 (avg 26s spent before gap), provider_incomplete_max_output_tokens ×2 (avg 82s wasted then discarded), provider_generation_failed ×1, retrieval_gap_or_conflict ×1, source_only_no_api ×1; success latency avg 13.1s / p90 25.2s. Headroom classes for E-3: truncation waste (rag.ts:4265 self-heal insufficient live), late evidence-gap detection, p90 latency, thin CI answer coverage. Next: E-2 baseline dispatch (answer_case_limit=44 + answer_quality_eval=true, est $3-8 of user-authorized ≤$20 Phase E envelope) after this merges. | check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean; no provider calls this PR (workflow change only); RAG impact: none — retrieval steps untouched |
| 2026-07-20 | claude/clinical-kb-pwa-review-asi3wb (restarted from 5b4098d; PR: E-2 baseline record + instrument fix) | see PR head | ADDENDUM Phase E-2 BASELINE BANKED (canary dispatch #57, run 29786560936, main 5b4098d, answer_case_limit=44): golden retrieval 36/36 green in-run (no-regression net held); eval:quality full-44 RED on exactly two gates — citation_failure_rate 0.0227 (1/44: neuroleptic-side-effect-escalation — expected doc never retrieved, generation quality-failed, extractive fallback with 1 citation) and route_ceiling_failure_count 2 (clozapine-anc-withhold-threshold: 13.3s pure retrieval vs 12s extractive budget, RPC 9.3s, zero generation; agitation-arousal-typo-dosing: 25054ms vs 25000ms after provider_timeout ate 22.6s pre-recovery). Green gates: grounded_supported 1.0, unsupported_correct 1.0 (all 14 refusals incl. both prompt-injection probes at 2ms), numeric grounding failures 0, governance danger 0, p95 17.4s. Non-blocking signals: expected_source_hit 0.6136 (both admission-discharge cases miss MHSP.AdmissionCommunityPts.pdf to sibling NMHS/RKPG policies — labeling-vs-ranking question, §3.1 class), source governance warning rate 0.8182 (metadata debt, waivable class). SYSTEMIC E-3 TARGET: ~9 of 19 generation attempts discarded (fast output fails quality gate → extractive fallback wins; ~7 cases carry fast_quality_retry_strong→extractive reasons) = ~half of generation latency+spend wasted. Instrument defect found+fixed this PR: targeting step was skipped after the red gate (GitHub failure-skip semantics) → if: gains !cancelled() so baselines observe red gates. Spend: est ~$1-2 actual (19 OpenAI-request cases; cost rates unset in CI so report shows n/a) of ≤$20 Phase E envelope. Next: cheap re-dispatch (default limit 8 + answer_quality_eval=true) to bank the skipped 30-case targeting baseline. | Gates this PR: check:github-actions PASS; check:ci-scope PASS; check:gate-manifest PASS; prettier clean. Baseline evidence: job log run 29786560936 (5 failing-case diagnostics + Answer Metrics table read in full) |
Loading