fix(ci): run the answer-targeting instrument even when the gate subset fails (E-2 baseline record)#1016
Conversation
…t fails Dispatch #57 (the Phase E-2 baseline) proved the gap: eval:quality went red on two real gates (citation_failure_rate, route_ceiling_failure_count), and GitHub's failure-skip semantics then skipped the informational targeting step - exactly the run where its data was most needed. The step's if: now includes !cancelled() so baseline and paired runs observe red gates too. Ledger row banks the full E-2 baseline numbers from run 29786560936. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
📝 WalkthroughWalkthroughThe canary workflow now prevents answer-quality targeting evaluations from running after cancellation. The review ledger records the Phase E-2 baseline dispatch, gate outcomes, instrumentation fix, spend, and next steps. ChangesCanary evaluation tracking
Estimated code review effort: 2 (Simple) | ~5 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
This pull request has been ignored for the connected project Preview Branches by Supabase. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/eval-canary.yml:
- Line 175: Update the conditional guarding the optional targeting run to
require successful completion of the preflight, setup_node, install, and
project_guard setup steps, while retaining the workflow_dispatch and
answer_quality_eval checks. Replace the broad !cancelled() condition so the live
targeting step cannot run after any required setup failure.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 039e2c6f-0b78-47ca-a799-97393bd3b31f
📒 Files selected for processing (2)
.github/workflows/eval-canary.ymldocs/branch-review-ledger.md
| - name: Answer-quality targeting eval (optional, dispatch-only) | ||
| id: answer_targeting | ||
| if: github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true' | ||
| if: ${{ !cancelled() && github.event_name == 'workflow_dispatch' && github.event.inputs.answer_quality_eval == 'true' }} |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant workflow section with line numbers.
sed -n '130,220p' .github/workflows/eval-canary.yml | cat -n
# Map the workflow structure to locate the step names and surrounding conditions.
python3 - <<'PY'
from pathlib import Path
p = Path('.github/workflows/eval-canary.yml')
text = p.read_text()
for needle in [
'answer_quality_eval',
'golden_retrieval',
'answer_quality',
'preflight',
'install',
'project_guard',
'cancelled()',
]:
print(f"\n## {needle}")
for i, line in enumerate(text.splitlines(), 1):
if needle in line:
print(f"{i}: {line}")
PYRepository: BigSimmo/Database
Length of output: 7041
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the full workflow section around the relevant steps.
sed -n '100,190p' .github/workflows/eval-canary.yml | cat -n
# Show the job-level conditions and any references to the step outcomes later in the file.
sed -n '190,280p' .github/workflows/eval-canary.yml | cat -nRepository: BigSimmo/Database
Length of output: 9589
Gate the optional targeting run on the setup steps.
!cancelled() lets this dispatch-only eval run after earlier setup failures too, not just after answer_quality fails. Require preflight, setup_node, install, and project_guard to succeed before launching the live targeting step.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/eval-canary.yml at line 175, Update the conditional
guarding the optional targeting run to require successful completion of the
preflight, setup_node, install, and project_guard setup steps, while retaining
the workflow_dispatch and answer_quality_eval checks. Replace the broad
!cancelled() condition so the live targeting step cannot run after any required
setup failure.
2 participants
Summary
The Phase E-2 baseline dispatch (#57, run 29786560936,
answer_case_limit=44) worked exactly as intended — and exposed one instrument defect this PR fixes.Baseline findings (banked in the ledger row): golden retrieval held 36/36 in-run;
eval:qualityover the full 44 cases went red on exactly two gates —citation_failure_rate0.0227 (one case whose expected document never surfaced, leaving a 1-citation extractive fallback) androute_ceiling_failure_count2 (one 13.3s pure-retrieval blowout of a 12s extractive budget; one 54ms miss after a provider timeout consumed 22.6s). Grounding, refusal correctness (including both prompt-injection probes), numeric faithfulness, and governance-danger gates all green. Systemic finding: ~9 of 19 generation attempts were discarded by the quality gate with the extractive fallback winning — the primary E-3 target.The fix: the new answer-targeting step was skipped in that run because GitHub skips subsequent steps once one fails, and its
if:didn't override that. A baseline/paired-run instrument must observe red gates, so the condition now includes!cancelled(). Still dispatch-only, default-off, informational (exit 0): scheduled runs remain byte-identical, and a failing gate subset still fails the run — this step can only ever add data, never verdicts.RAG impact: no retrieval behaviour change — workflow step-condition + ledger documentation only.
Verification
npm run check:github-actions— passnpm run check:ci-scope— passnpm run check:gate-manifest— passnpx prettier --checkon both files — cleanRisk and rollout
answer_case_limit(8) +answer_quality_eval=trueto bank the skipped 30-case targeting baseline (~$1–2 of the approved ≤$20 Phase E envelope).Clinical Governance Preflight
Clinical KB Database(sjrfecxgysukkwxsowpy) — no env changes🤖 Generated with Claude Code
https://claude.ai/code/session_01EXsJcLrbZUXwnBeG91cVo9
Generated by Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation