Skip to content

fix(eval): calibrate canary latency gates to the runner's measurement context #624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
BigSimmo merged 2 commits into from
Jul 13, 2026
Merged

fix(eval): calibrate canary latency gates to the runner's measurement context #624

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
91363de
fix(eval): calibrate canary latency gates to the runner's measurement…
BigSimmo Jul 13, 2026
0e45aa7
fix(eval): scope the 60s latency allowance to the canary runner context
BigSimmo Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/eval-canary.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,9 @@ jobs:
- name: Answer-quality subset (live generation)
env:
ANSWER_CASE_LIMIT: ${{ github.event.inputs.answer_case_limit || '8' }}
# Canary-only latency allowance: cross-region runner -> Sydney Supabase
# + OpenAI. Release/local eval:quality keeps the strict default gates.
EVAL_LATENCY_CONTEXT: cross-region-runner
run: |
if [[ ! "$ANSWER_CASE_LIMIT" =~ ^[0-9]+$ ]] || (( ANSWER_CASE_LIMIT < 1 || ANSWER_CASE_LIMIT > 100 )); then
echo "answer_case_limit must be an integer from 1 to 100" >&2
Expand Down
15 changes: 13 additions & 2 deletions scripts/eval-quality.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ export function deliveredGroundedAfterSourceGovernancePolicy(
return answer.grounded;
}

const crossRegionRunnerLatencyContext = process.env.EVAL_LATENCY_CONTEXT === "cross-region-runner";

const qualityThresholds = {
retrievalTopKHitRate: 0.8,
retrievalDocumentRecallAt5: 0.8,
Expand All @@ -137,10 +139,19 @@ const qualityThresholds = {
numericGroundingFailureRate: 0,
staleTopResultRate: 0.25,
reviewRequiredTopResultRate: 0.25,
ragP95LatencyMs: 25_000,
// Latency gates default to the strict near-region ceilings that release and
// local runs are held to. The Eval Canary measures from cross-region GitHub
// runners → Sydney Supabase + OpenAI, where a real grounded answer pays full
// generation time and the provider-timeout→extractive-fallback chain costs
// 30s+ per affected case (issue #459 post-#606: quality metrics perfect, p95
// measured 48,256ms) — that workflow opts into the wider allowance by setting
// EVAL_LATENCY_CONTEXT=cross-region-runner, so eval:quality:release keeps the
// strict gate. User-facing latency is enforced separately by the answer SLO
// deep probe, not this eval.
ragP95LatencyMs: crossRegionRunnerLatencyContext ? 60_000 : 25_000,
ragRouteP95LatencyMs: {
unsupported: 4_000,
extractive: 12_000,
extractive: crossRegionRunnerLatencyContext ? 60_000 : 12_000,

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 13, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Widen the route gate used by fallback latency

When EVAL_LATENCY_CONTEXT=cross-region-runner and a generation attempt times out but recovers with an extractive fallback, this route ceiling is not the one the eval applies: latencyRouteForAnswer charges any answer with generation_latency_ms > 0 to the attempted fast/strong route, and the existing eval-quality test explicitly covers an extractive fallback reported as fast. Because only the extractive ceiling is widened here, a 30s+ provider-timeout fallback can still fail the canary under the unchanged 25s/35s attempted-route gates, which leaves the cited failure mode unfixed for cross-region runs.

Useful? React with 👍 / 👎.

fast: 25_000,
strong: 35_000,
} as Record<string, number>,
Expand Down