Fix single-run canary trend classification

[BigSimmo]

Summary

• reserve repeated_* and variability classifications for comparisons containing at least two structured reports
• label one-report content/latency failures as observed, and a clean one-report case as single_run
• add focused regression coverage and exercise the merged-main canary artifact

Why

The first structured artifact from run 30018289898 exposed that equality over a one-item set made a single failure look repeated. That overstates evidence and could misdirect RAG triage.

Verification

• npx vitest run tests/eval-trend.test.ts --reporter=dot — 6/6 passed
• npx eslint scripts/eval-trend.mjs tests/eval-trend.test.ts --max-warnings 0 — passed
• npm run eval:trend -- --answer-quality <run-30018289898-report.json> — admission comparison and neuroleptic classified observed_content_failure; admission paraphrase classified observed_latency_failure
• git diff --check — passed

Risk and rollback

Offline diagnostic labels only. No eval threshold, workflow, retrieval, ranking, answer, or provider behavior changes. Rollback is a plain revert.

RAG impact: no retrieval behaviour change — offline trend classification only.

Checks not run

• no new provider run; this fix uses the already-downloaded single approved artifact
• full verify:cheap not repeated for this 28-line follow-up; focused tests, lint, and hosted required checks cover the changed surface

---

Note

Low Risk
Offline diagnostic labels in eval-trend only; no retrieval, answer, provider, or eval threshold behavior changes.

Overview
Fixes offline answer-quality trend labels when only one structured canary report is compared. A single failing run was previously labeled repeated_content_failure because identical fingerprints on a one-item set looked “repeated,” which overstated evidence for RAG triage.

Single-report cases now get observed_content_failure, observed_latency_failure, or single_run instead of variability/repeated buckets. Multi-report logic is unchanged except that repeated_content_failure, provider_route_variability, and latency_variability require matches.length > 1.

Adds regression tests so one-run failures are not called repeated and a clean one-run case is single_run.

^{Reviewed by Cursor Bugbot for commit 68bce0a. Configure here.}

[supabase]

This pull request has been ignored for the connected project sjrfecxgysukkwxsowpy because there are no changes detected in supabase directory. You can change this behaviour in Project Integrations Settings ↗︎.

---

Preview Branches by Supabase.
Learn more about Supabase Branching ↗︎.

[coderabbitai]

Warning

Review limit reached

@BigSimmo, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 4 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: bae356ff-2e91-49ca-937f-258fdb89b5e1

📥 Commits

Reviewing files that changed from the base of the PR and between 478b513 and 68bce0a.

📒 Files selected for processing (2)

• scripts/eval-trend.mjs
• tests/eval-trend.test.ts

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/rag-canary-single-run

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}