Skip to content

fix(cursor-review): fail loud when a panel model id is delisted#22

Open
mattmillerai wants to merge 1 commit into
matt/bump-kimi-k2.7-codefrom
matt/be-2627-fail-loud-delisted-models
Open

fix(cursor-review): fail loud when a panel model id is delisted#22
mattmillerai wants to merge 1 commit into
matt/bump-kimi-k2.7-codefrom
matt/be-2627-fail-loud-delisted-models

Conversation

@mattmillerai

Copy link
Copy Markdown
Contributor

ELI-5

The review panel pins one exact model name per AI lab. When a lab quietly
retires (delists) one of those names, the panel used to just… skip that model
and say nothing — so we'd run a review with 3 labs instead of 4 for weeks and
never notice. This PR makes that mistake loud: the run now checks every
pinned model against the live catalog before it starts, and fails the whole
run (with a clear "this model id is gone, here's the closest one" message) if
any pin is dead. As a backup, even if that check is bypassed, a dead model now
shows up as (error) in the panel summary instead of a silent "found nothing".

What changed

1. Preflight catalog check (fail-loud, primary). A new preflight job runs
cursor-agent models once, before the 8-cell fan-out, and hard-fails the run if
any pinned panel model id is absent from the live catalog — naming the missing
id(s) plus the closest same-lab hint. This job is also the single source of
truth for the matrix model list (the review matrix now consumes it via
fromJSON), so the list we validate and the list we review can never drift
apart. Fork PRs / runs without CURSOR_API_KEY skip the check cleanly, matching
the gate's existing fork handling (forks already yield should_run=false).

2. Defense-in-depth in extract-findings.py (backup). The per-cell run step
now forwards the cursor-agent exit code + stderr into extract-findings.py. A
cell whose call exited non-zero or whose stderr matches
Cannot use this model: is now classified status=error (which post-review.py
already renders as (error) and counts as a failed cell, firing the existing
degraded-panel path) instead of a silent empty. A non-zero exit that still
produced valid findings is left untouched, so real findings are never discarded.

Testing

  • python3 -m unittest discover -s .github/cursor-review/tests -p 'test_*.py'
    27 tests, all green. New coverage: ClassifyRunErrorTest, ParseExitCodeTest,
    and end-to-end test_delisted_model_is_error_not_empty (the core regression).
  • Workflow YAML validated (parses; the review matrix expands from the preflight
    output).

Notes / judgment calls

  • Stacked on fix(cursor-review): bump dead kimi-k2.5 model id to kimi-k2.7-code #21 (base = matt/bump-kimi-k2.7-code). That PR carries the
    live kimi-k2.7-code pin; this PR is the durability half. GitHub retargets to
    main once fix(cursor-review): bump dead kimi-k2.5 model id to kimi-k2.7-code #21 merges. Review sees only this net diff.
  • Single source of truth over a duplicated list. Rather than hardcode the
    model list a second time in the preflight (a new drift vector — the whole bug
    this fixes), the preflight owns the list and the matrix derives from it.
  • Judge model is out of scope. The preflight validates the 4 matrix pins, not
    the separate judge_model input. Its default equals a panel pin (so it's
    covered transitively), and a delisted judge already degrades via consolidate's
    judge-failed banner rather than silently. A custom, delisted judge_model
    override would not be caught here — noted as a possible follow-up.
  • The weekly scheduled drift-check (auto-diff the matrix vs the catalog and open a
    bump ticket) is intentionally not included — fail-loud on the next PR run is
    enough for now.

Cursor delisting a pinned catalog model id used to degrade the panel
silently: the per-cell `cursor-agent --model <id>` call exits non-zero with
0 bytes of stdout, which the extractor tagged `empty` — indistinguishable
from "the model ran and found nothing" — so the panel ran a lab short for
weeks, undetected.

Two changes make catalog drift fail loud:

- New preflight job runs `cursor-agent models` once before the 8-cell
  fan-out and hard-fails the run if any pinned id is absent from the live
  catalog, naming the missing id(s) and the closest same-lab hint. Fork PRs
  (no CURSOR_API_KEY) skip it, matching the gate's existing fork handling.
  The job is also the single source of truth for the matrix model list, so
  the validated list and the reviewed list cannot drift apart.

- Defense-in-depth in extract-findings.py: the run step now forwards the
  cursor-agent exit code + stderr, and a cell whose call exited non-zero or
  whose stderr matches `Cannot use this model:` is classified `error` (which
  post-review.py reports as `(error)` and the existing degraded-panel path
  fires) instead of a silent `empty`.
@mattmillerai mattmillerai added cursor-review Multi-model cursor review agent-coded Authored by the agent-work loop labels Jul 7, 2026
@mattmillerai mattmillerai marked this pull request as ready for review July 7, 2026 21:34
@coderabbitai

coderabbitai Bot commented Jul 7, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9bdd3d6c-c1bd-472b-9843-f8ae08e49282

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch matt/be-2627-fail-loud-delisted-models
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch matt/be-2627-fail-loud-delisted-models

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant