Skip to content

fix(solve): non-mutating dry-run and fail loudly on unmatched provider filter#766

Merged
lewisjared merged 5 commits into
mainfrom
feat/cli-solve-correctness
Jun 29, 2026
Merged

fix(solve): non-mutating dry-run and fail loudly on unmatched provider filter#766
lewisjared merged 5 commits into
mainfrom
feat/cli-solve-correctness

Conversation

@lewisjared

@lewisjared lewisjared commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Description

Fixing some oddities about ref solve

--dry-run is now genuinely non-mutating. Previously ref solve --dry-run still created ExecutionGroup rows (via get_or_create), reaped stale in-progress executions, and built/joined an executor. It now performs a read-only preview: it looks groups up without creating them, skips the stale-execution sweep, and never builds an executor. A group that does not yet exist is reported as one the solve would create. A regression test asserts the execution-group row count is unchanged by a dry run, and that a subsequent real solve over the same catalog does create groups.

Provider filters now fail loudly. ref solve --provider <name> previously exited 0 with no work done when no providers were configured or the filter matched none of them, which looks like success. The command now validates before solving and exits non-zero with remediation (install a provider, regenerate config, or add it under [[diagnostic_providers]]). Matching mirrors the solver: a filter matches when it is a case-insensitive substring of a configured provider slug.

Checklist

Please confirm that this pull request has done the following:

  • Tests added
  • Documentation added (where applicable)
  • Changelog item added to changelog/

Summary by CodeRabbit

Summary by CodeRabbit

  • Bug Fixes

    • ref solve now fails with error + remediation when no diagnostic providers are configured, or when --provider matches none of the configured providers.
    • ref solve --dry-run is now strictly read-only: it no longer creates execution groups or triggers execution steps, and it won’t run or wait on executors.
    • Dry-run scheduling now uses a stale cutoff so stale in-progress executions are treated as runnable (and can rerun when enabled).
  • Tests

    • Updated and expanded unit tests for provider-filter validation and dry-run persistence/scheduling behaviour.

…r filter

- `ref solve --dry-run` is now a read-only preview: it no longer creates
  execution groups, reaps stale executions, or builds an executor
- `ref solve` now fails with remediation when no diagnostic providers are
  configured, or when a --provider filter matches none of them, instead of
  exiting 0 having done nothing
@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6e1a04a1-54f1-4184-9f32-320a795984b5

📥 Commits

Reviewing files that changed from the base of the PR and between 264fc8d and cfb698c.

📒 Files selected for processing (5)
  • packages/climate-ref/src/climate_ref/cli/solve.py
  • packages/climate-ref/src/climate_ref/models/execution.py
  • packages/climate-ref/src/climate_ref/solver.py
  • packages/climate-ref/tests/unit/cli/test_solve.py
  • packages/climate-ref/tests/unit/models/test_metric_execution.py
🚧 Files skipped from review as they are similar to previous changes (4)
  • packages/climate-ref/src/climate_ref/cli/solve.py
  • packages/climate-ref/tests/unit/models/test_metric_execution.py
  • packages/climate-ref/src/climate_ref/solver.py
  • packages/climate-ref/tests/unit/cli/test_solve.py

📝 Walkthrough

Walkthrough

ref solve now fails fast when provider configuration is missing or unmatched, and --dry-run remains read-only while still allowing stale in-progress executions to be treated as runnable.

Changes

Provider Validation and Dry-Run Execution Flow

Layer / File(s) Summary
CLI provider filter validation
packages/climate-ref/src/climate_ref/cli/solve.py, packages/climate-ref/tests/unit/cli/test_solve.py
Adds _validate_provider_filter, wires it into solve, and updates CLI coverage for configured providers, case-insensitive matches, and failure exits.
ExecutionGroup stale cutoff handling
packages/climate-ref/src/climate_ref/models/execution.py, packages/climate-ref/tests/unit/models/test_metric_execution.py
Extends ExecutionGroup.should_run with stale_cutoff and updates model tests for stale and fresh in-progress execution cases.
Solver dry-run read-only changes
packages/climate-ref/src/climate_ref/solver.py, packages/climate-ref/tests/unit/test_solver.py
Computes a stale cutoff, skips stale reaping and executor creation in dry-run, avoids creating execution groups during dry-run, and updates solver coverage for execution-group persistence.
Changelog entry
changelog/766.fix.md
Documents the dry-run and provider-validation behaviour changes.
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 52.63% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and clearly describes the main changes to ref solve.
Description check ✅ Passed The description follows the template and includes a complete checklist with tests, docs, and changelog items.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/cli-solve-correctness

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
packages/climate-ref/tests/unit/cli/test_solve.py (1)

44-70: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a mixed-case substring assertion here.

This only exercises full lower-case matches. Since _validate_provider_filter() is meant to mirror the solver’s case-insensitive substring matching, adding something like --provider ESM or --provider Lamb would lock that contract down and catch future drift.


ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 8021f0f3-660f-42c6-a431-015e1f983e70

📥 Commits

Reviewing files that changed from the base of the PR and between fda1ac6 and eea479f.

📒 Files selected for processing (5)
  • changelog/766.fix.md
  • packages/climate-ref/src/climate_ref/cli/solve.py
  • packages/climate-ref/src/climate_ref/solver.py
  • packages/climate-ref/tests/unit/cli/test_solve.py
  • packages/climate-ref/tests/unit/test_solver.py

Comment thread packages/climate-ref/src/climate_ref/solver.py Outdated
@codecov

codecov Bot commented Jun 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
core 92.62% <100.00%> (+0.02%) ⬆️
providers 85.47% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/climate-ref/src/climate_ref/cli/solve.py 100.00% <100.00%> (ø)
...es/climate-ref/src/climate_ref/models/execution.py 98.64% <100.00%> (+0.01%) ⬆️
packages/climate-ref/src/climate_ref/solver.py 98.93% <100.00%> (+0.03%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Lock down the case-insensitive substring matching contract in
_validate_provider_filter() so future drift is caught.
A real solve reaps stale in-progress executions before evaluating
ExecutionGroup.should_run, flipping abandoned dirty groups to runnable.
The read-only dry-run skipped that reap and so under-reported them.

Pass the same stale cutoff to should_run during dry-run so abandoned
executions are treated as failed (reaped) for the preview, without
mutating the database.
@lewisjared lewisjared merged commit b478a2e into main Jun 29, 2026
28 checks passed
@lewisjared lewisjared deleted the feat/cli-solve-correctness branch June 29, 2026 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant