diff --git a/CLAUDE.md b/CLAUDE.md index 16ef30cb..664499d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -141,8 +141,8 @@ action.yml # Published composite GitHub Action (coder-ev - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly. -- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. -- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged. +- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `max_steps_to_decide`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step-budget breach to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. +- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm — "is this criterion type live-observable" is `models.LiveSuccessCriterion` subclassing (currently `skill_triggered`, `command_executed`), the single source of truth `validate_early_stop`/`EarlyStopWatcher` check directly via `isinstance`; each subclass implements the abstract, checker-independent `live_decidable_polarities()` (a pure function of its own fields) alongside the checker's `live_verdict` override, and lint rule CE025 (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`, a registry-based whole-tree check, not a per-file AST rule) keeps the two paired. `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing pre-weighting behavior byte-for-byte) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed subset's **ceiling** (best case for everything still undecided) can no longer reach the threshold, a pass-stop once the pass-armed subset's **floor** (worst case) already meets it — both **deferred while any pass-armed criterion is undecided**, so a distractor misfire never truncates a positive row's recall signal before the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. A per-criterion `max_steps_to_decide` (on `LiveSuccessCriterion` only, requires `stop_when`) caps tool-call steps spent still undecided — cumulative across retry attempts of the same turn — before `EarlyStopReason.DECISION_BUDGET_EXCEEDED` force-fails the run outright, bypassing the weighted gate (nothing to weigh a criterion that never decided against). Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally (even with `stop_early: true`) gates on the full set via the strict-AND `all_criteria_passed` — weight magnitude only forgives under the former, so the weighted gate is contingent on the watcher itself firing, not solely on the configured threshold. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged. ## Success Criteria (14 types) diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index aa3ba39e..288cabf2 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -210,8 +210,16 @@ Notes: `count/mean/median/std/min/max`, so your criterion is suite-thresholdable for free. Classification-style criteria return a `ClassificationCriterionResult` and layer accuracy / precision / recall / F1 / confusion on top. -- For **early stop**, implement `live_verdict(...)` and declare - `live_stop_polarities` — a lint rule keeps the two consistent. +- For **early stop**, make your criterion model subclass `LiveSuccessCriterion` + (`models/criteria.py`) instead of `BaseSuccessCriterion`, implement its + abstract `live_decidable_polarities()` (a pure function of the criterion's + own fields — no `turn_records`, no checker instance), and override the + checker's `live_verdict(...)`. `LiveSuccessCriterion` subclassing is the + single source of truth for "is this criterion type live-observable" — + `validate_early_stop`/`EarlyStopWatcher` check `isinstance(c, + LiveSuccessCriterion)` directly, no separate checker-side flag. A lint rule + (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`) keeps the + model subclassing and the checker's `live_verdict` override paired. > A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not > a hard error, unlike agents) — keep type strings unique. diff --git a/docs/REPORT_SCHEMA.md b/docs/REPORT_SCHEMA.md index ef38b1a2..3fb492c5 100644 --- a/docs/REPORT_SCHEMA.md +++ b/docs/REPORT_SCHEMA.md @@ -188,10 +188,13 @@ backend), `num_turns`, `max_turns_exhausted`, ### EarlyStopInfo Present (non-`null`) iff the run stopped early — there is no separate boolean. -Fields: `reason` (`criterion_passed` / `criterion_failed`), +Fields: `reason` (`criterion_passed` / `criterion_failed` / +`decision_budget_exceeded` — the last forces `FinalStatus.FAILURE` outright, +bypassing the weighted gate), `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`, `sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call), -`elapsed_seconds`, `turns_remaining_at_stop`. +`elapsed_seconds`, `turns_remaining_at_stop`, `gate_threshold` (the +`run_limits.stop_early_gate_threshold` in effect for this stop; default `1.0`). --- diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index c2de203a..69cbde15 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -260,6 +260,7 @@ run_limits: | `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. | | `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. | | `stop_early` | `false` | — | Opt-in master switch for early-stop-on-criterion. See [`stop_early`](#stop_early-opt-in-early-stop). | +| `stop_early_gate_threshold` | `1.0` | `[0.0, 1.0]` (but `> 0.0` is enforced at resolution when `stop_early: true`) | Minimum weighted score over the armed subset required to gate as a pass. See [`stop_early`](#stop_early-opt-in-early-stop). | The authoritative source is `src/coder_eval/models/limits.py`. A lint rule (CE030) fails the build if a field defined there goes undocumented in this guide, so the table can't quietly fall behind the @@ -376,11 +377,14 @@ Semantics: (e.g. `stop_when: pass` alongside a `max_count`, or `auto` on an instance that can decide neither) is likewise a hard error at resolution, not a silent full run. -- **Verdict.** An early-stopped run is gated on the **armed subset only**; the - non-armed criteria become **advisory** and are clearly marked (report badge + - per-criterion note + `stopped_early` row). A run that completes naturally is - gated on the **full** set, as always. This is what lets one file serve both a - `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) — +- **Verdict.** Any task armed for early-stop (`stop_early: true`) is gated on + the **armed subset only** — the non-armed criteria become **advisory** and + are clearly marked (report badge + per-criterion note + `stopped_early` + row when the watcher actually fired) — whether or not the watcher actually + cut the run short; one task config maps to one gate semantic. Only a task + that never armed `stop_early` at all is gated on the **full** set, as + always. This is what lets one file serve both a `smoke` flavor + (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md). Verdict parity between the flavors is one-sided: a **fail-stop** is verdict-preserving (the deferral above guarantees every pass-armed signal was allowed to resolve first), but a @@ -390,6 +394,57 @@ Semantics: authoritative precision/recall belongs on the `stop_early: false` run. - **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) — it can never silently disable a criterion or cause a false early stop. +- **Weighting.** `run_limits.stop_early_gate_threshold` (default `1.0`) is the + minimum weighted score (`Σ weight·score / Σ weight`, over the armed subset) + required to gate as a pass — both for the post-hoc verdict and for the live + stop rule itself. A fail-stop fires once the armed subset's **ceiling** (best + case: every still-undecided or already-passed criterion ends up scoring 1.0, + every live-failed one scores 0) can no longer reach the threshold — the gate + is mathematically guaranteed to fail regardless of how the trajectory + continues. A pass-stop fires once the pass-armed subset's **floor** (worst + case: every still-undecided one scores 0) already meets it. At the default + `1.0` both bounds collapse to the pre-weighting rules above exactly (any + single armed criterion's live-fail already drops the ceiling below 1.0, and + the floor only reaches 1.0 once every pass-armed criterion has actually + passed) — lowering it lets a low-weight armed criterion's failure be absorbed + without truncating the run, at the cost of the gate becoming a genuine + weighted average rather than a strict AND. **The armed weighted gate applies + whenever `stop_early: true` is set — one task config, one gate semantic — + regardless of whether the watcher actually fired a stop.** A task armed for + early-stop that instead completes naturally (the agent finishes, or + `max_turns` is hit, before the bound ever trips) is gated on the *same* + weighted armed-subset formula as an actual early stop, not the full-run + `all_criteria_passed`; only a task that never armed `stop_early` at all uses + the strict full-set gate. Each armed criterion's own `pass_threshold` still + decides whether it individually passed (converted to a binary 1.0/0.0 + before weighting) — only the combination rule (weighted average vs strict + AND) changes, which is what makes the `gate_threshold=1.0` default an exact + equivalence with the pre-weighting `all(...)` rule. +- **Decision-step budget.** `max_steps_to_decide` (per armed criterion, only + on `skill_triggered` / `command_executed`, requires `stop_when`) caps how + many tool-call steps that criterion may spend still **undecided** before the + run gives up on it: + + ```yaml + success_criteria: + - type: skill_triggered + description: "date-teller must activate within 5 steps" + skill_name: date-teller + expected_skill: date-teller + stop_when: pass + max_steps_to_decide: 5 + ``` + + Once the cap is exceeded (checked AFTER the normal fail-/pass-stop checks + each round, so a criterion that decides on that very step is never + penalized), the watcher fires `reason: decision_budget_exceeded` and the run + is forced to `FinalStatus.FAILURE` outright — bypassing + `stop_early_gate_threshold`'s weighted gate entirely, since a criterion that + never reached a verdict has nothing meaningful to weigh against the others. + `None` (default) = no cap; the run relies solely on `run_limits.max_turns`. + The step count is **cumulative across every retry attempt** of the turn — + including an attempt that crashed or timed out before this criterion's own + investigation even began — so size the budget with that headroom in mind. Observability (every early-stopped run is flagged everywhere so analysis never compares a truncated run against a full one): @@ -535,6 +590,7 @@ All criteria share these fields: | `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate | | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass | | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`/`auto`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). `auto` arms whichever polarity this instance can decide (for dataset-fanned criteria whose positive/distractor role flips per row). See [`stop_early`](#stop_early-opt-in-early-stop). | +| `max_steps_to_decide` | `null` | **Only on live-observable criteria** (`skill_triggered`, `command_executed`) — requires `stop_when` to be set. Caps the tool-call steps this armed criterion may spend still undecided before the run gives up and force-fails. See [`stop_early`](#stop_early-opt-in-early-stop). | **Scoring types:** - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered` diff --git a/src/coder_eval/criteria/base.py b/src/coder_eval/criteria/base.py index 503c31f1..cbe7b9a8 100644 --- a/src/coder_eval/criteria/base.py +++ b/src/coder_eval/criteria/base.py @@ -26,6 +26,19 @@ # A criterion's verdict from a PARTIAL, mid-run trajectory (early-stop observability). # "undecided" means the outcome is not yet knowable from the events seen so far. +# +# CONTRACT every live_verdict override must satisfy (see BaseCriterion.live_verdict): +# - Deterministic: a pure function of the ``turn_records`` prefix passed in — no +# wall-clock, randomness, or other hidden state. +# - Monotonic: once it returns "pass"/"fail" for some trajectory prefix, it MUST +# return that SAME verdict for every longer prefix (i.e. every later call in the +# same run). "undecided" is the only verdict allowed to change on a later call. +# EarlyStopWatcher's deferred fail-stop and pass/fail flip-attribution +# (early_stop.py::_prev_verdicts) are correct only because both existing +# implementations (skill_triggered, command_executed) honor this. A non-monotonic or +# non-deterministic override compiles and passes CE025 (which only checks +# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts the stop +# logic — there is currently no automated enforcement beyond this docstring. LiveVerdict = Literal["pass", "fail", "undecided"] @@ -200,13 +213,6 @@ def _check_impl( # Subclasses MUST define this as a class variable criterion_type: ClassVar[str] - # Which polarities this criterion can decide from a PARTIAL, mid-run trajectory. - # Empty (base default) = not observable mid-run, so it can never arm early-stop. - # A subclass that reads only turn_records and can decide mid-run declares the - # polarities it supports (e.g. frozenset({"pass", "fail"})) AND overrides - # live_verdict; CE025 enforces that the two stay consistent. - live_stop_polarities: ClassVar[frozenset[str]] = frozenset() - def __new__(cls, *args: Any, **kwargs: Any) -> "BaseCriterion[C]": """Block direct instantiation of ``BaseCriterion`` itself. @@ -426,36 +432,17 @@ def live_verdict( from ``check()``/``_check_impl`` run on the frozen trajectory after the stop, so a live/final divergence can never corrupt scoring. - Base default: ``"undecided"`` (not observable mid-run). Subclasses that - override this MUST also declare a non-empty ``live_stop_polarities`` (and - vice versa) — enforced by lint rule CE025. + Base default: ``"undecided"`` (not observable mid-run). A checker + overrides this iff its criterion model is a ``LiveSuccessCriterion`` + subclass (``models/criteria.py``) — that subclassing is the single + source of truth for "is this criterion type live-observable", checked + by ``validate_early_stop`` / ``EarlyStopWatcher`` and enforced by lint + rule CE025. An override MUST also satisfy the deterministic + monotonic + contract documented on the ``LiveVerdict`` type above (not enforced by + CE025 or any other automated check). """ return "undecided" - @classmethod - def live_decidable_polarities(cls, criterion: C) -> frozenset[str]: - """Which polarities THIS criterion *instance* can actually decide mid-run. - - ``live_stop_polarities`` is a class-level *capability* — the widest set - of polarities the checker's ``live_verdict`` could ever emit. But for - some criteria whether a given polarity can fire depends on the instance's - configuration, not just its type. ``command_executed`` is the canonical - case: it can live-``pass`` only with no upper bound, and live-``fail`` - only with one, so a specific criterion may support strictly fewer - polarities than its class advertises (down to none — a "dead arm"). - - ``validate_early_stop`` gates the requested ``stop_when`` polarity on THIS - set, not the ClassVar, so an instance that can never decide its armed - polarity is rejected at resolution rather than silently degrading to a - full run (the "never a silent no-op" guarantee). - - Default: the class-level ``live_stop_polarities`` — correct for every - criterion whose decidability is purely type-level (e.g. ``skill_triggered``). - Overrides MUST return a subset of ``live_stop_polarities`` (a criterion - cannot decide a polarity its ``live_verdict`` never emits). - """ - return cls.live_stop_polarities - def aggregate( self, criterion: C, diff --git a/src/coder_eval/criteria/command_executed.py b/src/coder_eval/criteria/command_executed.py index 92d06ec6..e5343ef2 100644 --- a/src/coder_eval/criteria/command_executed.py +++ b/src/coder_eval/criteria/command_executed.py @@ -3,7 +3,7 @@ import json import logging import re -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING from coder_eval.criteria.base import BaseCriterion, CheckContext, LiveVerdict, register_criterion from coder_eval.models import CommandExecutedCriterion, CriterionResult @@ -30,42 +30,6 @@ class CommandExecutedChecker(BaseCriterion[CommandExecutedCriterion]): criterion_type = "command_executed" - # Observable mid-run: command matches accumulate monotonically in the live - # stream, so a min_count pass (no upper bound) and a max_count exceedance - # (incl. the must-NOT-run 0/0 form) are both decidable before end-of-run. - live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"}) - - @classmethod - def live_decidable_polarities(cls, criterion: CommandExecutedCriterion) -> frozenset[str]: - """Narrow the class capability to what THIS instance can decide mid-run. - - The class advertises ``{"pass", "fail"}``, but ``live_verdict`` can only: - - - ``pass`` when there is no upper bound and a positive floor - (``max_count is None and min_count > 0``) — with an upper bound a pass - is not final until end-of-run, so it never fires live; and - - ``fail`` when there IS an upper bound (``max_count is not None``), the - moment the count exceeds it (this includes the ``min_count: 0, - max_count: 0`` "must NOT run" form). - - So these instance shapes are dead arms the class-level check misses: - - - ``stop_when: pass`` with ``max_count`` set → pass can never fire; - - ``stop_when: fail`` with ``max_count: None`` → fail can never fire; - - ``min_count: 0, max_count: None`` → neither can ever fire. - - Reporting the true per-instance set here lets ``validate_early_stop`` - reject such arming at resolution instead of silently degrading to a full - run. Stays a subset of ``live_stop_polarities`` by construction. - """ - decidable: set[str] = set() - if criterion.max_count is None: - if criterion.min_count > 0: - decidable.add("pass") - else: - decidable.add("fail") - return frozenset(decidable) - @staticmethod def _matching_commands( criterion: CommandExecutedCriterion, diff --git a/src/coder_eval/criteria/skill_triggered.py b/src/coder_eval/criteria/skill_triggered.py index 999d8ce5..2b1ded9c 100644 --- a/src/coder_eval/criteria/skill_triggered.py +++ b/src/coder_eval/criteria/skill_triggered.py @@ -10,7 +10,7 @@ import logging import re -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING from coder_eval.criteria._classification_aggregate import overlay_classification_metrics from coder_eval.criteria.base import BaseCriterion, LiveVerdict, register_criterion @@ -109,14 +109,6 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]): criterion_type = "skill_triggered" - # Observable mid-run: a Skill tool call (or a skill file read) is a positive - # event in the live stream. The TYPE can decide either polarity — a positive - # criterion live-passes when its expected skill is engaged, a - # distractor/negative one live-fails when its (wrong) skill is engaged — but - # any single INSTANCE decides only one of the two; see - # ``live_decidable_polarities``. - live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"}) - def _check_impl( self, criterion: SkillTriggeredCriterion, @@ -179,37 +171,15 @@ def live_verdict( frozen trajectory by construction — whether or not the run stopped early. A positive criterion can therefore only ever live-``pass`` and a distractor/negative one only ever live-``fail``; their *absence* is never - decidable mid-run (see ``live_decidable_polarities``). This is the change - from first-engagement: a wrong skill engaged first no longer live-fails a - positive row — the run keeps going so the expected skill can still load. + decidable mid-run (see ``SkillTriggeredCriterion.live_decidable_polarities`` + in models/criteria.py). This is the change from first-engagement: a wrong + skill engaged first no longer live-fails a positive row — the run keeps + going so the expected skill can still load. """ if criterion.skill_name not in _all_engaged_skill_names(turn_records): return "undecided" return "pass" if criterion.expected_skill == criterion.skill_name else "fail" - @classmethod - def live_decidable_polarities(cls, criterion: SkillTriggeredCriterion) -> frozenset[str]: - """Per-instance narrowing under the any-engagement latch. - - Unlike the type-level capability (``live_stop_polarities`` = both), a - single instance decides exactly one polarity: - - - a **positive** criterion (``skill_name == expected_skill``) can only - live-``pass`` (the expected skill engaging is a decidable hit; its - absence is not knowable mid-run); - - a **distractor/negative** criterion (``skill_name != expected_skill``, - including the ``expected_skill == ""`` negatives) can only - live-``fail`` (a wrong skill engaging is a decidable miss; its absence - is not). - - ``validate_early_stop`` gates the requested ``stop_when`` on this set, so - arming a positive with ``fail`` / a distractor with ``pass`` — or either - with ``decided`` (which needs both) — is rejected at resolution rather - than silently degrading to a full run. - """ - expected_yes = criterion.expected_skill == criterion.skill_name - return frozenset({"pass"}) if expected_yes else frozenset({"fail"}) - def aggregate( self, criterion: SkillTriggeredCriterion, diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index 1d155604..47927cd0 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -40,6 +40,8 @@ FileMatchesRegexCriterion, JMESPathAssertion, JsonCheckCriterion, + LivePolarity, + LiveSuccessCriterion, LLMJudgeCriterion, ReferenceComparisonCriterion, RegexPattern, @@ -232,6 +234,8 @@ "LLMJudgeCriterion", "AgentJudgeCriterion", "SkillTriggeredCriterion", + "LiveSuccessCriterion", + "LivePolarity", "SuccessCriterion", # Routing "ROUTE_NAMES", diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index 2605e1cf..26a1142c 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -8,7 +8,7 @@ from __future__ import annotations -from abc import ABC +from abc import ABC, abstractmethod from typing import Annotated, Any, ClassVar, Literal, Self from pydantic import BaseModel, ConfigDict, Field, model_validator @@ -187,6 +187,86 @@ def is_gating(self) -> bool: # Business logic (check operations) moved to SuccessChecker in evaluator.py +# The two polarities a live-observable criterion can decide mid-run — distinct +# from the 3-value LiveVerdict ("pass"/"fail"/"undecided") the checker's +# live_verdict returns: this is the narrower CAPABILITY type, "undecided" is +# never a valid decidable polarity. Typed here (not a bare frozenset[str]) so +# a live_decidable_polarities override returning a stray/typo'd string, or +# "undecided" itself, is a pyright error rather than a runtime-only lint gap. +LivePolarity = Literal["pass", "fail"] + + +class LiveSuccessCriterion(BaseSuccessCriterion): + """Base for criteria observable from a PARTIAL, mid-run trajectory (early-stop). + + ``live_decidable_polarities`` is a pure function of THIS instance's own + fields — no ``turn_records``, no sandbox, no checker instance needed (e.g. + ``command_executed`` can decide this purely from whether ``max_count`` is + set). That makes it genuinely computable on the data model rather than the + checker, unlike the checker's ``live_verdict`` (``criteria/base.py``), + which reads the actual trajectory and stays checker-side logic. Moving + decidability here also gives early-stop-only config (e.g. + ``max_steps_to_decide``) a home that doesn't pollute ``BaseSuccessCriterion`` + with a field meaningless for every non-observable criterion type. + + Only ``SkillTriggeredCriterion`` / ``CommandExecutedCriterion`` subclass + this today; a criterion type is "live-observable" iff it is a + ``LiveSuccessCriterion`` subclass — the single source of truth + ``validate_early_stop`` / ``EarlyStopWatcher`` consult (no separate + checker-side flag to keep in sync). + """ + + max_steps_to_decide: int | None = Field( + default=None, + ge=1, + description=( + "Cap on tool-call steps this ARMED criterion (stop_when must be set) " + "may spend still 'undecided' before the run gives up on it. Once " + "exceeded, EarlyStopWatcher fires an early stop with reason " + "'decision_budget_exceeded' and the run is forced to FinalStatus." + "FAILURE outright — regardless of what any other armed criterion's " + "weighted score would otherwise gate to (this criterion never " + "reached a verdict at all, so there is nothing to weigh). None " + "(default) = no cap; the run relies solely on run_limits.max_turns. " + "Requires run_limits.stop_early and this criterion's own stop_when. " + "The step count is CUMULATIVE across every retry attempt of the " + "turn (the same EarlyStopWatcher instance, and its counters, " + "persist across retries) — including attempts that ultimately " + "crashed or timed out before this criterion's own investigation " + "even began. Size the budget with that headroom in mind." + ), + ) + + @model_validator(mode="after") + def _check_max_steps_requires_armed(self) -> Self: + """Reject a decision-step cap on a criterion that isn't armed for early-stop. + + ``max_steps_to_decide`` only means anything relative to a criterion + that ``EarlyStopWatcher`` is actually tracking (``stop_when`` set); + setting it without ``stop_when`` is a dead field that silently does + nothing, so reject it at load time rather than let it rot unnoticed. + """ + if self.max_steps_to_decide is not None and self.stop_when is None: + raise ValueError( + f"criterion {self.type!r}: max_steps_to_decide requires stop_when to be set " + + "(the decision-step budget is meaningless for a criterion that isn't armed " + + "for early-stop)." + ) + return self + + @abstractmethod + def live_decidable_polarities(self) -> frozenset[LivePolarity]: + """Which polarities THIS instance can decide mid-run, from its own fields alone. + + Must return a subset of the polarities the corresponding checker's + ``live_verdict`` can ever emit for this criterion type. Used by + ``validate_early_stop`` to reject arming a polarity this instance can + never reach, and by ``EarlyStopWatcher`` to resolve which polarities a + ``stop_when`` value actually arms for this instance (see + ``orchestration.early_stop._requested_polarities``). + """ + + class FileExistsCriterion(BaseSuccessCriterion): """Check if a file exists at the specified path. @@ -472,7 +552,7 @@ class CommandsEfficiencyCriterion(BaseSuccessCriterion): expected_commands: int = Field(ge=1, description="Expected number of tool commands to complete the task") -class CommandExecutedCriterion(BaseSuccessCriterion): +class CommandExecutedCriterion(LiveSuccessCriterion): """Check whether the agent executed specific commands/tools. Inspects CommandTelemetry records from TurnRecord.commands to verify @@ -548,6 +628,32 @@ def _validate_count_bounds(self) -> CommandExecutedCriterion: raise ValueError(f"max_count ({self.max_count}) must be >= min_count ({self.min_count})") return self + def live_decidable_polarities(self) -> frozenset[LivePolarity]: + """Narrow to what THIS instance can decide mid-run. + + The checker's ``live_verdict`` (``criteria/command_executed.py``) can + only: + + - ``pass`` when there is no upper bound and a positive floor + (``max_count is None and min_count > 0``) — with an upper bound a + pass is not final until end-of-run, so it never fires live; and + - ``fail`` when there IS an upper bound (``max_count is not None``), + the moment the count exceeds it (this includes the + ``min_count: 0, max_count: 0`` "must-NOT-run" form). + + So these instance shapes are dead arms the class-level check misses: + ``stop_when: pass`` with ``max_count`` set (pass can never fire); + ``stop_when: fail`` with ``max_count: None`` (fail can never fire); + ``min_count: 0, max_count: None`` (neither can ever fire). + """ + decidable: set[LivePolarity] = set() + if self.max_count is None: + if self.min_count > 0: + decidable.add("pass") + else: + decidable.add("fail") + return frozenset(decidable) + class UiPathEvalCriterion(BaseSuccessCriterion): """Check evaluation results against UiPath agent performance. @@ -608,7 +714,7 @@ class ClassificationMatchCriterion(BaseSuccessCriterion): ) -class SkillTriggeredCriterion(BaseSuccessCriterion): +class SkillTriggeredCriterion(LiveSuccessCriterion): """Binary classifier: did the agent engage the target skill during the run? Agent-agnostic. Observed label is ``"yes"`` when ``turn_records`` show the @@ -644,6 +750,29 @@ class SkillTriggeredCriterion(BaseSuccessCriterion): description="Only count Skill invocations whose 'skill' parameter matches this name.", ) + def live_decidable_polarities(self) -> frozenset[LivePolarity]: + """Per-instance narrowing under the checker's any-engagement latch. + + The checker's ``live_verdict`` (``criteria/skill_triggered.py``) can + decide either polarity at the TYPE level, but a single INSTANCE only + ever resolves one of them: + + - a **positive** criterion (``skill_name == expected_skill``) can only + live-``pass`` (the expected skill engaging is a decidable hit; its + absence is not knowable mid-run); + - a **distractor/negative** criterion (``skill_name != expected_skill``, + including the ``expected_skill == ""`` negatives) can only + live-``fail`` (a wrong skill engaging is a decidable miss; its + absence is not). + + ``validate_early_stop`` gates the requested ``stop_when`` on this set, + so arming a positive with ``fail`` / a distractor with ``pass`` — or + either with ``decided`` (which needs both) — is rejected at resolution + rather than silently degrading to a full run. + """ + expected_yes = self.expected_skill == self.skill_name + return frozenset({"pass"}) if expected_yes else frozenset({"fail"}) + class LLMJudgeCriterion(BaseSuccessCriterion): """Have an LLM grade the task's final state against an author-supplied prompt. diff --git a/src/coder_eval/models/limits.py b/src/coder_eval/models/limits.py index abce752b..7f9184a3 100644 --- a/src/coder_eval/models/limits.py +++ b/src/coder_eval/models/limits.py @@ -92,12 +92,48 @@ class RunLimits(BaseModel): description=( "Opt-in master switch for early-stop-on-criterion. When True, the run ends early " "once the armed criteria (those with stop_when set, incl. per-instance 'auto') " - "are decided mid-run: pass-stop when every pass-armed criterion live-passes, " - "fail-stop on the first fail-armed live-fail (deferred while any pass-armed " - "criterion is undecided, so a misfire never truncates the recall signal) - so a " - "raised max_turns is not wasted once the measured signal has happened. Default " - "False keeps behavior identical. Requires a Claude single-shot task with at least " - "one observable armed criterion; every unsupported combination is rejected at " - "resolution time." + "are decided mid-run: pass-stop when the armed subset's weighted score is " + "GUARANTEED to reach stop_early_gate_threshold regardless of any criterion still " + "undecided, fail-stop when it is GUARANTEED it never can (deferred while any " + "pass-armed criterion is undecided, so a misfire never truncates the recall " + "signal) - so a raised max_turns is not wasted once the outcome is locked in. " + "Default False keeps behavior identical. Requires a Claude single-shot task with " + "at least one observable armed criterion; every unsupported combination is " + "rejected at resolution time." ), ) + stop_early_gate_threshold: float = Field( + default=1.0, + ge=0.0, + le=1.0, + description=( + "Minimum weighted score (Σ weight_i·score_i / Σ weight_i, over the ARMED subset " + "only) required for an early-stopped run to gate as a pass. Also the bound " + "early-stop's trigger checks against: a fail-stop fires once no combination of " + "still-undecided armed criteria could raise the weighted score to this " + "threshold; a pass-stop fires once it is already guaranteed to meet it regardless " + "of what's still undecided. Default 1.0 reproduces the pre-weighting behavior " + "exactly (every armed criterion's live-observable score is binary 0/1, so a " + "weighted score of 1.0 requires every armed criterion to have actually passed) - " + "lowering it lets a low-weight armed criterion's failure be absorbed without " + "truncating the run, at the cost of the gate/trigger becoming a genuine weighted " + "average rather than a strict AND." + ), + ) + + # NOTE: stop_early_gate_threshold <= 0.0 together with stop_early: True is + # a degenerate, gate-neutralizing config (a threshold of 0 trivially + # passes the armed gate regardless of whether anything decided) and is + # rejected — but NOT here. RunLimits is field-merged across 5 layers, so a + # model-level validator has no visibility into which layer produced the + # merged value and cannot distinguish a real mistake from a value merged + # forward from a sibling layer (e.g. a task-level threshold inherited by a + # variant that only flips stop_early). That distinction requires seeing + # the whole resolved task, so the check lives in + # orchestration/early_stop.py::validate_early_stop instead, where it + # raises EarlyStopConfigError and gets the same hard-stop CLI treatment + # (flips the plan exit code, aborts run) as every other early-stop + # guardrail — a plain pydantic ValueError here would instead land in + # plan_command's generic per-variant "resolution failed" branch, which + # prints red text but does NOT flip the exit code by design (unlike + # EarlyStopConfigError), so a model-level raise would silently pass CI. diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index 68d7a179..7591ba3c 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -417,11 +417,17 @@ class EarlyStopReason(StrEnum): ``EvaluationResult`` carries it and ``models/`` is a leaf package that cannot import from ``simulation``. ``DialogStopReason`` is a stylistic reference only. Early-stop is orthogonal telemetry, NOT a ``FinalStatus`` — - the terminal-status set stays closed. + the terminal-status set stays closed — with ONE deliberate exception: + ``DECISION_BUDGET_EXCEEDED`` forces ``FinalStatus.FAILURE`` directly at the + orchestrator finalize step (``orchestrator.py``), bypassing + ``armed_criteria_passed``'s weighted gate entirely — the criterion whose + budget expired never reached a verdict at all, so there is nothing + meaningful to weigh it against. """ CRITERION_PASSED = "criterion_passed" CRITERION_FAILED = "criterion_failed" + DECISION_BUDGET_EXCEEDED = "decision_budget_exceeded" class EarlyStopInfo(BaseModel): @@ -434,7 +440,10 @@ class EarlyStopInfo(BaseModel): the round-trip is safe. """ - reason: EarlyStopReason = Field(description="Why the run stopped: armed criteria passed, or definitively failed.") + reason: EarlyStopReason = Field( + description="Why the run stopped: armed criteria passed, definitively failed, or an armed " + + "criterion's decision-step budget (max_steps_to_decide) expired unresolved." + ) deciding_criterion_type: str = Field( description="Type of the criterion whose live verdict fired the stop (the failing one on " + "fail-stop; the last-to-pass on pass-stop)." @@ -463,6 +472,16 @@ class EarlyStopInfo(BaseModel): description="max_turns - sdk_turn_index (an upper bound on turns avoided, not a measured " + "saving); None when max_turns is unset.", ) + gate_threshold: float = Field( + default=1.0, + ge=0.0, + le=1.0, + description="run_limits.stop_early_gate_threshold in effect for this stop — captured so a " + + "persisted task.json is self-describing (e.g. comparing early-stopped runs across an " + + "experiment sweep that varies the threshold) without needing the resolved task config. " + + "Bounded to mirror the source field so a hand-edited or externally produced record " + + "cannot represent a value the authoritative field would reject.", + ) class EvaluationResult(BaseModel): @@ -654,8 +673,8 @@ def all_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: if c.is_gating ) - def armed_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: - """True iff every ARMED criterion (``stop_when`` set) meets its pass_threshold. + def armed_criteria_passed(self, criteria: list[SuccessCriterion], gate_threshold: float = 1.0) -> bool: + """True iff the ARMED subset's weighted score meets ``gate_threshold``. The early-stop gate: on an early-stopped run only the armed subset gates ``final_status`` (non-armed criteria are advisory — recorded but never @@ -667,6 +686,22 @@ def armed_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: is a defensive guard against misuse. No ``is_gating`` filter is needed here: ``BaseSuccessCriterion`` rejects ``weight: 0`` together with ``stop_when``, so every armed criterion is gating by construction. + + Each armed criterion's OWN ``pass_threshold`` still decides whether it + individually passed — ``r.score`` is converted to a binary 1.0/0.0 via + ``r.score >= c.pass_threshold`` before weighting, exactly mirroring + ``all_criteria_passed``'s per-criterion comparison. Only the + combination rule changes: ``all_criteria_passed`` ANDs those binary + outcomes, this weights and averages them against ``gate_threshold``. + This is what makes the ``gate_threshold=1.0`` default an EXACT + equivalence with the pre-weighting ``all(...)`` rule, not merely an + approximation that happens to hold for binary-scoring criteria: a + weighted average of 1.0 requires every armed criterion's binary + outcome to be 1.0, i.e. every one to have individually passed its own + ``pass_threshold`` — identical to ``all(...)`` regardless of what + ``r.score`` itself was. Callers pass + ``run_limits.stop_early_gate_threshold`` to opt into a genuine + weighted average below 1.0. """ if len(self.success_criteria_results) != len(criteria): raise ValueError( @@ -681,7 +716,19 @@ def armed_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: f"armed_criteria_passed called with no armed criteria for task {self.task_id}; " + "the early-stop gate is only valid when at least one criterion sets stop_when." ) - return all(r.score >= c.pass_threshold for r, c in armed) + total_weight = sum(c.weight for _, c in armed) + if total_weight <= 0.0: + # Unreachable today (weight=0 + stop_when is rejected at the model + # layer, so every armed criterion carries weight > 0) — but a + # defensive guard on a pass/fail gate must fail CLOSED, not open, + # against a future criterion subclass that bypasses that + # validator. Mirrors EarlyStopWatcher._ceiling's lack of an + # equivalent guard: that one would raise ZeroDivisionError instead + # (fails by crashing, not by silently passing) rather than diverge + # toward a false pass. + return False + weighted_score = sum((1.0 if r.score >= c.pass_threshold else 0.0) * c.weight for r, c in armed) / total_weight + return weighted_score >= gate_threshold class CriterionStats(BaseModel): diff --git a/src/coder_eval/orchestration/early_stop.py b/src/coder_eval/orchestration/early_stop.py index 09d7acc7..379eac4d 100644 --- a/src/coder_eval/orchestration/early_stop.py +++ b/src/coder_eval/orchestration/early_stop.py @@ -27,9 +27,24 @@ Live verdicts only *trigger* the stop; the authoritative scores always come from the standard ``check_all_async`` on the frozen trajectory after the cut. -Precision trade-off: a pass-stop cuts the run the instant every *pass-armed* -criterion is decided, so a *fail-armed* criterion (e.g. a distractor) that would -only misfire on a LATER tool call is never observed — the frozen trajectory then +Weighting: both the stop rule and the post-hoc gate +(``EvaluationResult.armed_criteria_passed``) consult ``run_limits. +stop_early_gate_threshold`` (default ``1.0``) rather than treating every armed +criterion's pass/fail as equally decisive. A fail-stop fires once the armed +subset's CEILING (best case: every still-``undecided``/``pass`` criterion +scores 1.0, every live-``fail``ed one scores 0) can no longer reach the +threshold — i.e. the gate is mathematically guaranteed to fail regardless of +how the trajectory continues. A pass-stop fires once the pass-armed subset's +FLOOR (worst case: every still-undecided one scores 0) already meets it. At the +default threshold of 1.0 both bounds collapse exactly to "any single armed +criterion's live-fail stops the run" / "every pass-armed criterion has +live-passed" — byte-for-byte the pre-weighting behavior, since the only +live-observable criteria score binary 0/1. Lowering the threshold lets a +low-weight armed criterion's failure be absorbed without truncating the run. + +Precision trade-off: a pass-stop cuts the run the instant the pass-armed floor +locks in, so a *fail-armed* criterion (e.g. a distractor) that would only +misfire on a LATER tool call is never observed — the frozen trajectory then scores that row as a clean pass. This is an intentional precision-for-budget trade of the opt-in "smoke" flavor; the authoritative precision/recall must come from a non-early-stop (``stop_early: false``) run. @@ -52,7 +67,7 @@ import time from typing import TYPE_CHECKING, Any, Literal, assert_never -from coder_eval.models import EarlyStopInfo, EarlyStopReason +from coder_eval.models import EarlyStopInfo, EarlyStopReason, LivePolarity, LiveSuccessCriterion from coder_eval.streaming.collector import EventCollector from coder_eval.streaming.events import ( AgentStartEvent, @@ -66,21 +81,21 @@ if TYPE_CHECKING: from coder_eval.criteria.base import BaseCriterion, LiveVerdict - from coder_eval.models import BaseSuccessCriterion, CommandTelemetry, TaskDefinition + from coder_eval.models import CommandTelemetry, TaskDefinition # Armed pair the watcher holds: (criterion model, its checker). Lives in the # TYPE_CHECKING block (only annotations reference it, and those are lazy under # `from __future__ import annotations`), so the names are real references # rather than quoted strings static analyzers cannot resolve. - _ArmedPair = tuple[BaseSuccessCriterion, BaseCriterion[Any]] + _ArmedPair = tuple[LiveSuccessCriterion, BaseCriterion[Any]] logger = logging.getLogger(__name__) def _requested_polarities( - stop_when: Literal["pass", "fail", "decided", "auto"], decidable: frozenset[str] -) -> frozenset[str]: + stop_when: Literal["pass", "fail", "decided", "auto"], decidable: frozenset[LivePolarity] +) -> frozenset[LivePolarity]: """The polarities a ``stop_when`` value requests to arm, given what the instance can decide. The single source of truth for the ``stop_when`` -> polarity mapping — both @@ -170,20 +185,33 @@ def validate_early_stop(task: TaskDefinition) -> None: + "arming requires at least one stop criterion (e.g. stop_when: auto)." ) - # (3)+(4) Per armed criterion: observable, then the requested polarity is - # decidable. The criteria registry is not initialized at resolution time. - from coder_eval.criteria import CriterionRegistry, init_criteria + # (0) A threshold of exactly 0 trivially satisfies both the pass-stop + # floor check and the final weighted gate regardless of whether any armed + # criterion has actually decided — neutralizing the armed pass/fail gate + # with one YAML line (coder-eval is used as a CI gate). Checked here + # (not on RunLimits itself) because this is the whole-task, hard-stop + # surface: an EarlyStopConfigError here flips the plan exit code and + # aborts run, whereas a plain ValueError on the merged RunLimits model + # would land in the CLI's generic "resolution failed" branch, which + # prints red text but does not flip the exit code. + if limits.stop_early_gate_threshold <= 0.0: + raise EarlyStopConfigError( + f"run_limits.stop_early_gate_threshold ({limits.stop_early_gate_threshold}) must be " + + "> 0.0 when stop_early is True (a threshold of 0 trivially passes the armed gate " + + "regardless of whether any armed criterion actually decided)." + ) - init_criteria(validate=False) + # (3)+(4) Per armed criterion: observable, then the requested polarity is + # decidable. for c in armed: # `armed` filtered on `stop_when is not None`; re-bind + assert so pyright # narrows the Literal away from None for the set arithmetic below. polarity = c.stop_when assert polarity is not None - checker_cls = CriterionRegistry.get_checker(c.type) - # (3) Class-level observability: an empty ``live_stop_polarities`` means - # the criterion TYPE can never decide mid-run, regardless of config. - if not checker_cls.live_stop_polarities: + # (3) Type-level observability: a criterion type is live-observable iff + # its model is a ``LiveSuccessCriterion`` subclass (models/criteria.py) + # — the single source of truth, replacing a separate checker-side flag. + if not isinstance(c, LiveSuccessCriterion): raise EarlyStopConfigError( f"criterion type {c.type!r} is armed (stop_when={polarity!r}) but is not " + "observable mid-run; early-stop supports only live-observable criteria " @@ -194,7 +222,7 @@ def validate_early_stop(task: TaskDefinition) -> None: # instance's config, so gate on the per-instance set — otherwise a dead # arm (a polarity this instance can never fire) would silently degrade to # a full run instead of erroring here. - polarities = checker_cls.live_decidable_polarities(c) + polarities = c.live_decidable_polarities() requested = _requested_polarities(polarity, polarities) # Dead arm: only `auto` can request the empty set (it requests exactly # the instance's decidable polarities) — an instance that can decide @@ -217,6 +245,24 @@ def validate_early_stop(task: TaskDefinition) -> None: + "can live-pass only with max_count unset + min_count>0, and live-fail only " + "with max_count set)." ) + # (6) A decision-step budget is meaningless for a fail-only-decidable + # instance (a pure distractor/guard, e.g. a "must-NOT-run" command or a + # negative-row skill_triggered): its "undecided" IS the success + # state — the forbidden event simply hasn't happened yet, and staying + # undecided forever is correct, not a stall. The budget only makes + # sense for an instance that can decide "pass": something it is + # actively waiting to observe. Rejecting this at resolution (rather + # than letting EarlyStopWatcher force-fail a clean run) matters most + # for a dataset-fanned `auto` criterion, where the same YAML line + # would force-fail every negative row. + if c.max_steps_to_decide is not None and "pass" not in polarities: + raise EarlyStopConfigError( + f"criterion {c.type!r} ({c.description!r}) sets max_steps_to_decide but this " + + f"instance can only ever live-decide {sorted(polarities) or 'no polarities'} — " + + "a fail-only-decidable instance's 'undecided' is its success state (the " + + "forbidden event hasn't happened), so a decision-step budget would force-fail " + + "a clean run. max_steps_to_decide requires an instance that can live-pass." + ) class EarlyStopWatcher: @@ -246,16 +292,19 @@ def __init__( armed: list[_ArmedPair], *, max_turns: int | None, + gate_threshold: float = 1.0, ) -> None: self._task_id = task_id self._armed = armed + self._gate_threshold = gate_threshold + self._armed_weight = sum(c.weight for c, _ in armed) # Per-instance resolved arming polarities, aligned with ``_armed``. Static # for the run, resolved through ``_requested_polarities`` (the single # stop_when -> polarity mapping). The stop rule consults this, not the raw # ``stop_when`` string, so a distractor armed ``auto`` (fail only) is not # required to live-pass for a pass-stop. - self._armed_polarities: list[frozenset[str]] = [ - self._resolve_armed_polarities(criterion, checker) for criterion, checker in armed + self._armed_polarities: list[frozenset[LivePolarity]] = [ + self._resolve_armed_polarities(criterion) for criterion, _checker in armed ] self._max_turns = max_turns self._collector = EventCollector() @@ -280,11 +329,18 @@ def for_task(cls, task: TaskDefinition) -> EarlyStopWatcher: from coder_eval.criteria import CriterionRegistry, init_criteria init_criteria(validate=False) + # The `isinstance` check is defense-in-depth, not load-bearing: every + # call site (`run`, `plan`) runs `validate_early_stop` first, which + # already hard-rejects an armed non-observable criterion at resolution + # time. Narrows `c` to `LiveSuccessCriterion` for pyright either way. armed: list[_ArmedPair] = [ - (c, CriterionRegistry.get_checker(c.type)()) for c in task.success_criteria if c.stop_when is not None + (c, CriterionRegistry.get_checker(c.type)()) + for c in task.success_criteria + if c.stop_when is not None and isinstance(c, LiveSuccessCriterion) ] max_turns = task.run_limits.max_turns if task.run_limits is not None else None - return cls(task.task_id, armed, max_turns=max_turns) + gate_threshold = task.run_limits.stop_early_gate_threshold if task.run_limits is not None else 1.0 + return cls(task.task_id, armed, max_turns=max_turns, gate_threshold=gate_threshold) # --- StreamCallback -------------------------------------------------- # @@ -357,7 +413,7 @@ def disarmed(self) -> bool: # --- Stop rule -------------------------------------------------- # @staticmethod - def _resolve_armed_polarities(criterion: BaseSuccessCriterion, checker: BaseCriterion[Any]) -> frozenset[str]: + def _resolve_armed_polarities(criterion: LiveSuccessCriterion) -> frozenset[LivePolarity]: """The polarities this armed instance may fire, via ``_requested_polarities``. Validation has already guaranteed the resolved set is non-empty and @@ -368,9 +424,61 @@ def _resolve_armed_polarities(criterion: BaseSuccessCriterion, checker: BaseCrit sw = criterion.stop_when if sw is None: return frozenset() - return _requested_polarities(sw, checker.live_decidable_polarities(criterion)) + return _requested_polarities(sw, criterion.live_decidable_polarities()) + + def _ceiling(self, verdicts: list[LiveVerdict]) -> float: + """Best-case weighted score over the WHOLE armed set, given current verdicts. + + Every already-live-failed criterion is pinned at 0 (a monotonic + ``live_verdict`` guarantees it stays failed); every ``pass`` or still + ``undecided`` criterion is credited its full weight (the optimistic + assumption that it could still end up scoring 1.0). This is the same + weighting ``EvaluationResult.armed_criteria_passed`` uses for the real, + final gate, so ``ceiling < gate_threshold`` means the gate is + mathematically guaranteed to fail no matter how the trajectory continues. + """ + return sum(c.weight for (c, _checker), v in zip(self._armed, verdicts, strict=True) if v != "fail") / ( + self._armed_weight + ) + + def _floor(self, verdicts: list[LiveVerdict], indices: list[int]) -> float | None: + """Worst-case weighted score over the given armed-index subset, given current verdicts. + + Mirrors ``_ceiling`` for the opposite direction: every still-undecided + (or already-``fail``) criterion in ``indices`` is credited nothing (the + pessimistic assumption that it could still end up scoring 0); only an + already-``pass`` criterion contributes its weight. Returns ``None`` + when the subset's total weight is 0 (the vacuous case — nothing to + bound), so callers don't have to special-case an empty numerator over + an empty denominator. + """ + weight = sum(self._armed[i][0].weight for i in indices) + if weight <= 0.0: + return None + return sum(self._armed[i][0].weight for i in indices if verdicts[i] == "pass") / weight def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: + """Fail-open wrapper: any unexpected exception anywhere in the round + disarms the watcher and degrades to a full run, exactly like a raising + ``live_verdict`` — not just the verdict-collection loop. The + ceiling/floor arithmetic below is currently guarded (an empty armed + set is unreachable, and every candidate short-circuits before + dividing), but a future change to that arithmetic — or to + ``_fire`` — must not be able to leave the watcher stuck re-raising on + every subsequent event with ``_disarmed`` still False (which is what a + narrower try/except would risk). + """ + try: + self._evaluate_impl(in_flight) + except Exception: + self._disarmed = True + logger.error( + "[%s] early-stop round raised unexpectedly; disarming watcher, run degrades to a full run", + self._task_id, + exc_info=True, + ) + + def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: record = self._collector.build_turn_record() if in_flight is not None: # The in-flight call has no ToolEnd yet, so the collector (which @@ -387,43 +495,71 @@ def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: try: verdicts.append(checker.live_verdict(criterion, records)) except Exception: - # Fail-open: a raising verdict disarms; the run degrades to full. - self._disarmed = True + # Re-raise to the wrapping try/except in _evaluate, which sets + # _disarmed — but log the specific criterion here first, since + # that context (which criterion's live_verdict raised) would + # otherwise be lost once the exception is caught generically. logger.error( - "[%s] early-stop live_verdict raised for criterion %r; disarming watcher, " - + "run degrades to a full run", + "[%s] early-stop live_verdict raised for criterion %r", self._task_id, criterion.type, exc_info=True, ) - return + raise pass_armed = [i for i, pol in enumerate(self._armed_polarities) if "pass" in pol] - # Fail-stop: first armed criterion (criteria order) that live-fails AND - # whose resolved arming permits fail decides the run — but DEFERRED while - # any pass-armed criterion is still undecided. Cutting a positive row on a - # distractor misfire before its expected signal could appear would freeze a - # would-be TP as an FN (truncating the suite's recall); the misfire is - # latched by the criterion's own monotone semantics, so the deferred fail - # still fires the moment every pass-armed criterion decides, and a row with - # zero pass-armed criteria (a negative row) defers nothing. + # Fail-stop: at least one armed criterion (criteria order) that live-fails + # AND whose resolved arming permits fail is a CANDIDATE — but the stop only + # actually fires once the ceiling bound (best case: every still-undecided + # or already-passed armed criterion ends up scoring 1.0, every live-failed + # one scores 0) can no longer reach ``gate_threshold``, i.e. the armed gate + # (``EvaluationResult.armed_criteria_passed``) is GUARANTEED to fail no + # matter what happens on the rest of the trajectory. At the default + # ``gate_threshold=1.0`` this is equivalent to firing on the first + # candidate (any armed criterion's weight is > 0 by construction, so a + # single fail already drops the ceiling below 1.0) — below 1.0 a + # low-weight candidate's failure may not be enough to doom the gate, so the + # run keeps going. This is DEFERRED while any pass-armed criterion is still + # undecided. Cutting a positive row on a distractor misfire before its + # expected signal could appear would freeze a would-be TP as an FN + # (truncating the suite's recall); the misfire is latched by the + # criterion's own monotone semantics, so the deferred fail still fires the + # moment every pass-armed criterion decides, and a row with zero + # pass-armed criteria (a negative row) defers nothing. if not any(verdicts[i] == "undecided" for i in pass_armed): - for (criterion, _checker), verdict, armed_pol in zip( - self._armed, verdicts, self._armed_polarities, strict=True - ): - if verdict == "fail" and "fail" in armed_pol: - self._fire(EarlyStopReason.CRITERION_FAILED, criterion, tool_call_index=tool_call_index) - return - - # Pass-stop: every PASS-ARMED criterion live-passes. Fail-armed criteria - # (e.g. distractors armed ``auto`` -> fail only) are NOT required to pass — - # they can never live-pass and only guard the fail side above, so requiring - # them would veto every pass-stop (the mixed-arming bug). Guard the vacuous - # case: with zero pass-armed criteria (a negative row whose criteria are all - # distractors) there is nothing to pass-stop on, so the run must continue to - # the cap rather than firing on turn 0 with an empty ``all()``. - if pass_armed and all(verdicts[i] == "pass" for i in pass_armed): + candidate = next( + ( + criterion + for (criterion, _checker), verdict, armed_pol in zip( + self._armed, verdicts, self._armed_polarities, strict=True + ) + if verdict == "fail" and "fail" in armed_pol + ), + None, + ) + if candidate is not None and self._ceiling(verdicts) < self._gate_threshold: + self._fire(EarlyStopReason.CRITERION_FAILED, candidate, tool_call_index=tool_call_index) + return + + # Pass-stop: the PASS-ARMED subset's own floor bound (worst case: every + # still-undecided pass-armed criterion ends up scoring 0, weighted against + # only the pass-armed subset's total weight) already meets + # ``gate_threshold`` — guaranteed regardless of what the rest of that + # subset still decides. Fail-armed criteria (e.g. distractors armed + # ``auto`` -> fail only) are excluded from both the numerator and the + # denominator: they can never live-pass and only guard the fail side + # above, so folding them in would veto every pass-stop (the mixed-arming + # bug) and penalize this bound for a criterion it was never scoped to + # cover. At the default ``gate_threshold=1.0`` this requires every + # pass-armed criterion to actually be "pass" (any non-pass drops the floor + # below 1.0) — identical to the pre-weighting ``all(...)`` rule. Guard the + # vacuous case: with zero pass-armed criteria (a negative row whose + # criteria are all distractors) there is nothing to pass-stop on, so the + # run must continue to the cap rather than firing on turn 0 with an empty + # numerator/denominator. + floor = self._floor(verdicts, pass_armed) + if floor is not None and floor >= self._gate_threshold: # Deciding criterion = the last pass-armed (criteria order) whose verdict # flipped vs the previous round; fall back to the last pass-armed. deciding = self._armed[pass_armed[-1]][0] @@ -433,10 +569,25 @@ def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: self._fire(EarlyStopReason.CRITERION_PASSED, deciding, tool_call_index=tool_call_index) return + # Decision-step budget: an armed criterion with max_steps_to_decide set + # that is STILL "undecided" once that many tool-call steps have elapsed + # forces a hard fail — checked last, after the real fail-/pass-stop + # checks above, so a criterion that decides on this very round (however + # late) is never punished for a budget it technically exceeded. It never + # reached a verdict at all, so there is nothing meaningful to weigh it + # against — this is why DECISION_BUDGET_EXCEEDED bypasses the weighted + # gate entirely at the orchestrator finalize step rather than folding + # into the ceiling/floor bounds above. + for (criterion, _checker), verdict in zip(self._armed, verdicts, strict=True): + budget = criterion.max_steps_to_decide + if budget is not None and verdict == "undecided" and tool_call_index >= budget: + self._fire(EarlyStopReason.DECISION_BUDGET_EXCEEDED, criterion, tool_call_index=tool_call_index) + return + # No stop this round — record the verdicts so the next round can detect flips. self._prev_verdicts = verdicts - def _fire(self, reason: EarlyStopReason, criterion: BaseSuccessCriterion, *, tool_call_index: int) -> None: + def _fire(self, reason: EarlyStopReason, criterion: LiveSuccessCriterion, *, tool_call_index: int) -> None: elapsed = 0.0 if self._started_monotonic is not None: elapsed = max(time.monotonic() - self._started_monotonic, 0.0) @@ -450,6 +601,7 @@ def _fire(self, reason: EarlyStopReason, criterion: BaseSuccessCriterion, *, too tool_call_index=tool_call_index, elapsed_seconds=elapsed, turns_remaining_at_stop=turns_remaining, + gate_threshold=self._gate_threshold, ) logger.info( "[%s] early-stop fired: reason=%s deciding=%s sdk_turn=%d tool_call=%d elapsed=%.2fs", diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 6d73840d..138ad722 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -11,7 +11,7 @@ from datetime import datetime from inspect import isawaitable from pathlib import Path -from typing import Any +from typing import Any, assert_never from urllib.parse import urlparse from .agent import Agent @@ -37,6 +37,7 @@ ConfigLineageEntry, CriterionResult, DirectRoute, + EarlyStopReason, EvaluationResult, FinalStatus, JudgeCriterionResult, @@ -1567,17 +1568,59 @@ async def _evaluation_loop(self) -> bool: pairs = list(zip(criteria_results, self.task.success_criteria, strict=True)) passed_count = sum(1 for r, c in pairs if r.score >= c.pass_threshold) total_count = len(pairs) - if self.result.early_stop is not None: - # Early-stopped run: only the armed subset gates final_status; the rest - # are advisory (recorded, never decisive) so a smoke flavor is not - # dragged to FAILURE by criteria whose work it deliberately skipped. - all_passed = self.result.armed_criteria_passed(self.task.success_criteria) - armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) - logger.info( - "Early-stopped run: gating on %d armed criteria (%d advisory, not gated).", - armed_count, - total_count - armed_count, - ) + # The armed weighted gate governs any task armed for early-stop + # (run_limits.stop_early: true), whether or not the watcher actually + # fired — one task config maps to one gate semantic. Without this, a + # run that happened to finish before the bound ever tripped would + # silently fall back to the strict full-set gate, re-failing on a + # low-weight criterion the weighted gate was configured to forgive, + # for a reason (incidental control flow) unrelated to configured + # intent. Only a task NOT armed for early-stop uses the full, + # strict-AND gate over every gating criterion. + if self.task.run_limits is not None and self.task.run_limits.stop_early: + if self.result.early_stop is not None: + reason = self.result.early_stop.reason + # Exhaustive on EarlyStopReason: adding a 4th member without a + # branch here is a type error (assert_never), not a silent + # fall-through into the weighted-gate branch below. + if reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED: + # Hard fail, bypassing the weighted gate entirely: the deciding + # criterion never reached a verdict within its max_steps_to_decide + # budget, so there is nothing meaningful to weigh it against. + all_passed = False + logger.info( + "Early-stopped run: decision-step budget exceeded for %r, forcing FAILURE.", + self.result.early_stop.deciding_criterion_description, + ) + elif reason == EarlyStopReason.CRITERION_PASSED or reason == EarlyStopReason.CRITERION_FAILED: + all_passed = self.result.armed_criteria_passed( + self.task.success_criteria, self.task.run_limits.stop_early_gate_threshold + ) + armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) + logger.info( + "Early-stopped run: gating on %d armed criteria (%d advisory, not gated).", + armed_count, + total_count - armed_count, + ) + else: + assert_never(reason) + else: + # Armed for early-stop but the watcher never fired (the agent + # finished, or max_turns was hit, before the bound tripped): + # only the armed subset gates final_status; the rest are + # advisory (recorded, never decisive) — same gate as an + # actual early stop, so a smoke flavor is not dragged to + # FAILURE by criteria whose work it deliberately skipped. + all_passed = self.result.armed_criteria_passed( + self.task.success_criteria, self.task.run_limits.stop_early_gate_threshold + ) + armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) + logger.info( + "stop_early armed but never fired (run completed naturally): gating on " + + "%d armed criteria (%d advisory, not gated).", + armed_count, + total_count - armed_count, + ) else: all_passed = self.result.all_criteria_passed(self.task.success_criteria) diff --git a/src/coder_eval/reports.py b/src/coder_eval/reports.py index 743dcf92..b55f6f8c 100644 --- a/src/coder_eval/reports.py +++ b/src/coder_eval/reports.py @@ -438,10 +438,14 @@ def _runtime_notes_lines(summary: RunSummary) -> list[str]: reason = t.get("early_stop_reason") or "unknown" turns_remaining = t.get("turns_remaining_at_stop") avoided = f" <= {turns_remaining} turn(s) avoided —" if isinstance(turns_remaining, int) else "" - notes.append( - f"> **NOTE:** [{task_id}] stopped early ({reason});{avoided}" - + " gated on armed criteria only; other criteria are advisory" - ) + if reason == "decision_budget_exceeded": + # No criterion gated here at all — an armed criterion's + # decision-step budget expired unresolved, forcing FAILURE + # outright, bypassing the weighted gate entirely. + gate_note = " forced to FAILURE (decision-step budget exceeded, bypassing the gate)" + else: + gate_note = " gated on armed criteria only; other criteria are advisory" + notes.append(f"> **NOTE:** [{task_id}] stopped early ({reason});{avoided}" + gate_note) if not notes: return [] return ["## Run-time Notes", "", *notes] diff --git a/src/coder_eval/reports_experiment.py b/src/coder_eval/reports_experiment.py index e362d271..8f2ec675 100644 --- a/src/coder_eval/reports_experiment.py +++ b/src/coder_eval/reports_experiment.py @@ -210,6 +210,10 @@ def eval_result_to_task_dict( "turns_remaining_at_stop": ( result.early_stop.turns_remaining_at_stop if result.early_stop is not None else None ), + # The threshold in effect for this stop, so a downstream consumer + # comparing early-stopped runs across an experiment sweep that varies + # it can tell which weighted-gate value produced a given verdict. + "gate_threshold": (result.early_stop.gate_threshold if result.early_stop is not None else None), } d["variant_id"] = variant_id return d diff --git a/src/coder_eval/reports_html.py b/src/coder_eval/reports_html.py index 3fef5d12..e5a2c048 100644 --- a/src/coder_eval/reports_html.py +++ b/src/coder_eval/reports_html.py @@ -17,7 +17,7 @@ from pathlib import Path from typing import TYPE_CHECKING, Any -from coder_eval.models import FinalStatus, eval_result_total_cost, sum_costs +from coder_eval.models import EarlyStopReason, FinalStatus, eval_result_total_cost, sum_costs if TYPE_CHECKING: @@ -345,9 +345,15 @@ def _render_header(result: EvaluationResult) -> str: expected_turns_badge = f'expected_turns exceeded ({actual}/{expected})' early_stop_badge = "" if result.early_stop is not None: + if result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED: + # No criterion gated here at all — forced to FAILURE outright, + # bypassing the weighted gate. + title = "Forced to FAILURE (decision-step budget exceeded, bypassing the gate)" + else: + title = "Gated on armed criteria only; other criteria are advisory" early_stop_badge = ( - 'stopped early ({_esc(result.early_stop.reason.value)})' + f'' + + f"stopped early ({_esc(result.early_stop.reason.value)})" ) return f"""