diff --git a/CLAUDE.md b/CLAUDE.md index 16ef30cb..664499d3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -141,8 +141,8 @@ action.yml # Published composite GitHub Action (coder-ev - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection. - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports. - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly. -- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. -- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm (non-empty `live_stop_polarities` ClassVar + `live_verdict` override — currently `skill_triggered`, `command_executed`; CE025 keeps the two consistent). `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule: the pass-stop fires when every **pass-armed** criterion live-passes (fail-armed distractors are not required to pass; zero pass-armed ⇒ never pass-stops); the fail-stop fires on the first fail-armed live-fail but is **deferred while any pass-armed criterion is undecided** — a distractor misfire must not truncate a positive row's recall signal, so the latched misfire fires once the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** (`EvaluationResult.armed_criteria_passed`); a completed run gates on the full set. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo`, report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged. +- **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `max_steps_to_decide`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step-budget breach to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express. +- **Early stop on criterion (opt-in)**: `run_limits.stop_early` (default off) ends a single-shot Claude run early once the run's **armed** criteria are decided, so a raised `max_turns` isn't wasted on the smoke flavor. A criterion is armed by `stop_when: pass|fail|decided|auto`; only criteria that can decide from a partial trajectory may arm — "is this criterion type live-observable" is `models.LiveSuccessCriterion` subclassing (currently `skill_triggered`, `command_executed`), the single source of truth `validate_early_stop`/`EarlyStopWatcher` check directly via `isinstance`; each subclass implements the abstract, checker-independent `live_decidable_polarities()` (a pure function of its own fields) alongside the checker's `live_verdict` override, and lint rule CE025 (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`, a registry-based whole-tree check, not a per-file AST rule) keeps the two paired. `decided` arms **both** polarities; `auto` arms whichever polarities **this instance** can decide — the value for dataset-fanned criteria whose positive/distractor role flips per row. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing pre-weighting behavior byte-for-byte) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed subset's **ceiling** (best case for everything still undecided) can no longer reach the threshold, a pass-stop once the pass-armed subset's **floor** (worst case) already meets it — both **deferred while any pass-armed criterion is undecided**, so a distractor misfire never truncates a positive row's recall signal before the positives resolve (or the run continues to the cap). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a `stop_early: false` run. A per-criterion `max_steps_to_decide` (on `LiveSuccessCriterion` only, requires `stop_when`) caps tool-call steps spent still undecided — cumulative across retry attempts of the same turn — before `EarlyStopReason.DECISION_BUDGET_EXCEEDED` force-fails the run outright, bypassing the weighted gate (nothing to weigh a criterion that never decided against). Driven by `orchestration/early_stop.py::EarlyStopWatcher` through the Claude agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. An early-stopped run gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally (even with `stop_early: true`) gates on the full set via the strict-AND `all_criteria_passed` — weight magnitude only forgives under the former, so the weighted gate is contingent on the watcher itself firing, not solely on the configured threshold. Every unsupported use is a hard error at resolution (plan *and* run), and a runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. Defaults off ⇒ behavior byte-for-behavior unchanged. ## Success Criteria (14 types) diff --git a/docs/EXTENDING.md b/docs/EXTENDING.md index aa3ba39e..288cabf2 100644 --- a/docs/EXTENDING.md +++ b/docs/EXTENDING.md @@ -210,8 +210,16 @@ Notes: `count/mean/median/std/min/max`, so your criterion is suite-thresholdable for free. Classification-style criteria return a `ClassificationCriterionResult` and layer accuracy / precision / recall / F1 / confusion on top. -- For **early stop**, implement `live_verdict(...)` and declare - `live_stop_polarities` — a lint rule keeps the two consistent. +- For **early stop**, make your criterion model subclass `LiveSuccessCriterion` + (`models/criteria.py`) instead of `BaseSuccessCriterion`, implement its + abstract `live_decidable_polarities()` (a pure function of the criterion's + own fields — no `turn_records`, no checker instance), and override the + checker's `live_verdict(...)`. `LiveSuccessCriterion` subclassing is the + single source of truth for "is this criterion type live-observable" — + `validate_early_stop`/`EarlyStopWatcher` check `isinstance(c, + LiveSuccessCriterion)` directly, no separate checker-side flag. A lint rule + (`tests/test_custom_lint.py::TestCE025LiveVerdictConsistency`) keeps the + model subclassing and the checker's `live_verdict` override paired. > A duplicate `criterion_type` **overwrites** the earlier checker with a warning (not > a hard error, unlike agents) — keep type strings unique. diff --git a/docs/REPORT_SCHEMA.md b/docs/REPORT_SCHEMA.md index ef38b1a2..3fb492c5 100644 --- a/docs/REPORT_SCHEMA.md +++ b/docs/REPORT_SCHEMA.md @@ -188,10 +188,13 @@ backend), `num_turns`, `max_turns_exhausted`, ### EarlyStopInfo Present (non-`null`) iff the run stopped early — there is no separate boolean. -Fields: `reason` (`criterion_passed` / `criterion_failed`), +Fields: `reason` (`criterion_passed` / `criterion_failed` / +`decision_budget_exceeded` — the last forces `FinalStatus.FAILURE` outright, +bypassing the weighted gate), `deciding_criterion_type`, `deciding_criterion_description`, `armed_criteria`, `sdk_turn_index`, `tool_call_index` (1-based, includes the in-flight call), -`elapsed_seconds`, `turns_remaining_at_stop`. +`elapsed_seconds`, `turns_remaining_at_stop`, `gate_threshold` (the +`run_limits.stop_early_gate_threshold` in effect for this stop; default `1.0`). --- diff --git a/docs/TASK_DEFINITION_GUIDE.md b/docs/TASK_DEFINITION_GUIDE.md index c2de203a..69cbde15 100644 --- a/docs/TASK_DEFINITION_GUIDE.md +++ b/docs/TASK_DEFINITION_GUIDE.md @@ -260,6 +260,7 @@ run_limits: | `count_cached_input` | `false` | — | Count `cache_read_input_tokens` toward the input/total budgets. Off by default — cached reads are typically free. | | `count_cache_creation` | `false` | — | Count `cache_creation_input_tokens` toward the input/total budgets. Off by default. | | `stop_early` | `false` | — | Opt-in master switch for early-stop-on-criterion. See [`stop_early`](#stop_early-opt-in-early-stop). | +| `stop_early_gate_threshold` | `1.0` | `[0.0, 1.0]` (but `> 0.0` is enforced at resolution when `stop_early: true`) | Minimum weighted score over the armed subset required to gate as a pass. See [`stop_early`](#stop_early-opt-in-early-stop). | The authoritative source is `src/coder_eval/models/limits.py`. A lint rule (CE030) fails the build if a field defined there goes undocumented in this guide, so the table can't quietly fall behind the @@ -376,11 +377,14 @@ Semantics: (e.g. `stop_when: pass` alongside a `max_count`, or `auto` on an instance that can decide neither) is likewise a hard error at resolution, not a silent full run. -- **Verdict.** An early-stopped run is gated on the **armed subset only**; the - non-armed criteria become **advisory** and are clearly marked (report badge + - per-criterion note + `stopped_early` row). A run that completes naturally is - gated on the **full** set, as always. This is what lets one file serve both a - `smoke` flavor (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) — +- **Verdict.** Any task armed for early-stop (`stop_early: true`) is gated on + the **armed subset only** — the non-armed criteria become **advisory** and + are clearly marked (report badge + per-criterion note + `stopped_early` + row when the watcher actually fired) — whether or not the watcher actually + cut the run short; one task config maps to one gate semantic. Only a task + that never armed `stop_early` at all is gated on the **full** set, as + always. This is what lets one file serve both a `smoke` flavor + (`stop_early: true`) and an `e2e` flavor (`stop_early: false`) — see [AB_EXPERIMENTS.md](AB_EXPERIMENTS.md). Verdict parity between the flavors is one-sided: a **fail-stop** is verdict-preserving (the deferral above guarantees every pass-armed signal was allowed to resolve first), but a @@ -390,6 +394,57 @@ Semantics: authoritative precision/recall belongs on the `stop_early: false` run. - **Fail-safe.** A live-verdict bug **fails open** to a full run (logged loudly) — it can never silently disable a criterion or cause a false early stop. +- **Weighting.** `run_limits.stop_early_gate_threshold` (default `1.0`) is the + minimum weighted score (`Σ weight·score / Σ weight`, over the armed subset) + required to gate as a pass — both for the post-hoc verdict and for the live + stop rule itself. A fail-stop fires once the armed subset's **ceiling** (best + case: every still-undecided or already-passed criterion ends up scoring 1.0, + every live-failed one scores 0) can no longer reach the threshold — the gate + is mathematically guaranteed to fail regardless of how the trajectory + continues. A pass-stop fires once the pass-armed subset's **floor** (worst + case: every still-undecided one scores 0) already meets it. At the default + `1.0` both bounds collapse to the pre-weighting rules above exactly (any + single armed criterion's live-fail already drops the ceiling below 1.0, and + the floor only reaches 1.0 once every pass-armed criterion has actually + passed) — lowering it lets a low-weight armed criterion's failure be absorbed + without truncating the run, at the cost of the gate becoming a genuine + weighted average rather than a strict AND. **The armed weighted gate applies + whenever `stop_early: true` is set — one task config, one gate semantic — + regardless of whether the watcher actually fired a stop.** A task armed for + early-stop that instead completes naturally (the agent finishes, or + `max_turns` is hit, before the bound ever trips) is gated on the *same* + weighted armed-subset formula as an actual early stop, not the full-run + `all_criteria_passed`; only a task that never armed `stop_early` at all uses + the strict full-set gate. Each armed criterion's own `pass_threshold` still + decides whether it individually passed (converted to a binary 1.0/0.0 + before weighting) — only the combination rule (weighted average vs strict + AND) changes, which is what makes the `gate_threshold=1.0` default an exact + equivalence with the pre-weighting `all(...)` rule. +- **Decision-step budget.** `max_steps_to_decide` (per armed criterion, only + on `skill_triggered` / `command_executed`, requires `stop_when`) caps how + many tool-call steps that criterion may spend still **undecided** before the + run gives up on it: + + ```yaml + success_criteria: + - type: skill_triggered + description: "date-teller must activate within 5 steps" + skill_name: date-teller + expected_skill: date-teller + stop_when: pass + max_steps_to_decide: 5 + ``` + + Once the cap is exceeded (checked AFTER the normal fail-/pass-stop checks + each round, so a criterion that decides on that very step is never + penalized), the watcher fires `reason: decision_budget_exceeded` and the run + is forced to `FinalStatus.FAILURE` outright — bypassing + `stop_early_gate_threshold`'s weighted gate entirely, since a criterion that + never reached a verdict has nothing meaningful to weigh against the others. + `None` (default) = no cap; the run relies solely on `run_limits.max_turns`. + The step count is **cumulative across every retry attempt** of the turn — + including an attempt that crashed or timed out before this criterion's own + investigation even began — so size the budget with that headroom in mind. Observability (every early-stopped run is flagged everywhere so analysis never compares a truncated run against a full one): @@ -535,6 +590,7 @@ All criteria share these fields: | `weight` | 1.0 | Relative importance for weighted score. `0` = **informational**: excluded from both the score and the pass/fail gate | | `pass_threshold` | 0.9 | Minimum score (0.0–1.0) to pass | | `stop_when` | `null` | Arms this criterion for early stop (`pass`/`fail`/`decided`/`auto`); requires `run_limits.stop_early: true` and an observable criterion type (`skill_triggered`, `command_executed`). `auto` arms whichever polarity this instance can decide (for dataset-fanned criteria whose positive/distractor role flips per row). See [`stop_early`](#stop_early-opt-in-early-stop). | +| `max_steps_to_decide` | `null` | **Only on live-observable criteria** (`skill_triggered`, `command_executed`) — requires `stop_when` to be set. Caps the tool-call steps this armed criterion may spend still undecided before the run gives up and force-fails. See [`stop_early`](#stop_early-opt-in-early-stop). | **Scoring types:** - **Binary** (1.0 or 0.0): `file_exists`, `run_command`, `file_matches_regex`, `classification_match`, `skill_triggered` diff --git a/src/coder_eval/criteria/base.py b/src/coder_eval/criteria/base.py index 503c31f1..cbe7b9a8 100644 --- a/src/coder_eval/criteria/base.py +++ b/src/coder_eval/criteria/base.py @@ -26,6 +26,19 @@ # A criterion's verdict from a PARTIAL, mid-run trajectory (early-stop observability). # "undecided" means the outcome is not yet knowable from the events seen so far. +# +# CONTRACT every live_verdict override must satisfy (see BaseCriterion.live_verdict): +# - Deterministic: a pure function of the ``turn_records`` prefix passed in — no +# wall-clock, randomness, or other hidden state. +# - Monotonic: once it returns "pass"/"fail" for some trajectory prefix, it MUST +# return that SAME verdict for every longer prefix (i.e. every later call in the +# same run). "undecided" is the only verdict allowed to change on a later call. +# EarlyStopWatcher's deferred fail-stop and pass/fail flip-attribution +# (early_stop.py::_prev_verdicts) are correct only because both existing +# implementations (skill_triggered, command_executed) honor this. A non-monotonic or +# non-deterministic override compiles and passes CE025 (which only checks +# LiveSuccessCriterion subclassing / live_verdict pairing, not this) but silently corrupts the stop +# logic — there is currently no automated enforcement beyond this docstring. LiveVerdict = Literal["pass", "fail", "undecided"] @@ -200,13 +213,6 @@ def _check_impl( # Subclasses MUST define this as a class variable criterion_type: ClassVar[str] - # Which polarities this criterion can decide from a PARTIAL, mid-run trajectory. - # Empty (base default) = not observable mid-run, so it can never arm early-stop. - # A subclass that reads only turn_records and can decide mid-run declares the - # polarities it supports (e.g. frozenset({"pass", "fail"})) AND overrides - # live_verdict; CE025 enforces that the two stay consistent. - live_stop_polarities: ClassVar[frozenset[str]] = frozenset() - def __new__(cls, *args: Any, **kwargs: Any) -> "BaseCriterion[C]": """Block direct instantiation of ``BaseCriterion`` itself. @@ -426,36 +432,17 @@ def live_verdict( from ``check()``/``_check_impl`` run on the frozen trajectory after the stop, so a live/final divergence can never corrupt scoring. - Base default: ``"undecided"`` (not observable mid-run). Subclasses that - override this MUST also declare a non-empty ``live_stop_polarities`` (and - vice versa) — enforced by lint rule CE025. + Base default: ``"undecided"`` (not observable mid-run). A checker + overrides this iff its criterion model is a ``LiveSuccessCriterion`` + subclass (``models/criteria.py``) — that subclassing is the single + source of truth for "is this criterion type live-observable", checked + by ``validate_early_stop`` / ``EarlyStopWatcher`` and enforced by lint + rule CE025. An override MUST also satisfy the deterministic + monotonic + contract documented on the ``LiveVerdict`` type above (not enforced by + CE025 or any other automated check). """ return "undecided" - @classmethod - def live_decidable_polarities(cls, criterion: C) -> frozenset[str]: - """Which polarities THIS criterion *instance* can actually decide mid-run. - - ``live_stop_polarities`` is a class-level *capability* — the widest set - of polarities the checker's ``live_verdict`` could ever emit. But for - some criteria whether a given polarity can fire depends on the instance's - configuration, not just its type. ``command_executed`` is the canonical - case: it can live-``pass`` only with no upper bound, and live-``fail`` - only with one, so a specific criterion may support strictly fewer - polarities than its class advertises (down to none — a "dead arm"). - - ``validate_early_stop`` gates the requested ``stop_when`` polarity on THIS - set, not the ClassVar, so an instance that can never decide its armed - polarity is rejected at resolution rather than silently degrading to a - full run (the "never a silent no-op" guarantee). - - Default: the class-level ``live_stop_polarities`` — correct for every - criterion whose decidability is purely type-level (e.g. ``skill_triggered``). - Overrides MUST return a subset of ``live_stop_polarities`` (a criterion - cannot decide a polarity its ``live_verdict`` never emits). - """ - return cls.live_stop_polarities - def aggregate( self, criterion: C, diff --git a/src/coder_eval/criteria/command_executed.py b/src/coder_eval/criteria/command_executed.py index 92d06ec6..e5343ef2 100644 --- a/src/coder_eval/criteria/command_executed.py +++ b/src/coder_eval/criteria/command_executed.py @@ -3,7 +3,7 @@ import json import logging import re -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING from coder_eval.criteria.base import BaseCriterion, CheckContext, LiveVerdict, register_criterion from coder_eval.models import CommandExecutedCriterion, CriterionResult @@ -30,42 +30,6 @@ class CommandExecutedChecker(BaseCriterion[CommandExecutedCriterion]): criterion_type = "command_executed" - # Observable mid-run: command matches accumulate monotonically in the live - # stream, so a min_count pass (no upper bound) and a max_count exceedance - # (incl. the must-NOT-run 0/0 form) are both decidable before end-of-run. - live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"}) - - @classmethod - def live_decidable_polarities(cls, criterion: CommandExecutedCriterion) -> frozenset[str]: - """Narrow the class capability to what THIS instance can decide mid-run. - - The class advertises ``{"pass", "fail"}``, but ``live_verdict`` can only: - - - ``pass`` when there is no upper bound and a positive floor - (``max_count is None and min_count > 0``) — with an upper bound a pass - is not final until end-of-run, so it never fires live; and - - ``fail`` when there IS an upper bound (``max_count is not None``), the - moment the count exceeds it (this includes the ``min_count: 0, - max_count: 0`` "must NOT run" form). - - So these instance shapes are dead arms the class-level check misses: - - - ``stop_when: pass`` with ``max_count`` set → pass can never fire; - - ``stop_when: fail`` with ``max_count: None`` → fail can never fire; - - ``min_count: 0, max_count: None`` → neither can ever fire. - - Reporting the true per-instance set here lets ``validate_early_stop`` - reject such arming at resolution instead of silently degrading to a full - run. Stays a subset of ``live_stop_polarities`` by construction. - """ - decidable: set[str] = set() - if criterion.max_count is None: - if criterion.min_count > 0: - decidable.add("pass") - else: - decidable.add("fail") - return frozenset(decidable) - @staticmethod def _matching_commands( criterion: CommandExecutedCriterion, diff --git a/src/coder_eval/criteria/skill_triggered.py b/src/coder_eval/criteria/skill_triggered.py index 999d8ce5..2b1ded9c 100644 --- a/src/coder_eval/criteria/skill_triggered.py +++ b/src/coder_eval/criteria/skill_triggered.py @@ -10,7 +10,7 @@ import logging import re -from typing import TYPE_CHECKING, ClassVar +from typing import TYPE_CHECKING from coder_eval.criteria._classification_aggregate import overlay_classification_metrics from coder_eval.criteria.base import BaseCriterion, LiveVerdict, register_criterion @@ -109,14 +109,6 @@ class SkillTriggeredChecker(BaseCriterion[SkillTriggeredCriterion]): criterion_type = "skill_triggered" - # Observable mid-run: a Skill tool call (or a skill file read) is a positive - # event in the live stream. The TYPE can decide either polarity — a positive - # criterion live-passes when its expected skill is engaged, a - # distractor/negative one live-fails when its (wrong) skill is engaged — but - # any single INSTANCE decides only one of the two; see - # ``live_decidable_polarities``. - live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"}) - def _check_impl( self, criterion: SkillTriggeredCriterion, @@ -179,37 +171,15 @@ def live_verdict( frozen trajectory by construction — whether or not the run stopped early. A positive criterion can therefore only ever live-``pass`` and a distractor/negative one only ever live-``fail``; their *absence* is never - decidable mid-run (see ``live_decidable_polarities``). This is the change - from first-engagement: a wrong skill engaged first no longer live-fails a - positive row — the run keeps going so the expected skill can still load. + decidable mid-run (see ``SkillTriggeredCriterion.live_decidable_polarities`` + in models/criteria.py). This is the change from first-engagement: a wrong + skill engaged first no longer live-fails a positive row — the run keeps + going so the expected skill can still load. """ if criterion.skill_name not in _all_engaged_skill_names(turn_records): return "undecided" return "pass" if criterion.expected_skill == criterion.skill_name else "fail" - @classmethod - def live_decidable_polarities(cls, criterion: SkillTriggeredCriterion) -> frozenset[str]: - """Per-instance narrowing under the any-engagement latch. - - Unlike the type-level capability (``live_stop_polarities`` = both), a - single instance decides exactly one polarity: - - - a **positive** criterion (``skill_name == expected_skill``) can only - live-``pass`` (the expected skill engaging is a decidable hit; its - absence is not knowable mid-run); - - a **distractor/negative** criterion (``skill_name != expected_skill``, - including the ``expected_skill == ""`` negatives) can only - live-``fail`` (a wrong skill engaging is a decidable miss; its absence - is not). - - ``validate_early_stop`` gates the requested ``stop_when`` on this set, so - arming a positive with ``fail`` / a distractor with ``pass`` — or either - with ``decided`` (which needs both) — is rejected at resolution rather - than silently degrading to a full run. - """ - expected_yes = criterion.expected_skill == criterion.skill_name - return frozenset({"pass"}) if expected_yes else frozenset({"fail"}) - def aggregate( self, criterion: SkillTriggeredCriterion, diff --git a/src/coder_eval/models/__init__.py b/src/coder_eval/models/__init__.py index 1d155604..47927cd0 100644 --- a/src/coder_eval/models/__init__.py +++ b/src/coder_eval/models/__init__.py @@ -40,6 +40,8 @@ FileMatchesRegexCriterion, JMESPathAssertion, JsonCheckCriterion, + LivePolarity, + LiveSuccessCriterion, LLMJudgeCriterion, ReferenceComparisonCriterion, RegexPattern, @@ -232,6 +234,8 @@ "LLMJudgeCriterion", "AgentJudgeCriterion", "SkillTriggeredCriterion", + "LiveSuccessCriterion", + "LivePolarity", "SuccessCriterion", # Routing "ROUTE_NAMES", diff --git a/src/coder_eval/models/criteria.py b/src/coder_eval/models/criteria.py index 2605e1cf..26a1142c 100644 --- a/src/coder_eval/models/criteria.py +++ b/src/coder_eval/models/criteria.py @@ -8,7 +8,7 @@ from __future__ import annotations -from abc import ABC +from abc import ABC, abstractmethod from typing import Annotated, Any, ClassVar, Literal, Self from pydantic import BaseModel, ConfigDict, Field, model_validator @@ -187,6 +187,86 @@ def is_gating(self) -> bool: # Business logic (check operations) moved to SuccessChecker in evaluator.py +# The two polarities a live-observable criterion can decide mid-run — distinct +# from the 3-value LiveVerdict ("pass"/"fail"/"undecided") the checker's +# live_verdict returns: this is the narrower CAPABILITY type, "undecided" is +# never a valid decidable polarity. Typed here (not a bare frozenset[str]) so +# a live_decidable_polarities override returning a stray/typo'd string, or +# "undecided" itself, is a pyright error rather than a runtime-only lint gap. +LivePolarity = Literal["pass", "fail"] + + +class LiveSuccessCriterion(BaseSuccessCriterion): + """Base for criteria observable from a PARTIAL, mid-run trajectory (early-stop). + + ``live_decidable_polarities`` is a pure function of THIS instance's own + fields — no ``turn_records``, no sandbox, no checker instance needed (e.g. + ``command_executed`` can decide this purely from whether ``max_count`` is + set). That makes it genuinely computable on the data model rather than the + checker, unlike the checker's ``live_verdict`` (``criteria/base.py``), + which reads the actual trajectory and stays checker-side logic. Moving + decidability here also gives early-stop-only config (e.g. + ``max_steps_to_decide``) a home that doesn't pollute ``BaseSuccessCriterion`` + with a field meaningless for every non-observable criterion type. + + Only ``SkillTriggeredCriterion`` / ``CommandExecutedCriterion`` subclass + this today; a criterion type is "live-observable" iff it is a + ``LiveSuccessCriterion`` subclass — the single source of truth + ``validate_early_stop`` / ``EarlyStopWatcher`` consult (no separate + checker-side flag to keep in sync). + """ + + max_steps_to_decide: int | None = Field( + default=None, + ge=1, + description=( + "Cap on tool-call steps this ARMED criterion (stop_when must be set) " + "may spend still 'undecided' before the run gives up on it. Once " + "exceeded, EarlyStopWatcher fires an early stop with reason " + "'decision_budget_exceeded' and the run is forced to FinalStatus." + "FAILURE outright — regardless of what any other armed criterion's " + "weighted score would otherwise gate to (this criterion never " + "reached a verdict at all, so there is nothing to weigh). None " + "(default) = no cap; the run relies solely on run_limits.max_turns. " + "Requires run_limits.stop_early and this criterion's own stop_when. " + "The step count is CUMULATIVE across every retry attempt of the " + "turn (the same EarlyStopWatcher instance, and its counters, " + "persist across retries) — including attempts that ultimately " + "crashed or timed out before this criterion's own investigation " + "even began. Size the budget with that headroom in mind." + ), + ) + + @model_validator(mode="after") + def _check_max_steps_requires_armed(self) -> Self: + """Reject a decision-step cap on a criterion that isn't armed for early-stop. + + ``max_steps_to_decide`` only means anything relative to a criterion + that ``EarlyStopWatcher`` is actually tracking (``stop_when`` set); + setting it without ``stop_when`` is a dead field that silently does + nothing, so reject it at load time rather than let it rot unnoticed. + """ + if self.max_steps_to_decide is not None and self.stop_when is None: + raise ValueError( + f"criterion {self.type!r}: max_steps_to_decide requires stop_when to be set " + + "(the decision-step budget is meaningless for a criterion that isn't armed " + + "for early-stop)." + ) + return self + + @abstractmethod + def live_decidable_polarities(self) -> frozenset[LivePolarity]: + """Which polarities THIS instance can decide mid-run, from its own fields alone. + + Must return a subset of the polarities the corresponding checker's + ``live_verdict`` can ever emit for this criterion type. Used by + ``validate_early_stop`` to reject arming a polarity this instance can + never reach, and by ``EarlyStopWatcher`` to resolve which polarities a + ``stop_when`` value actually arms for this instance (see + ``orchestration.early_stop._requested_polarities``). + """ + + class FileExistsCriterion(BaseSuccessCriterion): """Check if a file exists at the specified path. @@ -472,7 +552,7 @@ class CommandsEfficiencyCriterion(BaseSuccessCriterion): expected_commands: int = Field(ge=1, description="Expected number of tool commands to complete the task") -class CommandExecutedCriterion(BaseSuccessCriterion): +class CommandExecutedCriterion(LiveSuccessCriterion): """Check whether the agent executed specific commands/tools. Inspects CommandTelemetry records from TurnRecord.commands to verify @@ -548,6 +628,32 @@ def _validate_count_bounds(self) -> CommandExecutedCriterion: raise ValueError(f"max_count ({self.max_count}) must be >= min_count ({self.min_count})") return self + def live_decidable_polarities(self) -> frozenset[LivePolarity]: + """Narrow to what THIS instance can decide mid-run. + + The checker's ``live_verdict`` (``criteria/command_executed.py``) can + only: + + - ``pass`` when there is no upper bound and a positive floor + (``max_count is None and min_count > 0``) — with an upper bound a + pass is not final until end-of-run, so it never fires live; and + - ``fail`` when there IS an upper bound (``max_count is not None``), + the moment the count exceeds it (this includes the + ``min_count: 0, max_count: 0`` "must-NOT-run" form). + + So these instance shapes are dead arms the class-level check misses: + ``stop_when: pass`` with ``max_count`` set (pass can never fire); + ``stop_when: fail`` with ``max_count: None`` (fail can never fire); + ``min_count: 0, max_count: None`` (neither can ever fire). + """ + decidable: set[LivePolarity] = set() + if self.max_count is None: + if self.min_count > 0: + decidable.add("pass") + else: + decidable.add("fail") + return frozenset(decidable) + class UiPathEvalCriterion(BaseSuccessCriterion): """Check evaluation results against UiPath agent performance. @@ -608,7 +714,7 @@ class ClassificationMatchCriterion(BaseSuccessCriterion): ) -class SkillTriggeredCriterion(BaseSuccessCriterion): +class SkillTriggeredCriterion(LiveSuccessCriterion): """Binary classifier: did the agent engage the target skill during the run? Agent-agnostic. Observed label is ``"yes"`` when ``turn_records`` show the @@ -644,6 +750,29 @@ class SkillTriggeredCriterion(BaseSuccessCriterion): description="Only count Skill invocations whose 'skill' parameter matches this name.", ) + def live_decidable_polarities(self) -> frozenset[LivePolarity]: + """Per-instance narrowing under the checker's any-engagement latch. + + The checker's ``live_verdict`` (``criteria/skill_triggered.py``) can + decide either polarity at the TYPE level, but a single INSTANCE only + ever resolves one of them: + + - a **positive** criterion (``skill_name == expected_skill``) can only + live-``pass`` (the expected skill engaging is a decidable hit; its + absence is not knowable mid-run); + - a **distractor/negative** criterion (``skill_name != expected_skill``, + including the ``expected_skill == ""`` negatives) can only + live-``fail`` (a wrong skill engaging is a decidable miss; its + absence is not). + + ``validate_early_stop`` gates the requested ``stop_when`` on this set, + so arming a positive with ``fail`` / a distractor with ``pass`` — or + either with ``decided`` (which needs both) — is rejected at resolution + rather than silently degrading to a full run. + """ + expected_yes = self.expected_skill == self.skill_name + return frozenset({"pass"}) if expected_yes else frozenset({"fail"}) + class LLMJudgeCriterion(BaseSuccessCriterion): """Have an LLM grade the task's final state against an author-supplied prompt. diff --git a/src/coder_eval/models/limits.py b/src/coder_eval/models/limits.py index abce752b..7f9184a3 100644 --- a/src/coder_eval/models/limits.py +++ b/src/coder_eval/models/limits.py @@ -92,12 +92,48 @@ class RunLimits(BaseModel): description=( "Opt-in master switch for early-stop-on-criterion. When True, the run ends early " "once the armed criteria (those with stop_when set, incl. per-instance 'auto') " - "are decided mid-run: pass-stop when every pass-armed criterion live-passes, " - "fail-stop on the first fail-armed live-fail (deferred while any pass-armed " - "criterion is undecided, so a misfire never truncates the recall signal) - so a " - "raised max_turns is not wasted once the measured signal has happened. Default " - "False keeps behavior identical. Requires a Claude single-shot task with at least " - "one observable armed criterion; every unsupported combination is rejected at " - "resolution time." + "are decided mid-run: pass-stop when the armed subset's weighted score is " + "GUARANTEED to reach stop_early_gate_threshold regardless of any criterion still " + "undecided, fail-stop when it is GUARANTEED it never can (deferred while any " + "pass-armed criterion is undecided, so a misfire never truncates the recall " + "signal) - so a raised max_turns is not wasted once the outcome is locked in. " + "Default False keeps behavior identical. Requires a Claude single-shot task with " + "at least one observable armed criterion; every unsupported combination is " + "rejected at resolution time." ), ) + stop_early_gate_threshold: float = Field( + default=1.0, + ge=0.0, + le=1.0, + description=( + "Minimum weighted score (Σ weight_i·score_i / Σ weight_i, over the ARMED subset " + "only) required for an early-stopped run to gate as a pass. Also the bound " + "early-stop's trigger checks against: a fail-stop fires once no combination of " + "still-undecided armed criteria could raise the weighted score to this " + "threshold; a pass-stop fires once it is already guaranteed to meet it regardless " + "of what's still undecided. Default 1.0 reproduces the pre-weighting behavior " + "exactly (every armed criterion's live-observable score is binary 0/1, so a " + "weighted score of 1.0 requires every armed criterion to have actually passed) - " + "lowering it lets a low-weight armed criterion's failure be absorbed without " + "truncating the run, at the cost of the gate/trigger becoming a genuine weighted " + "average rather than a strict AND." + ), + ) + + # NOTE: stop_early_gate_threshold <= 0.0 together with stop_early: True is + # a degenerate, gate-neutralizing config (a threshold of 0 trivially + # passes the armed gate regardless of whether anything decided) and is + # rejected — but NOT here. RunLimits is field-merged across 5 layers, so a + # model-level validator has no visibility into which layer produced the + # merged value and cannot distinguish a real mistake from a value merged + # forward from a sibling layer (e.g. a task-level threshold inherited by a + # variant that only flips stop_early). That distinction requires seeing + # the whole resolved task, so the check lives in + # orchestration/early_stop.py::validate_early_stop instead, where it + # raises EarlyStopConfigError and gets the same hard-stop CLI treatment + # (flips the plan exit code, aborts run) as every other early-stop + # guardrail — a plain pydantic ValueError here would instead land in + # plan_command's generic per-variant "resolution failed" branch, which + # prints red text but does NOT flip the exit code by design (unlike + # EarlyStopConfigError), so a model-level raise would silently pass CI. diff --git a/src/coder_eval/models/results.py b/src/coder_eval/models/results.py index 68d7a179..7591ba3c 100644 --- a/src/coder_eval/models/results.py +++ b/src/coder_eval/models/results.py @@ -417,11 +417,17 @@ class EarlyStopReason(StrEnum): ``EvaluationResult`` carries it and ``models/`` is a leaf package that cannot import from ``simulation``. ``DialogStopReason`` is a stylistic reference only. Early-stop is orthogonal telemetry, NOT a ``FinalStatus`` — - the terminal-status set stays closed. + the terminal-status set stays closed — with ONE deliberate exception: + ``DECISION_BUDGET_EXCEEDED`` forces ``FinalStatus.FAILURE`` directly at the + orchestrator finalize step (``orchestrator.py``), bypassing + ``armed_criteria_passed``'s weighted gate entirely — the criterion whose + budget expired never reached a verdict at all, so there is nothing + meaningful to weigh it against. """ CRITERION_PASSED = "criterion_passed" CRITERION_FAILED = "criterion_failed" + DECISION_BUDGET_EXCEEDED = "decision_budget_exceeded" class EarlyStopInfo(BaseModel): @@ -434,7 +440,10 @@ class EarlyStopInfo(BaseModel): the round-trip is safe. """ - reason: EarlyStopReason = Field(description="Why the run stopped: armed criteria passed, or definitively failed.") + reason: EarlyStopReason = Field( + description="Why the run stopped: armed criteria passed, definitively failed, or an armed " + + "criterion's decision-step budget (max_steps_to_decide) expired unresolved." + ) deciding_criterion_type: str = Field( description="Type of the criterion whose live verdict fired the stop (the failing one on " + "fail-stop; the last-to-pass on pass-stop)." @@ -463,6 +472,16 @@ class EarlyStopInfo(BaseModel): description="max_turns - sdk_turn_index (an upper bound on turns avoided, not a measured " + "saving); None when max_turns is unset.", ) + gate_threshold: float = Field( + default=1.0, + ge=0.0, + le=1.0, + description="run_limits.stop_early_gate_threshold in effect for this stop — captured so a " + + "persisted task.json is self-describing (e.g. comparing early-stopped runs across an " + + "experiment sweep that varies the threshold) without needing the resolved task config. " + + "Bounded to mirror the source field so a hand-edited or externally produced record " + + "cannot represent a value the authoritative field would reject.", + ) class EvaluationResult(BaseModel): @@ -654,8 +673,8 @@ def all_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: if c.is_gating ) - def armed_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: - """True iff every ARMED criterion (``stop_when`` set) meets its pass_threshold. + def armed_criteria_passed(self, criteria: list[SuccessCriterion], gate_threshold: float = 1.0) -> bool: + """True iff the ARMED subset's weighted score meets ``gate_threshold``. The early-stop gate: on an early-stopped run only the armed subset gates ``final_status`` (non-armed criteria are advisory — recorded but never @@ -667,6 +686,22 @@ def armed_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: is a defensive guard against misuse. No ``is_gating`` filter is needed here: ``BaseSuccessCriterion`` rejects ``weight: 0`` together with ``stop_when``, so every armed criterion is gating by construction. + + Each armed criterion's OWN ``pass_threshold`` still decides whether it + individually passed — ``r.score`` is converted to a binary 1.0/0.0 via + ``r.score >= c.pass_threshold`` before weighting, exactly mirroring + ``all_criteria_passed``'s per-criterion comparison. Only the + combination rule changes: ``all_criteria_passed`` ANDs those binary + outcomes, this weights and averages them against ``gate_threshold``. + This is what makes the ``gate_threshold=1.0`` default an EXACT + equivalence with the pre-weighting ``all(...)`` rule, not merely an + approximation that happens to hold for binary-scoring criteria: a + weighted average of 1.0 requires every armed criterion's binary + outcome to be 1.0, i.e. every one to have individually passed its own + ``pass_threshold`` — identical to ``all(...)`` regardless of what + ``r.score`` itself was. Callers pass + ``run_limits.stop_early_gate_threshold`` to opt into a genuine + weighted average below 1.0. """ if len(self.success_criteria_results) != len(criteria): raise ValueError( @@ -681,7 +716,19 @@ def armed_criteria_passed(self, criteria: list[SuccessCriterion]) -> bool: f"armed_criteria_passed called with no armed criteria for task {self.task_id}; " + "the early-stop gate is only valid when at least one criterion sets stop_when." ) - return all(r.score >= c.pass_threshold for r, c in armed) + total_weight = sum(c.weight for _, c in armed) + if total_weight <= 0.0: + # Unreachable today (weight=0 + stop_when is rejected at the model + # layer, so every armed criterion carries weight > 0) — but a + # defensive guard on a pass/fail gate must fail CLOSED, not open, + # against a future criterion subclass that bypasses that + # validator. Mirrors EarlyStopWatcher._ceiling's lack of an + # equivalent guard: that one would raise ZeroDivisionError instead + # (fails by crashing, not by silently passing) rather than diverge + # toward a false pass. + return False + weighted_score = sum((1.0 if r.score >= c.pass_threshold else 0.0) * c.weight for r, c in armed) / total_weight + return weighted_score >= gate_threshold class CriterionStats(BaseModel): diff --git a/src/coder_eval/orchestration/early_stop.py b/src/coder_eval/orchestration/early_stop.py index 09d7acc7..379eac4d 100644 --- a/src/coder_eval/orchestration/early_stop.py +++ b/src/coder_eval/orchestration/early_stop.py @@ -27,9 +27,24 @@ Live verdicts only *trigger* the stop; the authoritative scores always come from the standard ``check_all_async`` on the frozen trajectory after the cut. -Precision trade-off: a pass-stop cuts the run the instant every *pass-armed* -criterion is decided, so a *fail-armed* criterion (e.g. a distractor) that would -only misfire on a LATER tool call is never observed — the frozen trajectory then +Weighting: both the stop rule and the post-hoc gate +(``EvaluationResult.armed_criteria_passed``) consult ``run_limits. +stop_early_gate_threshold`` (default ``1.0``) rather than treating every armed +criterion's pass/fail as equally decisive. A fail-stop fires once the armed +subset's CEILING (best case: every still-``undecided``/``pass`` criterion +scores 1.0, every live-``fail``ed one scores 0) can no longer reach the +threshold — i.e. the gate is mathematically guaranteed to fail regardless of +how the trajectory continues. A pass-stop fires once the pass-armed subset's +FLOOR (worst case: every still-undecided one scores 0) already meets it. At the +default threshold of 1.0 both bounds collapse exactly to "any single armed +criterion's live-fail stops the run" / "every pass-armed criterion has +live-passed" — byte-for-byte the pre-weighting behavior, since the only +live-observable criteria score binary 0/1. Lowering the threshold lets a +low-weight armed criterion's failure be absorbed without truncating the run. + +Precision trade-off: a pass-stop cuts the run the instant the pass-armed floor +locks in, so a *fail-armed* criterion (e.g. a distractor) that would only +misfire on a LATER tool call is never observed — the frozen trajectory then scores that row as a clean pass. This is an intentional precision-for-budget trade of the opt-in "smoke" flavor; the authoritative precision/recall must come from a non-early-stop (``stop_early: false``) run. @@ -52,7 +67,7 @@ import time from typing import TYPE_CHECKING, Any, Literal, assert_never -from coder_eval.models import EarlyStopInfo, EarlyStopReason +from coder_eval.models import EarlyStopInfo, EarlyStopReason, LivePolarity, LiveSuccessCriterion from coder_eval.streaming.collector import EventCollector from coder_eval.streaming.events import ( AgentStartEvent, @@ -66,21 +81,21 @@ if TYPE_CHECKING: from coder_eval.criteria.base import BaseCriterion, LiveVerdict - from coder_eval.models import BaseSuccessCriterion, CommandTelemetry, TaskDefinition + from coder_eval.models import CommandTelemetry, TaskDefinition # Armed pair the watcher holds: (criterion model, its checker). Lives in the # TYPE_CHECKING block (only annotations reference it, and those are lazy under # `from __future__ import annotations`), so the names are real references # rather than quoted strings static analyzers cannot resolve. - _ArmedPair = tuple[BaseSuccessCriterion, BaseCriterion[Any]] + _ArmedPair = tuple[LiveSuccessCriterion, BaseCriterion[Any]] logger = logging.getLogger(__name__) def _requested_polarities( - stop_when: Literal["pass", "fail", "decided", "auto"], decidable: frozenset[str] -) -> frozenset[str]: + stop_when: Literal["pass", "fail", "decided", "auto"], decidable: frozenset[LivePolarity] +) -> frozenset[LivePolarity]: """The polarities a ``stop_when`` value requests to arm, given what the instance can decide. The single source of truth for the ``stop_when`` -> polarity mapping — both @@ -170,20 +185,33 @@ def validate_early_stop(task: TaskDefinition) -> None: + "arming requires at least one stop criterion (e.g. stop_when: auto)." ) - # (3)+(4) Per armed criterion: observable, then the requested polarity is - # decidable. The criteria registry is not initialized at resolution time. - from coder_eval.criteria import CriterionRegistry, init_criteria + # (0) A threshold of exactly 0 trivially satisfies both the pass-stop + # floor check and the final weighted gate regardless of whether any armed + # criterion has actually decided — neutralizing the armed pass/fail gate + # with one YAML line (coder-eval is used as a CI gate). Checked here + # (not on RunLimits itself) because this is the whole-task, hard-stop + # surface: an EarlyStopConfigError here flips the plan exit code and + # aborts run, whereas a plain ValueError on the merged RunLimits model + # would land in the CLI's generic "resolution failed" branch, which + # prints red text but does not flip the exit code. + if limits.stop_early_gate_threshold <= 0.0: + raise EarlyStopConfigError( + f"run_limits.stop_early_gate_threshold ({limits.stop_early_gate_threshold}) must be " + + "> 0.0 when stop_early is True (a threshold of 0 trivially passes the armed gate " + + "regardless of whether any armed criterion actually decided)." + ) - init_criteria(validate=False) + # (3)+(4) Per armed criterion: observable, then the requested polarity is + # decidable. for c in armed: # `armed` filtered on `stop_when is not None`; re-bind + assert so pyright # narrows the Literal away from None for the set arithmetic below. polarity = c.stop_when assert polarity is not None - checker_cls = CriterionRegistry.get_checker(c.type) - # (3) Class-level observability: an empty ``live_stop_polarities`` means - # the criterion TYPE can never decide mid-run, regardless of config. - if not checker_cls.live_stop_polarities: + # (3) Type-level observability: a criterion type is live-observable iff + # its model is a ``LiveSuccessCriterion`` subclass (models/criteria.py) + # — the single source of truth, replacing a separate checker-side flag. + if not isinstance(c, LiveSuccessCriterion): raise EarlyStopConfigError( f"criterion type {c.type!r} is armed (stop_when={polarity!r}) but is not " + "observable mid-run; early-stop supports only live-observable criteria " @@ -194,7 +222,7 @@ def validate_early_stop(task: TaskDefinition) -> None: # instance's config, so gate on the per-instance set — otherwise a dead # arm (a polarity this instance can never fire) would silently degrade to # a full run instead of erroring here. - polarities = checker_cls.live_decidable_polarities(c) + polarities = c.live_decidable_polarities() requested = _requested_polarities(polarity, polarities) # Dead arm: only `auto` can request the empty set (it requests exactly # the instance's decidable polarities) — an instance that can decide @@ -217,6 +245,24 @@ def validate_early_stop(task: TaskDefinition) -> None: + "can live-pass only with max_count unset + min_count>0, and live-fail only " + "with max_count set)." ) + # (6) A decision-step budget is meaningless for a fail-only-decidable + # instance (a pure distractor/guard, e.g. a "must-NOT-run" command or a + # negative-row skill_triggered): its "undecided" IS the success + # state — the forbidden event simply hasn't happened yet, and staying + # undecided forever is correct, not a stall. The budget only makes + # sense for an instance that can decide "pass": something it is + # actively waiting to observe. Rejecting this at resolution (rather + # than letting EarlyStopWatcher force-fail a clean run) matters most + # for a dataset-fanned `auto` criterion, where the same YAML line + # would force-fail every negative row. + if c.max_steps_to_decide is not None and "pass" not in polarities: + raise EarlyStopConfigError( + f"criterion {c.type!r} ({c.description!r}) sets max_steps_to_decide but this " + + f"instance can only ever live-decide {sorted(polarities) or 'no polarities'} — " + + "a fail-only-decidable instance's 'undecided' is its success state (the " + + "forbidden event hasn't happened), so a decision-step budget would force-fail " + + "a clean run. max_steps_to_decide requires an instance that can live-pass." + ) class EarlyStopWatcher: @@ -246,16 +292,19 @@ def __init__( armed: list[_ArmedPair], *, max_turns: int | None, + gate_threshold: float = 1.0, ) -> None: self._task_id = task_id self._armed = armed + self._gate_threshold = gate_threshold + self._armed_weight = sum(c.weight for c, _ in armed) # Per-instance resolved arming polarities, aligned with ``_armed``. Static # for the run, resolved through ``_requested_polarities`` (the single # stop_when -> polarity mapping). The stop rule consults this, not the raw # ``stop_when`` string, so a distractor armed ``auto`` (fail only) is not # required to live-pass for a pass-stop. - self._armed_polarities: list[frozenset[str]] = [ - self._resolve_armed_polarities(criterion, checker) for criterion, checker in armed + self._armed_polarities: list[frozenset[LivePolarity]] = [ + self._resolve_armed_polarities(criterion) for criterion, _checker in armed ] self._max_turns = max_turns self._collector = EventCollector() @@ -280,11 +329,18 @@ def for_task(cls, task: TaskDefinition) -> EarlyStopWatcher: from coder_eval.criteria import CriterionRegistry, init_criteria init_criteria(validate=False) + # The `isinstance` check is defense-in-depth, not load-bearing: every + # call site (`run`, `plan`) runs `validate_early_stop` first, which + # already hard-rejects an armed non-observable criterion at resolution + # time. Narrows `c` to `LiveSuccessCriterion` for pyright either way. armed: list[_ArmedPair] = [ - (c, CriterionRegistry.get_checker(c.type)()) for c in task.success_criteria if c.stop_when is not None + (c, CriterionRegistry.get_checker(c.type)()) + for c in task.success_criteria + if c.stop_when is not None and isinstance(c, LiveSuccessCriterion) ] max_turns = task.run_limits.max_turns if task.run_limits is not None else None - return cls(task.task_id, armed, max_turns=max_turns) + gate_threshold = task.run_limits.stop_early_gate_threshold if task.run_limits is not None else 1.0 + return cls(task.task_id, armed, max_turns=max_turns, gate_threshold=gate_threshold) # --- StreamCallback -------------------------------------------------- # @@ -357,7 +413,7 @@ def disarmed(self) -> bool: # --- Stop rule -------------------------------------------------- # @staticmethod - def _resolve_armed_polarities(criterion: BaseSuccessCriterion, checker: BaseCriterion[Any]) -> frozenset[str]: + def _resolve_armed_polarities(criterion: LiveSuccessCriterion) -> frozenset[LivePolarity]: """The polarities this armed instance may fire, via ``_requested_polarities``. Validation has already guaranteed the resolved set is non-empty and @@ -368,9 +424,61 @@ def _resolve_armed_polarities(criterion: BaseSuccessCriterion, checker: BaseCrit sw = criterion.stop_when if sw is None: return frozenset() - return _requested_polarities(sw, checker.live_decidable_polarities(criterion)) + return _requested_polarities(sw, criterion.live_decidable_polarities()) + + def _ceiling(self, verdicts: list[LiveVerdict]) -> float: + """Best-case weighted score over the WHOLE armed set, given current verdicts. + + Every already-live-failed criterion is pinned at 0 (a monotonic + ``live_verdict`` guarantees it stays failed); every ``pass`` or still + ``undecided`` criterion is credited its full weight (the optimistic + assumption that it could still end up scoring 1.0). This is the same + weighting ``EvaluationResult.armed_criteria_passed`` uses for the real, + final gate, so ``ceiling < gate_threshold`` means the gate is + mathematically guaranteed to fail no matter how the trajectory continues. + """ + return sum(c.weight for (c, _checker), v in zip(self._armed, verdicts, strict=True) if v != "fail") / ( + self._armed_weight + ) + + def _floor(self, verdicts: list[LiveVerdict], indices: list[int]) -> float | None: + """Worst-case weighted score over the given armed-index subset, given current verdicts. + + Mirrors ``_ceiling`` for the opposite direction: every still-undecided + (or already-``fail``) criterion in ``indices`` is credited nothing (the + pessimistic assumption that it could still end up scoring 0); only an + already-``pass`` criterion contributes its weight. Returns ``None`` + when the subset's total weight is 0 (the vacuous case — nothing to + bound), so callers don't have to special-case an empty numerator over + an empty denominator. + """ + weight = sum(self._armed[i][0].weight for i in indices) + if weight <= 0.0: + return None + return sum(self._armed[i][0].weight for i in indices if verdicts[i] == "pass") / weight def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: + """Fail-open wrapper: any unexpected exception anywhere in the round + disarms the watcher and degrades to a full run, exactly like a raising + ``live_verdict`` — not just the verdict-collection loop. The + ceiling/floor arithmetic below is currently guarded (an empty armed + set is unreachable, and every candidate short-circuits before + dividing), but a future change to that arithmetic — or to + ``_fire`` — must not be able to leave the watcher stuck re-raising on + every subsequent event with ``_disarmed`` still False (which is what a + narrower try/except would risk). + """ + try: + self._evaluate_impl(in_flight) + except Exception: + self._disarmed = True + logger.error( + "[%s] early-stop round raised unexpectedly; disarming watcher, run degrades to a full run", + self._task_id, + exc_info=True, + ) + + def _evaluate_impl(self, in_flight: CommandTelemetry | None = None) -> None: record = self._collector.build_turn_record() if in_flight is not None: # The in-flight call has no ToolEnd yet, so the collector (which @@ -387,43 +495,71 @@ def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: try: verdicts.append(checker.live_verdict(criterion, records)) except Exception: - # Fail-open: a raising verdict disarms; the run degrades to full. - self._disarmed = True + # Re-raise to the wrapping try/except in _evaluate, which sets + # _disarmed — but log the specific criterion here first, since + # that context (which criterion's live_verdict raised) would + # otherwise be lost once the exception is caught generically. logger.error( - "[%s] early-stop live_verdict raised for criterion %r; disarming watcher, " - + "run degrades to a full run", + "[%s] early-stop live_verdict raised for criterion %r", self._task_id, criterion.type, exc_info=True, ) - return + raise pass_armed = [i for i, pol in enumerate(self._armed_polarities) if "pass" in pol] - # Fail-stop: first armed criterion (criteria order) that live-fails AND - # whose resolved arming permits fail decides the run — but DEFERRED while - # any pass-armed criterion is still undecided. Cutting a positive row on a - # distractor misfire before its expected signal could appear would freeze a - # would-be TP as an FN (truncating the suite's recall); the misfire is - # latched by the criterion's own monotone semantics, so the deferred fail - # still fires the moment every pass-armed criterion decides, and a row with - # zero pass-armed criteria (a negative row) defers nothing. + # Fail-stop: at least one armed criterion (criteria order) that live-fails + # AND whose resolved arming permits fail is a CANDIDATE — but the stop only + # actually fires once the ceiling bound (best case: every still-undecided + # or already-passed armed criterion ends up scoring 1.0, every live-failed + # one scores 0) can no longer reach ``gate_threshold``, i.e. the armed gate + # (``EvaluationResult.armed_criteria_passed``) is GUARANTEED to fail no + # matter what happens on the rest of the trajectory. At the default + # ``gate_threshold=1.0`` this is equivalent to firing on the first + # candidate (any armed criterion's weight is > 0 by construction, so a + # single fail already drops the ceiling below 1.0) — below 1.0 a + # low-weight candidate's failure may not be enough to doom the gate, so the + # run keeps going. This is DEFERRED while any pass-armed criterion is still + # undecided. Cutting a positive row on a distractor misfire before its + # expected signal could appear would freeze a would-be TP as an FN + # (truncating the suite's recall); the misfire is latched by the + # criterion's own monotone semantics, so the deferred fail still fires the + # moment every pass-armed criterion decides, and a row with zero + # pass-armed criteria (a negative row) defers nothing. if not any(verdicts[i] == "undecided" for i in pass_armed): - for (criterion, _checker), verdict, armed_pol in zip( - self._armed, verdicts, self._armed_polarities, strict=True - ): - if verdict == "fail" and "fail" in armed_pol: - self._fire(EarlyStopReason.CRITERION_FAILED, criterion, tool_call_index=tool_call_index) - return - - # Pass-stop: every PASS-ARMED criterion live-passes. Fail-armed criteria - # (e.g. distractors armed ``auto`` -> fail only) are NOT required to pass — - # they can never live-pass and only guard the fail side above, so requiring - # them would veto every pass-stop (the mixed-arming bug). Guard the vacuous - # case: with zero pass-armed criteria (a negative row whose criteria are all - # distractors) there is nothing to pass-stop on, so the run must continue to - # the cap rather than firing on turn 0 with an empty ``all()``. - if pass_armed and all(verdicts[i] == "pass" for i in pass_armed): + candidate = next( + ( + criterion + for (criterion, _checker), verdict, armed_pol in zip( + self._armed, verdicts, self._armed_polarities, strict=True + ) + if verdict == "fail" and "fail" in armed_pol + ), + None, + ) + if candidate is not None and self._ceiling(verdicts) < self._gate_threshold: + self._fire(EarlyStopReason.CRITERION_FAILED, candidate, tool_call_index=tool_call_index) + return + + # Pass-stop: the PASS-ARMED subset's own floor bound (worst case: every + # still-undecided pass-armed criterion ends up scoring 0, weighted against + # only the pass-armed subset's total weight) already meets + # ``gate_threshold`` — guaranteed regardless of what the rest of that + # subset still decides. Fail-armed criteria (e.g. distractors armed + # ``auto`` -> fail only) are excluded from both the numerator and the + # denominator: they can never live-pass and only guard the fail side + # above, so folding them in would veto every pass-stop (the mixed-arming + # bug) and penalize this bound for a criterion it was never scoped to + # cover. At the default ``gate_threshold=1.0`` this requires every + # pass-armed criterion to actually be "pass" (any non-pass drops the floor + # below 1.0) — identical to the pre-weighting ``all(...)`` rule. Guard the + # vacuous case: with zero pass-armed criteria (a negative row whose + # criteria are all distractors) there is nothing to pass-stop on, so the + # run must continue to the cap rather than firing on turn 0 with an empty + # numerator/denominator. + floor = self._floor(verdicts, pass_armed) + if floor is not None and floor >= self._gate_threshold: # Deciding criterion = the last pass-armed (criteria order) whose verdict # flipped vs the previous round; fall back to the last pass-armed. deciding = self._armed[pass_armed[-1]][0] @@ -433,10 +569,25 @@ def _evaluate(self, in_flight: CommandTelemetry | None = None) -> None: self._fire(EarlyStopReason.CRITERION_PASSED, deciding, tool_call_index=tool_call_index) return + # Decision-step budget: an armed criterion with max_steps_to_decide set + # that is STILL "undecided" once that many tool-call steps have elapsed + # forces a hard fail — checked last, after the real fail-/pass-stop + # checks above, so a criterion that decides on this very round (however + # late) is never punished for a budget it technically exceeded. It never + # reached a verdict at all, so there is nothing meaningful to weigh it + # against — this is why DECISION_BUDGET_EXCEEDED bypasses the weighted + # gate entirely at the orchestrator finalize step rather than folding + # into the ceiling/floor bounds above. + for (criterion, _checker), verdict in zip(self._armed, verdicts, strict=True): + budget = criterion.max_steps_to_decide + if budget is not None and verdict == "undecided" and tool_call_index >= budget: + self._fire(EarlyStopReason.DECISION_BUDGET_EXCEEDED, criterion, tool_call_index=tool_call_index) + return + # No stop this round — record the verdicts so the next round can detect flips. self._prev_verdicts = verdicts - def _fire(self, reason: EarlyStopReason, criterion: BaseSuccessCriterion, *, tool_call_index: int) -> None: + def _fire(self, reason: EarlyStopReason, criterion: LiveSuccessCriterion, *, tool_call_index: int) -> None: elapsed = 0.0 if self._started_monotonic is not None: elapsed = max(time.monotonic() - self._started_monotonic, 0.0) @@ -450,6 +601,7 @@ def _fire(self, reason: EarlyStopReason, criterion: BaseSuccessCriterion, *, too tool_call_index=tool_call_index, elapsed_seconds=elapsed, turns_remaining_at_stop=turns_remaining, + gate_threshold=self._gate_threshold, ) logger.info( "[%s] early-stop fired: reason=%s deciding=%s sdk_turn=%d tool_call=%d elapsed=%.2fs", diff --git a/src/coder_eval/orchestrator.py b/src/coder_eval/orchestrator.py index 6d73840d..138ad722 100644 --- a/src/coder_eval/orchestrator.py +++ b/src/coder_eval/orchestrator.py @@ -11,7 +11,7 @@ from datetime import datetime from inspect import isawaitable from pathlib import Path -from typing import Any +from typing import Any, assert_never from urllib.parse import urlparse from .agent import Agent @@ -37,6 +37,7 @@ ConfigLineageEntry, CriterionResult, DirectRoute, + EarlyStopReason, EvaluationResult, FinalStatus, JudgeCriterionResult, @@ -1567,17 +1568,59 @@ async def _evaluation_loop(self) -> bool: pairs = list(zip(criteria_results, self.task.success_criteria, strict=True)) passed_count = sum(1 for r, c in pairs if r.score >= c.pass_threshold) total_count = len(pairs) - if self.result.early_stop is not None: - # Early-stopped run: only the armed subset gates final_status; the rest - # are advisory (recorded, never decisive) so a smoke flavor is not - # dragged to FAILURE by criteria whose work it deliberately skipped. - all_passed = self.result.armed_criteria_passed(self.task.success_criteria) - armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) - logger.info( - "Early-stopped run: gating on %d armed criteria (%d advisory, not gated).", - armed_count, - total_count - armed_count, - ) + # The armed weighted gate governs any task armed for early-stop + # (run_limits.stop_early: true), whether or not the watcher actually + # fired — one task config maps to one gate semantic. Without this, a + # run that happened to finish before the bound ever tripped would + # silently fall back to the strict full-set gate, re-failing on a + # low-weight criterion the weighted gate was configured to forgive, + # for a reason (incidental control flow) unrelated to configured + # intent. Only a task NOT armed for early-stop uses the full, + # strict-AND gate over every gating criterion. + if self.task.run_limits is not None and self.task.run_limits.stop_early: + if self.result.early_stop is not None: + reason = self.result.early_stop.reason + # Exhaustive on EarlyStopReason: adding a 4th member without a + # branch here is a type error (assert_never), not a silent + # fall-through into the weighted-gate branch below. + if reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED: + # Hard fail, bypassing the weighted gate entirely: the deciding + # criterion never reached a verdict within its max_steps_to_decide + # budget, so there is nothing meaningful to weigh it against. + all_passed = False + logger.info( + "Early-stopped run: decision-step budget exceeded for %r, forcing FAILURE.", + self.result.early_stop.deciding_criterion_description, + ) + elif reason == EarlyStopReason.CRITERION_PASSED or reason == EarlyStopReason.CRITERION_FAILED: + all_passed = self.result.armed_criteria_passed( + self.task.success_criteria, self.task.run_limits.stop_early_gate_threshold + ) + armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) + logger.info( + "Early-stopped run: gating on %d armed criteria (%d advisory, not gated).", + armed_count, + total_count - armed_count, + ) + else: + assert_never(reason) + else: + # Armed for early-stop but the watcher never fired (the agent + # finished, or max_turns was hit, before the bound tripped): + # only the armed subset gates final_status; the rest are + # advisory (recorded, never decisive) — same gate as an + # actual early stop, so a smoke flavor is not dragged to + # FAILURE by criteria whose work it deliberately skipped. + all_passed = self.result.armed_criteria_passed( + self.task.success_criteria, self.task.run_limits.stop_early_gate_threshold + ) + armed_count = sum(1 for c in self.task.success_criteria if c.stop_when is not None) + logger.info( + "stop_early armed but never fired (run completed naturally): gating on " + + "%d armed criteria (%d advisory, not gated).", + armed_count, + total_count - armed_count, + ) else: all_passed = self.result.all_criteria_passed(self.task.success_criteria) diff --git a/src/coder_eval/reports.py b/src/coder_eval/reports.py index 743dcf92..b55f6f8c 100644 --- a/src/coder_eval/reports.py +++ b/src/coder_eval/reports.py @@ -438,10 +438,14 @@ def _runtime_notes_lines(summary: RunSummary) -> list[str]: reason = t.get("early_stop_reason") or "unknown" turns_remaining = t.get("turns_remaining_at_stop") avoided = f" <= {turns_remaining} turn(s) avoided —" if isinstance(turns_remaining, int) else "" - notes.append( - f"> **NOTE:** [{task_id}] stopped early ({reason});{avoided}" - + " gated on armed criteria only; other criteria are advisory" - ) + if reason == "decision_budget_exceeded": + # No criterion gated here at all — an armed criterion's + # decision-step budget expired unresolved, forcing FAILURE + # outright, bypassing the weighted gate entirely. + gate_note = " forced to FAILURE (decision-step budget exceeded, bypassing the gate)" + else: + gate_note = " gated on armed criteria only; other criteria are advisory" + notes.append(f"> **NOTE:** [{task_id}] stopped early ({reason});{avoided}" + gate_note) if not notes: return [] return ["## Run-time Notes", "", *notes] diff --git a/src/coder_eval/reports_experiment.py b/src/coder_eval/reports_experiment.py index e362d271..8f2ec675 100644 --- a/src/coder_eval/reports_experiment.py +++ b/src/coder_eval/reports_experiment.py @@ -210,6 +210,10 @@ def eval_result_to_task_dict( "turns_remaining_at_stop": ( result.early_stop.turns_remaining_at_stop if result.early_stop is not None else None ), + # The threshold in effect for this stop, so a downstream consumer + # comparing early-stopped runs across an experiment sweep that varies + # it can tell which weighted-gate value produced a given verdict. + "gate_threshold": (result.early_stop.gate_threshold if result.early_stop is not None else None), } d["variant_id"] = variant_id return d diff --git a/src/coder_eval/reports_html.py b/src/coder_eval/reports_html.py index 3fef5d12..e5a2c048 100644 --- a/src/coder_eval/reports_html.py +++ b/src/coder_eval/reports_html.py @@ -17,7 +17,7 @@ from pathlib import Path from typing import TYPE_CHECKING, Any -from coder_eval.models import FinalStatus, eval_result_total_cost, sum_costs +from coder_eval.models import EarlyStopReason, FinalStatus, eval_result_total_cost, sum_costs if TYPE_CHECKING: @@ -345,9 +345,15 @@ def _render_header(result: EvaluationResult) -> str: expected_turns_badge = f'expected_turns exceeded ({actual}/{expected})' early_stop_badge = "" if result.early_stop is not None: + if result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED: + # No criterion gated here at all — forced to FAILURE outright, + # bypassing the weighted gate. + title = "Forced to FAILURE (decision-step budget exceeded, bypassing the gate)" + else: + title = "Gated on armed criteria only; other criteria are advisory" early_stop_badge = ( - 'stopped early ({_esc(result.early_stop.reason.value)})' + f'' + + f"stopped early ({_esc(result.early_stop.reason.value)})" ) return f"""
diff --git a/tasks/early_stop_decision_budget_exceeded.yaml b/tasks/early_stop_decision_budget_exceeded.yaml new file mode 100644 index 00000000..0d287f5a --- /dev/null +++ b/tasks/early_stop_decision_budget_exceeded.yaml @@ -0,0 +1,40 @@ +task_id: "early_stop_decision_budget_exceeded" +description: > + Decision-step budget (GitHub issue #61, item 3): the armed command_executed + criterion caps at max_steps_to_decide: 3 — if the agent hasn't run the + script within its first 3 tool calls, EarlyStopWatcher fires + reason=decision_budget_exceeded and the run is force-failed outright, + bypassing the weighted stop_early_gate_threshold gate entirely (a criterion + that never reached a verdict has nothing meaningful to weigh against the + advisory file_exists criterion below). This is a NON-CI example task + (deliberately untagged for smoke-pass/smoke-fail): the agent's actual + exploration behavior is not deterministic enough for a live-agent CI + assertion — run it manually with `coder-eval run` to observe the mechanism. +tags: [early-stop, decision-budget] + +initial_prompt: > + Before doing anything else, spend at least 5 tool calls exploring the + repository (ls, find, read a few files) before creating app.py. Then create + a Python file named app.py that prints 'Hello, Claude!' and run it with: + python app.py + +agent: + type: "claude-code" + permission_mode: "acceptEdits" + allowed_tools: ["Read", "Write", "Bash", "Glob"] + +run_limits: + max_turns: 20 + stop_early: true + +success_criteria: + - type: "command_executed" + description: "Agent ran the script within its decision budget" + tool_name: "Bash" + command_pattern: "python app\\.py" + min_count: 1 + stop_when: "auto" + max_steps_to_decide: 3 + - type: "file_exists" + path: "app.py" + description: "app.py must exist (advisory — not armed, only gates a full run)" diff --git a/tasks/early_stop_weighted_high_weight_kills_run.yaml b/tasks/early_stop_weighted_high_weight_kills_run.yaml new file mode 100644 index 00000000..141e7892 --- /dev/null +++ b/tasks/early_stop_weighted_high_weight_kills_run.yaml @@ -0,0 +1,55 @@ +task_id: "early_stop_weighted_high_weight_kills_run" +description: > + Weighted early-stop (GitHub issue #61, item 1) — the mirror case of + early_stop_weighted_low_weight_absorbed.yaml. Same two armed criteria, but + the weights are swapped: a LOW-weight (0.2) positive ("ran python app.py") + and a HIGH-weight (0.8) distractor ("never called curl"). If the distractor + misfires, the ceiling (best case: the low-weight positive still passes) is + only 0.2, which can never reach the 0.7 gate threshold — the run is + mathematically guaranteed to fail the armed gate no matter what happens + next. The fail-stop is DEFERRED, though, while the low-weight positive is + still undecided (a distractor misfire must not truncate recall) — with the + prompt ordering curl before python app.py, the fail-stop actually fires as + soon as the positive resolves (not "immediately" on the curl misfire + itself), rather than burning the rest of run_limits.max_turns on a doomed + run. This is a NON-CI example task + (deliberately untagged for smoke-pass/smoke-fail): whether the agent + actually calls curl as instructed is not deterministic enough for a + live-agent CI assertion — run it manually with `coder-eval run` to observe + the mechanism. +tags: [early-stop, weighted-early-stop] + +initial_prompt: > + Create a Python file named app.py in the current working directory that + prints 'Hello, Claude!'. Then run it with: python app.py + As a first step, sanity-check network access by running: curl -sI https://example.com + +agent: + type: "claude-code" + permission_mode: "acceptEdits" + allowed_tools: ["Read", "Write", "Bash"] + +run_limits: + max_turns: 20 + stop_early: true + stop_early_gate_threshold: 0.7 + +success_criteria: + - type: "command_executed" + description: "Agent ran the script (the LOW-weight positive signal)" + tool_name: "Bash" + command_pattern: "python app\\.py" + min_count: 1 + weight: 0.2 + stop_when: "auto" + - type: "command_executed" + description: "Agent did NOT call curl (the HIGH-weight distractor)" + tool_name: "Bash" + command_pattern: "curl" + min_count: 0 + max_count: 0 + weight: 0.8 + stop_when: "auto" + - type: "file_exists" + path: "app.py" + description: "app.py must exist (advisory — not armed, only gates a full run)" diff --git a/tasks/early_stop_weighted_low_weight_absorbed.yaml b/tasks/early_stop_weighted_low_weight_absorbed.yaml new file mode 100644 index 00000000..a4e13a47 --- /dev/null +++ b/tasks/early_stop_weighted_low_weight_absorbed.yaml @@ -0,0 +1,54 @@ +task_id: "early_stop_weighted_low_weight_absorbed" +description: > + Weighted early-stop (GitHub issue #61, item 1): a LOW-weight armed criterion + misfiring must not unilaterally truncate the run once run_limits. + stop_early_gate_threshold is below 1.0. Two armed criteria: a HIGH-weight + (0.8) positive ("ran python app.py") and a LOW-weight (0.2) distractor + ("never called curl"). If the distractor misfires alone, the ceiling + (best case: the positive still passes) is 0.8, which still clears the 0.7 + gate threshold — so the watcher must keep running instead of fail-stopping, + and the low-weight failure must not sink the final armed_criteria_passed + gate either (weighted score 0.8 >= 0.7). Note: the weighted forgiveness + applies only when the watcher itself fires the stop (pass-stops here once + the positive resolves, regardless of the distractor) — a stop_early: true + run that instead completes naturally still gates on the full strict-AND + set, unaffected by weight. This is a NON-CI example task (deliberately + untagged for smoke-pass/smoke-fail): agent behavior around curl is not + deterministic enough for a live-agent CI assertion — run it manually with + `coder-eval run` to observe the mechanism. +tags: [early-stop, weighted-early-stop] + +initial_prompt: > + Create a Python file named app.py in the current working directory that + prints 'Hello, Claude!'. Then run it with: python app.py + Do not use curl for anything in this task. + +agent: + type: "claude-code" + permission_mode: "acceptEdits" + allowed_tools: ["Read", "Write", "Bash"] + +run_limits: + max_turns: 20 + stop_early: true + stop_early_gate_threshold: 0.7 + +success_criteria: + - type: "command_executed" + description: "Agent ran the script (the HIGH-weight positive signal)" + tool_name: "Bash" + command_pattern: "python app\\.py" + min_count: 1 + weight: 0.8 + stop_when: "auto" + - type: "command_executed" + description: "Agent did NOT call curl (the LOW-weight distractor)" + tool_name: "Bash" + command_pattern: "curl" + min_count: 0 + max_count: 0 + weight: 0.2 + stop_when: "auto" + - type: "file_exists" + path: "app.py" + description: "app.py must exist (advisory — not armed, only gates a full run)" diff --git a/tests/lint/rules/ce025_live_verdict_consistency.py b/tests/lint/rules/ce025_live_verdict_consistency.py deleted file mode 100644 index 162c1c8d..00000000 --- a/tests/lint/rules/ce025_live_verdict_consistency.py +++ /dev/null @@ -1,111 +0,0 @@ -"""CE025: a criterion's ``live_stop_polarities`` and ``live_verdict`` must agree. - -The early-stop live-verdict contract (``criteria/base.py``) pairs two members on -every ``BaseCriterion`` subclass: - -- ``live_stop_polarities: ClassVar[frozenset[str]]`` — the polarities the - criterion can decide from a PARTIAL, mid-run trajectory. Empty (the base - default) means "not observable mid-run", so the criterion can never arm - early-stop. -- ``live_verdict(...)`` — the method that actually reads the partial trajectory - and returns ``"pass"``/``"fail"``/``"undecided"``. - -The two must move together: a non-empty ``live_stop_polarities`` without a -``live_verdict`` override arms a criterion whose base ``live_verdict`` always -returns ``"undecided"`` (it can never stop — a silent dead arm); a -``live_verdict`` override without a non-empty ``live_stop_polarities`` writes -decision logic the arming path can never reach (dead code that reads as -supported). Either drift is a mechanically detectable bug, so flag it. - -Scoped to ``src/coder_eval/criteria/`` and exempts ``criteria/base.py`` (which -legitimately declares the empty default alongside the default ``live_verdict``). - -Non-empty detection: ``frozenset({...})`` / ``frozenset([...])`` with a -non-empty literal is non-empty; bare ``frozenset()`` is empty; a ``frozenset`` -call over a non-literal argument (or any other RHS) is conservatively treated as -non-empty. - -Add ``# noqa: CE025`` on the class line for a deliberate exception. -""" - -import ast -import re - -from tests.lint.rules.base import BaseRule - - -_CRITERIA_DIR = re.compile(r"[/\\]coder_eval[/\\]criteria[/\\]") -_BASE_FILE = re.compile(r"[/\\]coder_eval[/\\]criteria[/\\]base\.py$") - - -def _is_frozenset(func: ast.expr) -> bool: - return isinstance(func, ast.Name) and func.id == "frozenset" - - -def _polarities_nonempty(value: ast.expr) -> bool: - """Whether the RHS of a ``live_stop_polarities`` assignment is non-empty. - - ``frozenset()`` reads the literal's element count; bare - ``frozenset()`` is empty; anything else is conservatively non-empty so a - computed value is never mistaken for a dead arm. - """ - if isinstance(value, ast.Call) and _is_frozenset(value.func): - if not value.args: - return False - arg = value.args[0] - if isinstance(arg, ast.Set | ast.List | ast.Tuple): - return len(arg.elts) > 0 - if isinstance(arg, ast.Dict): - return len(arg.keys) > 0 - # Non-literal argument (a name, comprehension, …) — conservatively non-empty. - return True - # Not a frozenset() call at all — conservatively treat the declaration as non-empty. - return True - - -class LiveVerdictConsistency(BaseRule): - id = "CE025" - - def __init__(self, filepath: str) -> None: - super().__init__(filepath) - self._active = bool(_CRITERIA_DIR.search(filepath)) and not bool(_BASE_FILE.search(filepath)) - - def check(self, tree: ast.AST) -> list: # type: ignore[override] - if not self._active or not isinstance(tree, ast.Module): - return [] - for node in ast.walk(tree): - if isinstance(node, ast.ClassDef): - self._check_class(node) - return self.violations - - def _check_class(self, node: ast.ClassDef) -> None: - has_polarities = False - polarities_nonempty = False - has_live_verdict = False - for stmt in node.body: - if isinstance(stmt, ast.FunctionDef) and stmt.name == "live_verdict": - has_live_verdict = True - elif isinstance(stmt, ast.AnnAssign) and isinstance(stmt.target, ast.Name): - if stmt.target.id == "live_stop_polarities" and stmt.value is not None: - has_polarities = True - polarities_nonempty = _polarities_nonempty(stmt.value) - elif isinstance(stmt, ast.Assign): - for target in stmt.targets: - if isinstance(target, ast.Name) and target.id == "live_stop_polarities": - has_polarities = True - polarities_nonempty = _polarities_nonempty(stmt.value) - - if has_polarities and polarities_nonempty and not has_live_verdict: - self.violation( - node, - "declares a non-empty `live_stop_polarities` but no `live_verdict` override; " - "the base `live_verdict` always returns 'undecided', so this arms a criterion " - "that can never stop (add a `live_verdict` override or clear the polarities).", - ) - elif has_live_verdict and not (has_polarities and polarities_nonempty): - self.violation( - node, - "overrides `live_verdict` but declares no non-empty `live_stop_polarities`; " - "the arming path can never reach this decision logic (declare the polarities " - "it supports or drop the override).", - ) diff --git a/tests/lint/runner.py b/tests/lint/runner.py index 966c08ad..e360b8ec 100644 --- a/tests/lint/runner.py +++ b/tests/lint/runner.py @@ -20,7 +20,6 @@ from tests.lint.rules.ce022_dialog_loop_statement_cap import SimulationDialogLoopStatementCap from tests.lint.rules.ce023_no_proxy_shim_import import NoProxyShimImports from tests.lint.rules.ce024_discriminated_unions import DiscriminatedUnions -from tests.lint.rules.ce025_live_verdict_consistency import LiveVerdictConsistency from tests.lint.rules.no_agent_timing_access import NoAgentTimingAccess from tests.lint.rules.no_blocking_io_in_async import NoBlockingIoInAsync from tests.lint.rules.no_cli_imports_in_core import NoCliImportsInCore @@ -64,7 +63,6 @@ SimulationDialogLoopStatementCap, NoProxyShimImports, DiscriminatedUnions, - LiveVerdictConsistency, ] # Anti-shadow invariant (mirrors AgentRegistry / register_pricing): every CE rule diff --git a/tests/test_custom_lint.py b/tests/test_custom_lint.py index c4b0623b..a1f4da70 100644 --- a/tests/test_custom_lint.py +++ b/tests/test_custom_lint.py @@ -532,67 +532,110 @@ def test_flags_annassign_union(self): @pytest.mark.lint class TestCE025LiveVerdictConsistency: - """CE025 flags criteria whose `live_stop_polarities` and `live_verdict` disagree.""" - - _POLARITIES_NONEMPTY = ' live_stop_polarities: ClassVar[frozenset[str]] = frozenset({"pass", "fail"})\n' - _POLARITIES_EMPTY = " live_stop_polarities: ClassVar[frozenset[str]] = frozenset()\n" - _POLARITIES_PLAIN = ' live_stop_polarities = frozenset({"pass"})\n' - _POLARITIES_NONLITERAL = " live_stop_polarities: ClassVar[frozenset[str]] = frozenset(_SOME_SET)\n" - _LIVE_VERDICT = ' def live_verdict(self, criterion, turn_records):\n return "undecided"\n' - - @staticmethod - def _cls(*members: str) -> str: - body = "".join(members) if members else " pass\n" - return "from typing import ClassVar\nclass FakeChecker:\n" + body + """CE025: a criterion type's ``LiveSuccessCriterion`` subclassing (models/criteria.py) + and its checker's ``live_verdict`` override (criteria/) must agree. + + Whole-tree / registry-based (like CE027-31), not a per-file AST rule: the + invariant spans two separate class hierarchies (the criterion model in + ``models/criteria.py`` and its checker in ``criteria/``) linked only via the + shared ``type`` discriminator string through ``CriterionRegistry``, so it + cannot be checked by looking at one file's AST in isolation. + + A criterion model that is a ``LiveSuccessCriterion`` subclass without a + ``live_verdict`` override on its checker arms a criterion whose base + ``live_verdict`` always returns ``"undecided"`` (a silent dead arm); a + checker overriding ``live_verdict`` whose criterion model is NOT a + ``LiveSuccessCriterion`` writes decision logic the arming path + (``validate_early_stop`` gates on ``isinstance(c, LiveSuccessCriterion)``) + can never reach. + """ @staticmethod - def _run(src: str, *, path: str = "src/coder_eval/criteria/fake_checker.py"): - import ast - - from tests.lint.rules.ce025_live_verdict_consistency import LiveVerdictConsistency - - return LiveVerdictConsistency(path).check(ast.parse(src)) - - def test_flags_polarities_without_live_verdict(self): - assert self._run(self._cls(self._POLARITIES_NONEMPTY)) + def _type_to_model() -> dict[str, type]: + from typing import Annotated, get_args, get_origin - def test_flags_live_verdict_without_polarities(self): - assert self._run(self._cls(self._LIVE_VERDICT)) + from coder_eval.models import SuccessCriterion - def test_flags_live_verdict_with_empty_polarities(self): - assert self._run(self._cls(self._POLARITIES_EMPTY, self._LIVE_VERDICT)) + assert get_origin(SuccessCriterion) is Annotated + inner, *_ = get_args(SuccessCriterion) + return {model.model_fields["type"].default: model for model in get_args(inner)} - def test_allows_polarities_with_live_verdict(self): - assert not self._run(self._cls(self._POLARITIES_NONEMPTY, self._LIVE_VERDICT)) - - def test_allows_neither_declared(self): - # A plain checker (e.g. file_exists) that arms nothing declares neither member. - assert not self._run(self._cls(" path: str\n")) - - def test_allows_empty_polarities_without_live_verdict(self): - assert not self._run(self._cls(self._POLARITIES_EMPTY)) - - def test_detects_plain_assign_polarities(self): - # Non-empty `live_stop_polarities` via a plain (non-annotated) assignment still arms. - assert self._run(self._cls(self._POLARITIES_PLAIN)) - - def test_non_literal_arg_treated_nonempty(self): - # A computed frozenset() argument is conservatively non-empty → live_verdict required. - assert self._run(self._cls(self._POLARITIES_NONLITERAL)) - assert not self._run(self._cls(self._POLARITIES_NONLITERAL, self._LIVE_VERDICT)) - - def test_scope_exemptions(self): - # base.py legitimately pairs the empty default with the default live_verdict; and - # files outside criteria/ are out of scope entirely. - offending = self._cls(self._LIVE_VERDICT) - assert not self._run(offending, path="src/coder_eval/criteria/base.py") - assert not self._run(offending, path="src/coder_eval/orchestration/x.py") - - def test_real_criteria_tree_is_clean(self): - from tests.lint.rules.ce025_live_verdict_consistency import LiveVerdictConsistency - - criteria_dir = SRC / "coder_eval" / "criteria" - assert not check_paths([criteria_dir], rules=[LiveVerdictConsistency]) + @staticmethod + def _find_violations(pairs: dict[str, tuple[type, type]]) -> list[str]: + """Shared checker-vs-model pairing logic, driven by an explicit + ``{criterion_type: (checker_cls, model_cls)}`` mapping so both the real + registry and synthetic fixtures can exercise it identically.""" + from coder_eval.criteria.base import BaseCriterion + from coder_eval.models import LiveSuccessCriterion + + violations = [] + for criterion_type, (checker_cls, model) in pairs.items(): + overrides_live_verdict = checker_cls.live_verdict is not BaseCriterion.live_verdict + is_live_model = issubclass(model, LiveSuccessCriterion) + if is_live_model and not overrides_live_verdict: + violations.append( + f"{model.__name__} ({criterion_type!r}) is a LiveSuccessCriterion but its checker " + f"{checker_cls.__name__} does not override live_verdict — a dead arm." + ) + elif overrides_live_verdict and not is_live_model: + violations.append( + f"{checker_cls.__name__} ({criterion_type!r}) overrides live_verdict but its criterion " + f"model {model.__name__} is not a LiveSuccessCriterion — unreachable decision logic." + ) + return violations + + def test_real_criteria_tree_is_clean(self) -> None: + from coder_eval.criteria import CriterionRegistry, init_criteria + + init_criteria(validate=False) + pairs = { + criterion_type: (CriterionRegistry.get_checker(criterion_type), model) + for criterion_type, model in self._type_to_model().items() + } + assert not self._find_violations(pairs) + + def test_detects_live_model_with_no_live_verdict_override(self) -> None: + # A LiveSuccessCriterion subclass whose checker does NOT override + # live_verdict — the dead-arm branch. + from coder_eval.criteria.base import BaseCriterion + from coder_eval.models import LiveSuccessCriterion + + class _FakeLiveModel(LiveSuccessCriterion): + def live_decidable_polarities(self): + return frozenset({"pass"}) + + class _FakeCheckerNoOverride(BaseCriterion): + criterion_type = "fake_live_no_override" + + def _check_impl(self, criterion, sandbox, reference_code=None, *, turn_records=None, context=None): + raise NotImplementedError + + violations = self._find_violations({"fake_live_no_override": (_FakeCheckerNoOverride, _FakeLiveModel)}) + assert violations + assert "dead arm" in violations[0] + + def test_detects_live_verdict_override_on_non_live_model(self) -> None: + # A checker overriding live_verdict whose criterion model is a plain + # BaseSuccessCriterion, not LiveSuccessCriterion — the unreachable + # decision-logic branch. + from coder_eval.criteria.base import BaseCriterion, LiveVerdict + from coder_eval.models import BaseSuccessCriterion + + class _FakeNonLiveModel(BaseSuccessCriterion): + type: str = "fake_non_live_override" + + class _FakeCheckerOverrides(BaseCriterion): + criterion_type = "fake_non_live_override" + + def _check_impl(self, criterion, sandbox, reference_code=None, *, turn_records=None, context=None): + raise NotImplementedError + + def live_verdict(self, criterion, turn_records) -> LiveVerdict: + return "undecided" + + violations = self._find_violations({"fake_non_live_override": (_FakeCheckerOverrides, _FakeNonLiveModel)}) + assert violations + assert "unreachable" in violations[0] @pytest.mark.lint diff --git a/tests/test_early_stop.py b/tests/test_early_stop.py index 36daa3e0..9537853a 100644 --- a/tests/test_early_stop.py +++ b/tests/test_early_stop.py @@ -1,7 +1,7 @@ """Tests for early-stop-on-criterion (phases 1-3). Phase 1 (config + contract): the two new opt-in config fields, the -``live_verdict`` / ``live_stop_polarities`` observability contract on the two +``live_verdict`` / ``LiveSuccessCriterion`` observability contract on the two observable criteria, and ``validate_early_stop``'s guardrails on both the ``plan`` and ``run`` surfaces. @@ -40,7 +40,6 @@ from coder_eval.agents.registry import AgentRegistry from coder_eval.cli.plan_command import plan_command from coder_eval.criteria import CriterionRegistry, init_criteria -from coder_eval.criteria.base import BaseCriterion from coder_eval.criteria.command_executed import CommandExecutedChecker from coder_eval.criteria.skill_triggered import SkillTriggeredChecker, _engaged_skill_names from coder_eval.errors import AgentCrashError, TurnTimeoutError @@ -57,6 +56,7 @@ ExperimentVariant, FileExistsCriterion, FinalStatus, + LiveSuccessCriterion, RunLimits, RunSummary, SandboxConfig, @@ -112,6 +112,7 @@ def _task( stop_early: bool = False, agent_type: AgentKind | str = AgentKind.CLAUDE_CODE, simulation: SimulationConfig | None = None, + gate_threshold: float = 1.0, ) -> TaskDefinition: """Build a minimal resolved-style TaskDefinition for guardrail tests.""" return TaskDefinition( @@ -121,7 +122,7 @@ def _task( agent=parse_agent_config(type=agent_type), sandbox=SandboxConfig(driver="tempdir"), success_criteria=criteria, - run_limits=RunLimits(stop_early=stop_early, max_turns=20), + run_limits=RunLimits(stop_early=stop_early, max_turns=20, stop_early_gate_threshold=gate_threshold), simulation=simulation, ) @@ -152,18 +153,35 @@ def dummy_no_stop_kind() -> Iterator[str]: AgentRegistry._registry.pop(kind, None) -def _skill_crit(skill_name: str, expected_skill: str, *, stop_when: str | None = None) -> SkillTriggeredCriterion: +def _skill_crit( + skill_name: str, + expected_skill: str, + *, + stop_when: str | None = None, + weight: float = 1.0, + max_steps_to_decide: int | None = None, + pass_threshold: float = 0.9, +) -> SkillTriggeredCriterion: return SkillTriggeredCriterion( type="skill_triggered", description=f"{skill_name} activation", skill_name=skill_name, expected_skill=expected_skill, stop_when=stop_when, # type: ignore[arg-type] + weight=weight, + max_steps_to_decide=max_steps_to_decide, + pass_threshold=pass_threshold, ) def _cmd_crit( - *, min_count: int = 1, max_count: int | None = None, pattern: str | None = "curl", stop_when: str | None = None + *, + min_count: int = 1, + max_count: int | None = None, + pattern: str | None = "curl", + stop_when: str | None = None, + weight: float = 1.0, + max_steps_to_decide: int | None = None, ) -> CommandExecutedCriterion: return CommandExecutedCriterion( type="command_executed", @@ -173,6 +191,8 @@ def _cmd_crit( min_count=min_count, max_count=max_count, stop_when=stop_when, # type: ignore[arg-type] + weight=weight, + max_steps_to_decide=max_steps_to_decide, ) @@ -262,6 +282,34 @@ def test_stop_early_defaults_false(self) -> None: def test_stop_early_settable(self) -> None: assert RunLimits(stop_early=True).stop_early is True + def test_gate_threshold_out_of_bounds_rejected(self) -> None: + with pytest.raises(ValueError, match="less than or equal to 1"): + RunLimits(stop_early=True, stop_early_gate_threshold=1.5) + with pytest.raises(ValueError, match="greater than or equal to 0"): + RunLimits(stop_early=True, stop_early_gate_threshold=-0.1) + + def test_gate_threshold_nondefault_without_stop_early_allowed(self) -> None: + # A non-default threshold with stop_early=False is inert, not + # rejected: RunLimits is field-merged across 5 layers, so a variant + # that flips only stop_early: false must be able to legitimately + # inherit a threshold value set on a sibling layer (e.g. the + # early-stop-ab e2e variant) without that being a resolution error. + limits = RunLimits(stop_early=False, stop_early_gate_threshold=0.7) + assert limits.stop_early_gate_threshold == 0.7 + + def test_gate_threshold_zero_constructs_at_the_model_level(self) -> None: + # A threshold of exactly 0 is NOT rejected by RunLimits itself — that + # degeneracy check needs the whole task (validate_early_stop), since + # a model-level validator can't distinguish it from a value merged + # forward from a sibling layer. See TestValidateEarlyStop for the + # actual hard-stop rejection. + limits = RunLimits(stop_early=True, stop_early_gate_threshold=0.0) + assert limits.stop_early_gate_threshold == 0.0 + + def test_gate_threshold_default_is_valid_either_way(self) -> None: + assert RunLimits(stop_early=False).stop_early_gate_threshold == 1.0 + assert RunLimits(stop_early=True).stop_early_gate_threshold == 1.0 + def test_stop_when_defaults_none(self) -> None: assert _skill_crit("s", "s").stop_when is None @@ -281,6 +329,22 @@ def test_stop_when_auto_roundtrips(self) -> None: assert restored.stop_when == "auto" assert "stop_when" in restored.model_fields_set + def test_max_steps_to_decide_defaults_none(self) -> None: + assert _skill_crit("s", "s", stop_when="pass").max_steps_to_decide is None + + def test_max_steps_to_decide_requires_stop_when(self) -> None: + with pytest.raises(ValueError, match="max_steps_to_decide requires stop_when"): + _skill_crit("s", "s", max_steps_to_decide=5) + + def test_max_steps_to_decide_allowed_with_stop_when(self) -> None: + crit = _skill_crit("s", "s", stop_when="pass", max_steps_to_decide=5) + assert crit.max_steps_to_decide == 5 + + def test_max_steps_to_decide_requires_stop_when_command_executed(self) -> None: + # The other LiveSuccessCriterion subclass — same validator, same error. + with pytest.raises(ValueError, match="max_steps_to_decide requires stop_when"): + _cmd_crit(max_steps_to_decide=5) + # --------------------------------------------------------------------------- # # skill_triggered live verdict @@ -360,18 +424,14 @@ def test_expected_skill_engaged_after_wrong_is_pass(self) -> None: rec = [_turn(_cmd("Skill", {"skill": "wrong"}), _cmd("Skill", {"skill": "date-teller"}))] assert self.checker.live_verdict(crit, rec) == "pass" - def test_polarities_declared(self) -> None: - assert SkillTriggeredChecker.live_stop_polarities == frozenset({"pass", "fail"}) + def test_is_live_success_criterion(self) -> None: + assert isinstance(_skill_crit("date-teller", "date-teller"), LiveSuccessCriterion) def test_decidable_narrows_per_instance(self) -> None: # A positive instance decides only pass; a distractor/negative only fail. - assert SkillTriggeredChecker.live_decidable_polarities(_skill_crit("date-teller", "date-teller")) == frozenset( - {"pass"} - ) - assert SkillTriggeredChecker.live_decidable_polarities( - _skill_crit("weather-teller", "date-teller") - ) == frozenset({"fail"}) - assert SkillTriggeredChecker.live_decidable_polarities(_skill_crit("date-teller", "")) == frozenset({"fail"}) + assert _skill_crit("date-teller", "date-teller").live_decidable_polarities() == frozenset({"pass"}) + assert _skill_crit("weather-teller", "date-teller").live_decidable_polarities() == frozenset({"fail"}) + assert _skill_crit("date-teller", "").live_decidable_polarities() == frozenset({"fail"}) # --------------------------------------------------------------------------- # @@ -430,31 +490,31 @@ def test_matching_shared_with_check_impl(self) -> None: result = self.checker.check(crit, sandbox=None, turn_records=rec) # type: ignore[arg-type] assert result.score == 1.0 - def test_polarities_declared(self) -> None: - assert CommandExecutedChecker.live_stop_polarities == frozenset({"pass", "fail"}) + def test_is_live_success_criterion(self) -> None: + assert isinstance(_cmd_crit(min_count=1), LiveSuccessCriterion) def test_decidable_pass_only_when_no_upper_bound(self) -> None: crit = _cmd_crit(min_count=1, max_count=None) - assert CommandExecutedChecker.live_decidable_polarities(crit) == frozenset({"pass"}) + assert crit.live_decidable_polarities() == frozenset({"pass"}) def test_decidable_fail_only_when_upper_bound_set(self) -> None: crit = _cmd_crit(min_count=1, max_count=3) - assert CommandExecutedChecker.live_decidable_polarities(crit) == frozenset({"fail"}) + assert crit.live_decidable_polarities() == frozenset({"fail"}) def test_decidable_fail_for_must_not_run(self) -> None: crit = _cmd_crit(min_count=0, max_count=0) - assert CommandExecutedChecker.live_decidable_polarities(crit) == frozenset({"fail"}) + assert crit.live_decidable_polarities() == frozenset({"fail"}) def test_decidable_empty_for_zero_min_no_max(self) -> None: # min_count=0 + no upper bound: neither pass nor fail can ever fire. crit = _cmd_crit(min_count=0, max_count=None) - assert CommandExecutedChecker.live_decidable_polarities(crit) == frozenset() + assert crit.live_decidable_polarities() == frozenset() - def test_decidable_is_subset_of_class_polarities(self) -> None: - # The instance set can never exceed the class capability. + def test_decidable_is_subset_of_type_universe(self) -> None: + # The instance set can never exceed {"pass", "fail"} — the type's universe. for min_c, max_c in [(1, None), (1, 3), (0, 0), (0, None)]: crit = _cmd_crit(min_count=min_c, max_count=max_c) - assert CommandExecutedChecker.live_decidable_polarities(crit) <= CommandExecutedChecker.live_stop_polarities + assert crit.live_decidable_polarities() <= frozenset({"pass", "fail"}) # --------------------------------------------------------------------------- # @@ -463,29 +523,24 @@ def test_decidable_is_subset_of_class_polarities(self) -> None: class TestBaseLiveVerdictDefault: - def test_base_polarities_empty(self) -> None: - assert BaseCriterion.live_stop_polarities == frozenset() + def test_unobservable_criterion_is_not_a_live_success_criterion(self) -> None: + # file_exists is not observable mid-run: its model is plain + # BaseSuccessCriterion, not LiveSuccessCriterion — no + # live_decidable_polarities method to call at all. + crit = FileExistsCriterion(type="file_exists", path="x.txt", description="x") + assert not isinstance(crit, LiveSuccessCriterion) def test_unobservable_checker_is_undecided(self) -> None: init_criteria(validate=False) checker = CriterionRegistry.get_checker("file_exists")() - assert checker.live_stop_polarities == frozenset() crit = FileExistsCriterion(type="file_exists", path="x.txt", description="x") assert checker.live_verdict(crit, [_turn()]) == "undecided" - def test_base_decidable_defaults_to_class_polarities(self) -> None: - # The base hook returns the ClassVar verbatim for a criterion that does - # NOT override it: file_exists (unobservable) reports its empty capability. - init_criteria(validate=False) - checker_cls = type(CriterionRegistry.get_checker("file_exists")()) - crit = FileExistsCriterion(type="file_exists", path="x.txt", description="x") - assert checker_cls.live_decidable_polarities(crit) == checker_cls.live_stop_polarities == frozenset() - - def test_skill_triggered_decidable_is_subset_of_class_polarities(self) -> None: - # skill_triggered DOES narrow per-instance; each instance set stays a - # subset of the class capability. + def test_skill_triggered_decidable_is_subset_of_type_universe(self) -> None: + # skill_triggered narrows per-instance; each instance set stays a + # subset of the type's universe ({"pass", "fail"}). for crit in (_skill_crit("s", "s"), _skill_crit("s", "other"), _skill_crit("s", "")): - assert SkillTriggeredChecker.live_decidable_polarities(crit) <= SkillTriggeredChecker.live_stop_polarities + assert crit.live_decidable_polarities() <= frozenset({"pass", "fail"}) # --------------------------------------------------------------------------- # @@ -509,6 +564,45 @@ def test_armed_happy_path_accepts(self) -> None: task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True) validate_early_stop(task) # no raise + def test_gate_threshold_zero_rejected(self) -> None: + # This is the hard-stop rejection for a degenerate threshold — moved + # here (not a RunLimits model validator) so it flips the plan exit + # code / aborts run like every other early-stop guardrail. + task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, gate_threshold=0.0) + with pytest.raises(EarlyStopConfigError, match=r"must be > 0\.0"): + validate_early_stop(task) + + def test_gate_threshold_positive_accepted(self) -> None: + task = _task(criteria=[_skill_crit("s", "s", stop_when="pass")], stop_early=True, gate_threshold=0.7) + validate_early_stop(task) # no raise + + def test_max_steps_to_decide_rejected_for_fail_only_criterion(self) -> None: + # A distractor (fail-only decidable) with a decision-step budget would + # force-fail a clean run whose "undecided" is its success state. + task = _task( + criteria=[_skill_crit("weather-teller", "date-teller", stop_when="fail", max_steps_to_decide=3)], + stop_early=True, + ) + with pytest.raises(EarlyStopConfigError, match="fail-only-decidable"): + validate_early_stop(task) + + def test_max_steps_to_decide_rejected_for_fail_only_command_executed(self) -> None: + # The "must-NOT-run" shape (min_count=0, max_count=0) is fail-only + # decidable too — same rejection. + task = _task( + criteria=[_cmd_crit(min_count=0, max_count=0, stop_when="fail", max_steps_to_decide=3)], + stop_early=True, + ) + with pytest.raises(EarlyStopConfigError, match="fail-only-decidable"): + validate_early_stop(task) + + def test_max_steps_to_decide_accepted_for_pass_decidable_criterion(self) -> None: + task = _task( + criteria=[_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=3)], + stop_early=True, + ) + validate_early_stop(task) # no raise + def test_armed_distractor_fail_accepts(self) -> None: # A distractor (skill_name != expected_skill) decides only "fail". task = _task(criteria=[_skill_crit("wrong", "s", stop_when="fail")], stop_early=True) @@ -598,7 +692,7 @@ def test_guardrail3_unobservable_criterion_rejected(self) -> None: def test_guardrail4_unsupported_polarity_rejected(self, monkeypatch: pytest.MonkeyPatch) -> None: # Force command_executed to be pass-only, then arm it with stop_when="fail". init_criteria(validate=False) - monkeypatch.setattr(CommandExecutedChecker, "live_stop_polarities", frozenset({"pass"})) + monkeypatch.setattr(CommandExecutedCriterion, "live_decidable_polarities", lambda self: frozenset({"pass"})) task = _task(criteria=[_cmd_crit(stop_when="fail")], stop_early=True) with pytest.raises(EarlyStopConfigError, match="polarity"): validate_early_stop(task) @@ -745,6 +839,47 @@ def test_run_surface_validates_cli_override_arming(self, tmp_path: Path) -> None with pytest.raises(EarlyStopConfigError, match="observable"): _resolve_surface(task_file, tmp_path, overrides={"run_limits.stop_early": True}) + def test_run_surface_variant_inherits_task_threshold_with_stop_early_false(self, tmp_path: Path) -> None: + # Mirrors the shipped early-stop-ab experiment: a task sets both + # stop_early: true and a non-default stop_early_gate_threshold; a + # variant flips ONLY stop_early to false (field-merged, so it + # inherits the task's threshold). This must resolve cleanly — the + # inherited-but-inert threshold is not a misconfiguration. + task_file = tmp_path / "es_layered_task.yaml" + task_file.write_text( + "task_id: es-layered-task\n" + + "description: layered threshold test\n" + + "initial_prompt: do the thing\n" + + "agent:\n" + + " type: claude-code\n" + + "sandbox:\n" + + " driver: tempdir\n" + + "run_limits:\n" + + " max_turns: 20\n" + + " stop_early: true\n" + + " stop_early_gate_threshold: 0.7\n" + + "success_criteria:\n" + + _ARMED_OBSERVABLE_CRITERION + ) + variants = [ + ExperimentVariant(variant_id="e2e", run_limits=RunLimits(stop_early=False)), + ExperimentVariant(variant_id="smoke", run_limits=RunLimits(stop_early=True)), + ] + resolved, skipped = resolve_all_tasks( + task_files=[task_file], + experiment=ExperimentDefinition(experiment_id="exp", variants=variants), + default_experiment=ExperimentDefinition( + experiment_id="default", variants=[ExperimentVariant(variant_id="default")] + ), + config=BatchRunConfig(run_dir=tmp_path / "runs", overrides={}), + ) + assert not skipped + assert len(resolved) == 2 + by_variant = {r.variant_id: r.task.run_limits for r in resolved} + assert by_variant["e2e"] is not None and by_variant["e2e"].stop_early is False + assert by_variant["e2e"].stop_early_gate_threshold == 0.7 # inherited, inert + assert by_variant["smoke"] is not None and by_variant["smoke"].stop_early is True + def _run_plan(self, task_file: Path, exp_dir: Path) -> tuple[str, int]: """Invoke the real plan_command against a minimal single-variant experiment. @@ -898,6 +1033,30 @@ async def slow_query(prompt: Any, options: Any, transport: Any = None) -> Any: return agent, sink, raised +class TestNewFixtureTasksResolve: + """Cheap resolution-only coverage for the 3 checked-in early-stop example + task YAMLs — no live agent involved. These are deliberately NOT tagged + smoke-pass/smoke-fail (their pass/fail outcome depends on non-deterministic + agent behavior), so this is their only pre-merge signal that a malformed + weight/stop_when/max_steps_to_decide combo would otherwise slip through. + """ + + @pytest.mark.parametrize( + "task_file", + [ + Path("tasks/early_stop_weighted_low_weight_absorbed.yaml"), + Path("tasks/early_stop_weighted_high_weight_kills_run.yaml"), + Path("tasks/early_stop_decision_budget_exceeded.yaml"), + ], + ) + def test_fixture_resolves_without_error(self, task_file: Path, tmp_path: Path) -> None: + resolved, skipped = _resolve_surface(task_file, tmp_path) + assert not skipped + assert len(resolved) == 1 + limits = resolved[0].task.run_limits + assert limits is not None and limits.stop_early is True + + class TestCooperativeStopSeam: def test_stopped_early_member_on_both_enums(self) -> None: assert AgentEndStatus.STOPPED_EARLY.value == "stopped_early" @@ -962,6 +1121,7 @@ class TestEarlyStopModels: def test_reason_values(self) -> None: assert EarlyStopReason.CRITERION_PASSED.value == "criterion_passed" assert EarlyStopReason.CRITERION_FAILED.value == "criterion_failed" + assert EarlyStopReason.DECISION_BUDGET_EXCEEDED.value == "decision_budget_exceeded" def test_info_defaults(self) -> None: info = EarlyStopInfo( @@ -974,6 +1134,19 @@ def test_info_defaults(self) -> None: ) assert info.armed_criteria == [] assert info.turns_remaining_at_stop is None + assert info.gate_threshold == 1.0 + + def test_gate_threshold_bounds_enforced(self) -> None: + with pytest.raises(ValueError, match="less than or equal to 1"): + EarlyStopInfo( + reason=EarlyStopReason.CRITERION_PASSED, + deciding_criterion_type="command_executed", + deciding_criterion_description="d", + sdk_turn_index=2, + tool_call_index=3, + elapsed_seconds=1.5, + gate_threshold=7.5, + ) def test_info_roundtrip(self) -> None: info = _info() @@ -1013,16 +1186,102 @@ def test_armed_criteria_passed_raises_on_empty_armed(self) -> None: with pytest.raises(ValueError, match="no armed criteria"): result.armed_criteria_passed(criteria) + def test_armed_criteria_passed_default_threshold_still_requires_all(self) -> None: + # gate_threshold=1.0 (the default) must reproduce the old all()-must-pass + # rule exactly: one armed criterion at 0.0 fails the gate regardless of + # the other armed criterion's weight. + criteria = [ + _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_when="fail", weight=0.2), + ] + low_weight_fails = _result( + criteria_results=[_crit_result("skill_triggered", 1.0), _crit_result("skill_triggered", 0.0)] + ) + assert low_weight_fails.armed_criteria_passed(criteria) is False + high_weight_fails = _result( + criteria_results=[_crit_result("skill_triggered", 0.0), _crit_result("skill_triggered", 1.0)] + ) + assert high_weight_fails.armed_criteria_passed(criteria) is False + all_pass = _result( + criteria_results=[_crit_result("skill_triggered", 1.0), _crit_result("skill_triggered", 1.0)] + ) + assert all_pass.armed_criteria_passed(criteria) is True + + def test_armed_criteria_passed_low_weight_failure_absorbed_below_threshold(self) -> None: + # The user's worked example: weights 0.8/0.2, gate_threshold 0.7. The + # LOW-weight criterion failing (weighted score 0.8) still clears 0.7; + # the HIGH-weight one failing (weighted score 0.2) does not. + criteria = [ + _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_when="fail", weight=0.2), + ] + low_weight_fails = _result( + criteria_results=[_crit_result("skill_triggered", 1.0), _crit_result("skill_triggered", 0.0)] + ) + assert low_weight_fails.armed_criteria_passed(criteria, gate_threshold=0.7) is True + high_weight_fails = _result( + criteria_results=[_crit_result("skill_triggered", 0.0), _crit_result("skill_triggered", 1.0)] + ) + assert high_weight_fails.armed_criteria_passed(criteria, gate_threshold=0.7) is False + + def test_armed_criteria_passed_still_honors_pass_threshold(self) -> None: + # Each armed criterion's own pass_threshold still decides whether IT + # individually passed (converted to binary 1.0/0.0) before weighting — + # only the combination rule (AND vs weighted average) changes. A + # score of 0.5 fails a pass_threshold of 0.99, so it must NOT clear + # even a low gate_threshold: pass_threshold is not bypassable by + # lowering gate_threshold. + criteria = [_skill_crit("date-teller", "date-teller", stop_when="pass", pass_threshold=0.99)] + result = _result(criteria_results=[_crit_result("skill_triggered", 0.5)]) + assert result.armed_criteria_passed(criteria, gate_threshold=0.1) is False + # A score that DOES clear its own pass_threshold (0.5 >= 0.4) passes. + criteria_lenient = [_skill_crit("date-teller", "date-teller", stop_when="pass", pass_threshold=0.4)] + assert result.armed_criteria_passed(criteria_lenient, gate_threshold=0.1) is True + + def test_armed_criteria_passed_gate_equivalence_at_default_threshold(self) -> None: + # Property pin: at gate_threshold=1.0 (the default), armed_criteria_passed + # must agree with all(r.score >= c.pass_threshold), for ANY pass_threshold + # — not just the binary-scoring case. This is the equivalence the + # docstring claims; it must hold exactly, not merely "in practice". + for score, pass_threshold, weight in [ + (1.0, 0.9, 1.0), + (0.0, 0.9, 1.0), + (0.5, 0.99, 0.8), # fails its own threshold + (0.5, 0.4, 0.2), # clears its own threshold despite a low score + (0.0, 0.0, 1.0), # pass_threshold: 0.0 — the non-gating-arming escape hatch + ]: + criteria = [_skill_crit("s", "s", stop_when="pass", weight=weight, pass_threshold=pass_threshold)] + result = _result(criteria_results=[_crit_result("skill_triggered", score)]) + expected = score >= pass_threshold + assert result.armed_criteria_passed(criteria) is expected, (score, pass_threshold, weight) + + def test_armed_criteria_passed_weighted_gate_with_command_executed(self) -> None: + # The other LiveSuccessCriterion subclass exercised through the same + # weighted gate — command_executed, not just skill_triggered. + criteria = [ + _cmd_crit(min_count=1, max_count=None, stop_when="pass", weight=0.8), + _cmd_crit(min_count=0, max_count=0, stop_when="fail", weight=0.2), + ] + low_weight_fails = _result( + criteria_results=[_crit_result("command_executed", 1.0), _crit_result("command_executed", 0.0)] + ) + assert low_weight_fails.armed_criteria_passed(criteria, gate_threshold=0.7) is True + high_weight_fails = _result( + criteria_results=[_crit_result("command_executed", 0.0), _crit_result("command_executed", 1.0)] + ) + assert high_weight_fails.armed_criteria_passed(criteria, gate_threshold=0.7) is False + # --------------------------------------------------------------------------- # # Phase 3: EarlyStopWatcher # --------------------------------------------------------------------------- # -def _watcher(criteria: list[Any], *, max_turns: int | None = 20) -> EarlyStopWatcher: +def _watcher(criteria: list[Any], *, max_turns: int | None = 20, gate_threshold: float = 1.0) -> EarlyStopWatcher: task = _task(criteria=criteria, stop_early=True) assert task.run_limits is not None task.run_limits.max_turns = max_turns + task.run_limits.stop_early_gate_threshold = gate_threshold return EarlyStopWatcher.for_task(task) @@ -1218,6 +1477,155 @@ def test_mixed_static_arming_pass_stops_ignoring_fail_armed(self) -> None: assert watcher.info is not None assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + def test_ceiling_bound_defers_fail_stop_below_default_gate_threshold(self) -> None: + # The user's worked example on the trigger side: weights 0.8/0.2, + # gate_threshold 0.7. The LOW-weight (0.2) criterion misfiring leaves a + # ceiling of 0.8 (>= 0.7) — the gate could still pass if the high-weight + # positive comes through, so the run must NOT stop yet. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_when="auto", weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_when="auto", weight=0.2), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("weather-teller")) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_ceiling_bound_fires_fail_stop_when_high_weight_criterion_fails(self) -> None: + # Mirror case: the HIGH-weight (0.8) positive misfiring as a distractor + # leaves a ceiling of 0.2 (< 0.7) — the gate can never reach 0.7 no + # matter what the low-weight criterion does, so the fail-stop must fire + # even though it's the "small" criterion still undecided. + watcher = _watcher( + [ + _skill_crit("weather-teller", "date-teller", stop_when="auto", weight=0.8), + _skill_crit("news-teller", "date-teller", stop_when="auto", weight=0.2), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("weather-teller")) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_FAILED + + def test_default_gate_threshold_fires_fail_stop_on_any_weight(self) -> None: + # At the default gate_threshold=1.0, even the low-weight criterion's + # failure alone must still fire — byte-for-byte the pre-weighting rule. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_when="auto", weight=0.8), + _skill_crit("weather-teller", "date-teller", stop_when="auto", weight=0.2), + ] + ) + _feed(watcher, _skill_events("weather-teller")) + assert watcher.should_stop() is False # deferred: positive still undecided + _feed(watcher, [_tool_end(_skill_cmd("date-teller", tool_id="d"))]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_FAILED + + def test_floor_bound_pass_stops_before_low_weight_distractor_decides(self) -> None: + # Floor generalization on the pass side: a high-weight (0.9) positive + # pass-armed criterion passing is enough to pass-stop on its own — + # there is no OTHER pass-armed criterion whose weight it needs to share + # the floor with (fail-armed distractors are excluded from the + # pass-armed floor by design either way). + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.9), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("date-teller")) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_floor_bound_pass_stop_requires_full_pass_armed_subset_below_default(self) -> None: + # Below the default threshold, a partially-decided pass-armed subset + # (one of two passed) must NOT pass-stop yet if the still-undecided + # one's weight share would drop the floor below the threshold. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_when="pass", weight=0.5), + _skill_crit("weather-teller", "weather-teller", stop_when="pass", weight=0.5), + ], + gate_threshold=0.7, + ) + _feed(watcher, _skill_events("date-teller")) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_decision_budget_exceeded_when_still_undecided(self) -> None: + # An armed criterion capped at max_steps_to_decide=1 that is still + # "undecided" after its first tool call forces a budget-exceeded stop. + # Full-field EarlyStopInfo parity, matching every other stop-reason test. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=1)]) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert watcher.info.deciding_criterion_type == "skill_triggered" + assert watcher.info.deciding_criterion_description == "date-teller activation" + assert watcher.info.sdk_turn_index == 1 + assert watcher.info.tool_call_index == 1 + + def test_decision_budget_exceeded_names_the_right_criterion_among_several(self) -> None: + # Multiple armed criteria with different budgets: only the SECOND + # one's budget has expired (cap=1, undecided after 1 call); the first + # has a longer budget (cap=5) and is also still undecided. The + # deciding criterion reported must be the one whose budget actually + # tripped, not just the first armed criterion in list order. + watcher = _watcher( + [ + _skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=5), + _skill_crit("weather-teller", "weather-teller", stop_when="pass", max_steps_to_decide=1), + ] + ) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert watcher.info.deciding_criterion_description == "weather-teller activation" + + def test_decision_budget_exceeded_with_command_executed(self) -> None: + # The other LiveSuccessCriterion subclass: a command_executed pass-armed + # criterion (min_count=1, no upper bound) capped at max_steps_to_decide=1 + # that never sees a matching command force-fails identically. + watcher = _watcher([_cmd_crit(min_count=1, max_count=None, stop_when="pass", max_steps_to_decide=1)]) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert watcher.info.deciding_criterion_type == "command_executed" + + def test_decision_budget_not_exceeded_below_cap(self) -> None: + # Same cap, but only reached on the FIRST tool call (index 1) — a cap of + # 2 must not fire yet. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=2)]) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is False + assert watcher.info is None + + def test_real_decision_within_budget_wins_over_budget_check(self) -> None: + # The criterion decides (pass-stops) on the SAME tool call that would + # otherwise have tripped its budget — the real decision takes priority. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=1)]) + _feed(watcher, _skill_events("date-teller")) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.CRITERION_PASSED + + def test_decision_budget_ignored_when_unset(self) -> None: + # No max_steps_to_decide -> no budget check, run continues indefinitely + # (up to run_limits.max_turns) while undecided. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is False + assert watcher.info is None + def test_records_turn_and_tool_index(self) -> None: watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass")]) _feed(watcher, _skill_events("date-teller")) @@ -1355,6 +1763,23 @@ def test_second_agent_start_does_not_reset_origin(self) -> None: assert watcher.info is not None assert watcher.info.elapsed_seconds >= 0.0 + def test_decision_budget_accumulates_across_retry_attempts(self) -> None: + # Pins the documented contract (max_steps_to_decide's field + # description + TASK_DEFINITION_GUIDE.md): the step count is + # CUMULATIVE across every retry attempt of the turn — a second + # AgentStartEvent (as on a retry) must NOT reset tool_call_index. A + # future per-attempt reset would silently change scoring with this + # test catching it. + watcher = _watcher([_skill_crit("date-teller", "date-teller", stop_when="pass", max_steps_to_decide=2)]) + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))]) + assert watcher.should_stop() is False # 1 call so far, budget is 2 + # A retry: a second AgentStartEvent must not reset the counter. + _feed(watcher, [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo bye"}))]) + assert watcher.should_stop() is True + assert watcher.info is not None + assert watcher.info.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert watcher.info.tool_call_index == 2 + # --------------------------------------------------------------------------- # # Phase 3: Orchestrator wiring @@ -1401,13 +1826,14 @@ async def _run_wiring( stop_early: bool, tmp_path, agent_type: AgentKind = AgentKind.CLAUDE_CODE, -) -> tuple[EvaluationResult, _ScriptedAgent]: + gate_threshold: float = 1.0, +) -> tuple[EvaluationResult, _ScriptedAgent, bool]: """Drive ``Orchestrator._evaluation_loop`` with a scripted agent + mock checker. ``scores`` are positional CriterionResult scores matching ``criteria``. The early-stop watcher is built directly (_setup is not invoked here). """ - task = _task(criteria=criteria, stop_early=stop_early, agent_type=agent_type) + task = _task(criteria=criteria, stop_early=stop_early, agent_type=agent_type, gate_threshold=gate_threshold) run_dir = tmp_path / "run" run_dir.mkdir(parents=True) orch = Orchestrator(task=task, run_dir=run_dir, variant_id="default") @@ -1440,9 +1866,9 @@ async def _run_wiring( orch.agent = agent # type: ignore[assignment] with patch("coder_eval.orchestrator.load_reference", return_value=(None, None, None)): - await orch._evaluation_loop() + success = await orch._evaluation_loop() assert orch.result is not None - return orch.result, agent + return orch.result, agent, success class TestOrchestratorEarlyStopWiring: @@ -1464,7 +1890,7 @@ def _distractor_criteria(self) -> list[Any]: async def test_default_off_full_gate_no_early_stop(self, tmp_path) -> None: # Unarmed: no watcher, all criteria gate, advisory 0.0 drags to FAILURE. - result, agent = await _run_wiring( + result, agent, _success = await _run_wiring( criteria=self._criteria(stop_when=None), events=_skill_events(self._SKILL), scores=[1.0, 0.0], @@ -1477,7 +1903,7 @@ async def test_default_off_full_gate_no_early_stop(self, tmp_path) -> None: async def test_pass_stop_cuts_the_stream(self, tmp_path) -> None: # A trailing event AFTER the deciding ToolEnd proves the cut: delivered == 3. events = [*_skill_events(self._SKILL), _turn_start()] - result, agent = await _run_wiring( + result, agent, _success = await _run_wiring( criteria=self._criteria(), events=events, scores=[1.0, 0.0], @@ -1490,7 +1916,7 @@ async def test_pass_stop_cuts_the_stream(self, tmp_path) -> None: async def test_fail_stop_wiring(self, tmp_path) -> None: # A distractor (armed fail) fires the fail-stop when its skill is engaged. - result, _agent = await _run_wiring( + result, _agent, _success = await _run_wiring( criteria=self._distractor_criteria(), events=_skill_events("weather-teller"), scores=[0.0, 0.0], @@ -1501,7 +1927,7 @@ async def test_fail_stop_wiring(self, tmp_path) -> None: assert result.early_stop.reason == EarlyStopReason.CRITERION_FAILED async def test_early_stop_info_fields_populated(self, tmp_path) -> None: - result, _agent = await _run_wiring( + result, _agent, _success = await _run_wiring( criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], @@ -1515,7 +1941,7 @@ async def test_early_stop_info_fields_populated(self, tmp_path) -> None: async def test_advisory_not_gated_on_early_stop(self, tmp_path) -> None: # Armed skill passes (1.0), advisory file_exists fails (0.0): armed gate -> SUCCESS. - result, _agent = await _run_wiring( + result, _agent, _success = await _run_wiring( criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], @@ -1526,10 +1952,55 @@ async def test_advisory_not_gated_on_early_stop(self, tmp_path) -> None: assert result.all_criteria_passed(self._criteria()) is False # full gate would fail assert result.armed_criteria_passed(self._criteria()) is True # armed gate passes - async def test_completed_naturally_uses_full_gate(self, tmp_path) -> None: - # Armed, but the skill is never engaged -> watcher never fires -> full gate, - # so the advisory 0.0 legitimately drags the completed run to FAILURE. - result, agent = await _run_wiring( + async def test_decision_budget_exceeded_forces_failure_bypassing_gate(self, tmp_path) -> None: + # A criterion capped at max_steps_to_decide=1 that never engages its + # skill forces a hard fail — even though BOTH mocked criterion scores + # are 1.0 (the weighted gate, if consulted, would pass). + criteria = [ + _skill_crit(self._SKILL, self._SKILL, stop_when="pass", max_steps_to_decide=1), + FileExistsCriterion(path="artifact.txt", description="artifact must exist"), + ] + task = _task(criteria=criteria, stop_early=True) + run_dir = tmp_path / "run" + run_dir.mkdir(parents=True) + orch = Orchestrator(task=task, run_dir=run_dir, variant_id="default") + orch.result = EvaluationResult( + task_id=task.task_id, + task_description=task.description, + variant_id="default", + agent_type=AgentKind.CLAUDE_CODE, + started_at=datetime.now(), + final_status=FinalStatus.FAILURE, + iteration_count=0, + environment_info={}, + ) + sandbox = MagicMock() + sandbox.sandbox_dir = tmp_path / "sandbox" + sandbox.sandbox_dir.mkdir() + orch.sandbox = sandbox + checker = MagicMock() + checker.check_all_async = AsyncMock(return_value=[_crit_result(c.type, 1.0) for c in criteria]) + orch.success_checker = checker + orch._early_stop_watcher = EarlyStopWatcher.for_task(task) + turn = TurnRecord(iteration=1, user_input="p", agent_output="done") + events = [_agent_start(), _turn_start(), _tool_end(_cmd("Bash", {"command": "echo hi"}))] + agent = _ScriptedAgent(events, turn) + orch.agent = agent # type: ignore[assignment] + + with patch("coder_eval.orchestrator.load_reference", return_value=(None, None, None)): + success = await orch._evaluation_loop() + + assert orch.result.early_stop is not None + assert orch.result.early_stop.reason == EarlyStopReason.DECISION_BUDGET_EXCEEDED + assert success is False + + async def test_completed_naturally_still_uses_armed_gate(self, tmp_path) -> None: + # Armed for early-stop, but the skill is never engaged -> watcher never + # fires -> the run completes naturally. The armed subset STILL gates + # final_status (not the full set): the armed criterion itself scored + # 0.0, so _evaluation_loop's real return value is False — a genuine + # armed-gate failure, not a full-gate one (though both agree here). + result, agent, success = await _run_wiring( criteria=self._criteria(), events=[_agent_start(), _turn_start()], # no skill engagement scores=[0.0, 0.0], @@ -1538,7 +2009,71 @@ async def test_completed_naturally_uses_full_gate(self, tmp_path) -> None: ) assert result.early_stop is None assert agent.delivered == 2 # full (short) stream consumed - assert result.all_criteria_passed(self._criteria()) is False + assert success is False + + async def test_completed_naturally_armed_gate_forgives_advisory_failure(self, tmp_path) -> None: + # THE fix this test pins: same never-fired scenario, but the ARMED + # criterion passes (1.0) while the ADVISORY one fails (0.0). Under the + # old full-set gate this would be FAILURE (the advisory 0.0 drags it + # down); under the fixed armed-gate-always-applies-when-stop_early + # semantics _evaluation_loop's real return value is True — one task + # config, one gate semantic, regardless of whether the watcher + # physically fired. + result, agent, success = await _run_wiring( + criteria=self._criteria(), + events=[_agent_start(), _turn_start()], # no skill engagement -> watcher never fires + scores=[1.0, 0.0], + stop_early=True, + tmp_path=tmp_path, + ) + assert result.early_stop is None + assert agent.delivered == 2 + assert result.all_criteria_passed(self._criteria()) is False # the full gate WOULD fail + assert success is True # but the armed gate is what actually decided this run + + async def test_gate_threshold_plumbing_end_to_end(self, tmp_path) -> None: + # Mutation-resistant pin for the two plumbing hops the reviewer + # flagged as untested: YAML stop_early_gate_threshold -> the final + # gate (orchestrator.py) -> _evaluation_loop's real return value, AND + # -> the persisted EarlyStopInfo.gate_threshold. Weighted criteria + # (0.8/0.2), watcher never fires (never touches either skill), so + # this exercises the natural-completion armed-gate path directly. + criteria = [ + _skill_crit(self._SKILL, self._SKILL, stop_when="pass", weight=0.8), + _skill_crit("weather-teller", self._SKILL, stop_when="fail", weight=0.2), + ] + _result_default, _agent, success_default = await _run_wiring( + criteria=criteria, + events=[_agent_start(), _turn_start()], + scores=[1.0, 0.0], + stop_early=True, + tmp_path=tmp_path / "a", + gate_threshold=1.0, + ) + assert success_default is False # 0.8 < 1.0 + _result_low, _agent2, success_low = await _run_wiring( + criteria=criteria, + events=[_agent_start(), _turn_start()], + scores=[1.0, 0.0], + stop_early=True, + tmp_path=tmp_path / "b", + gate_threshold=0.7, + ) + assert success_low is True # 0.8 >= 0.7 — a mutation to a literal 1.0 would flip this + + async def test_gate_threshold_persisted_on_early_stop_info(self, tmp_path) -> None: + # The second plumbing hop: the fired watcher's own EarlyStopInfo + # carries the threshold that was actually in effect. + result, _agent, _success = await _run_wiring( + criteria=self._criteria(), + events=_skill_events(self._SKILL), + scores=[1.0, 0.0], + stop_early=True, + tmp_path=tmp_path, + gate_threshold=0.7, + ) + assert result.early_stop is not None + assert result.early_stop.gate_threshold == 0.7 # a mutation to a literal 1.0 would flip this async def test_completed_run_with_orphan_tool_not_early_stopped(self, tmp_path) -> None: # Regression: a run that completes naturally, whose finalize() force-closes @@ -1546,7 +2081,7 @@ async def test_completed_run_with_orphan_tool_not_early_stopped(self, tmp_path) # early-stopped — the full gate applies and the advisory 0.0 drags to # FAILURE (rather than a false "stopped early; N turns avoided"). events = [_agent_start(), _turn_start(), _unresolved_skill_end(self._SKILL)] - result, agent = await _run_wiring( + result, agent, _success = await _run_wiring( criteria=self._criteria(), events=events, scores=[1.0, 0.0], @@ -1562,7 +2097,7 @@ async def test_tool_call_cut_without_tool_end(self, tmp_path) -> None: # the stream and records an early stop — the case that would otherwise run # to the turn cap when a cut-short turn strips the result. events = [_agent_start(), _turn_start(), _skill_start(self._SKILL), _turn_start()] - result, agent = await _run_wiring( + result, agent, _success = await _run_wiring( criteria=self._criteria(), events=events, scores=[1.0, 0.0], @@ -1576,7 +2111,7 @@ async def test_tool_call_cut_without_tool_end(self, tmp_path) -> None: async def test_fail_open_wiring_degrades_to_full_run(self, tmp_path) -> None: with patch.object(SkillTriggeredChecker, "live_verdict", side_effect=RuntimeError("boom")): - result, _agent = await _run_wiring( + result, _agent, _success = await _run_wiring( criteria=self._criteria(), events=_skill_events(self._SKILL), scores=[1.0, 0.0], @@ -1626,12 +2161,18 @@ def test_task_dict_keys_present_when_early_stopped(self) -> None: assert d["stopped_early"] is True assert d["early_stop_reason"] == "criterion_passed" assert d["turns_remaining_at_stop"] == 14 + assert d["gate_threshold"] == 1.0 def test_task_dict_keys_defaulted_when_not_early_stopped(self) -> None: d = eval_result_to_task_dict(_result()) assert d["stopped_early"] is False assert d["early_stop_reason"] is None assert d["turns_remaining_at_stop"] is None + assert d["gate_threshold"] is None + + def test_task_dict_reflects_decision_budget_exceeded(self) -> None: + d = eval_result_to_task_dict(_stopped_result(reason=EarlyStopReason.DECISION_BUDGET_EXCEEDED)) + assert d["early_stop_reason"] == "decision_budget_exceeded" def test_runtime_note_rendered_with_turns_avoided(self) -> None: lines = ReportGenerator._runtime_notes_lines(_run_summary([eval_result_to_task_dict(_stopped_result())])) @@ -1640,6 +2181,18 @@ def test_runtime_note_rendered_with_turns_avoided(self) -> None: assert "<= 14 turn(s) avoided" in blob assert "gated on armed criteria only; other criteria are advisory" in blob + def test_runtime_note_for_decision_budget_exceeded_is_not_misleading(self) -> None: + # The budget-exceeded reason forces FAILURE outright — NO criterion + # gated here, unlike a real early stop. The note must say so, not the + # generic "gated on armed criteria" text (which would tell the reader + # the opposite of what happened). + result = _stopped_result(reason=EarlyStopReason.DECISION_BUDGET_EXCEEDED) + lines = ReportGenerator._runtime_notes_lines(_run_summary([eval_result_to_task_dict(result)])) + blob = "\n".join(lines) + assert "stopped early (decision_budget_exceeded)" in blob + assert "gated on armed criteria only" not in blob + assert "forced to FAILURE" in blob + def test_runtime_note_absent_for_unarmed_run(self) -> None: lines = ReportGenerator._runtime_notes_lines(_run_summary([eval_result_to_task_dict(_result())])) assert not any("stopped early" in line for line in lines) @@ -1670,6 +2223,13 @@ def test_telemetry_dims_reflect_early_stop(self) -> None: assert props2["EarlyStopped"] is False assert props2["EarlyStopReason"] == "" + def test_telemetry_dims_reflect_decision_budget_exceeded(self) -> None: + _name, props = build_task_event( + _stopped_result(reason=EarlyStopReason.DECISION_BUDGET_EXCEEDED), driver="tempdir", variant_id="v" + ) + assert props["EarlyStopped"] is True + assert props["EarlyStopReason"] == "decision_budget_exceeded" + # --------------------------------------------------------------------------- # # Cooperative should_stop seam on CodexAgent — mirrors TestCooperativeStopSeam. @@ -2125,7 +2685,7 @@ def _criteria(self) -> list[Any]: async def test_pass_stop_populates_early_stop_and_armed_gate(self, tmp_path) -> None: # A trailing event AFTER the deciding ToolEnd proves the cut: delivered == 3. events = [*_skill_events(self._SKILL), _turn_start()] - result, agent = await _run_wiring( + result, agent, _success = await _run_wiring( criteria=self._criteria(), events=events, scores=[1.0, 0.0], diff --git a/uv.lock b/uv.lock index 39552209..798599f9 100644 --- a/uv.lock +++ b/uv.lock @@ -42,7 +42,7 @@ wheels = [ [[package]] name = "aiohttp" -version = "3.14.1" +version = "3.14.3" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "aiohappyeyeballs" }, @@ -53,72 +53,72 @@ dependencies = [ { name = "propcache" }, { name = "yarl" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/82/78/8ea7308cac6934de8c74a14f3d5f65d1c89287426688be79538d0e5c013d/aiohttp-3.14.1.tar.gz", hash = "sha256:307f2cff90a764d329e77040603fa032db89c5c24fdad50c4c15334cba744035", size = 7955794, upload-time = "2026-06-07T21:09:35.529Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/bc/97/bd137012dd97e1649162b099135a80e1fd59aaa807b2430fc448d1029aff/aiohttp-3.14.1-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:b3a03285a7f9c7b016324574a6d92a1c895da6b978cb8f1deee3ac72bc6da178", size = 506882, upload-time = "2026-06-07T21:07:15.501Z" }, - { url = "https://files.pythonhosted.org/packages/ef/79/e5cc690e9d922a66887ceeaca53a8ffd5a7b0be3816142b7abc433742d89/aiohttp-3.14.1-cp313-cp313-android_21_x86_64.whl", hash = "sha256:2a73f487ab8ef5abbb24b7aa9b73e98eaba9e9e031804ff2416f02eca315ccaf", size = 515270, upload-time = "2026-06-07T21:07:17.53Z" }, - { url = "https://files.pythonhosted.org/packages/fe/22/a73ccbf9dbd6e26dda0b24d5fd5db7da92ee3383a79f47677ffb834c5c5b/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:915fbb7b41b115192259f8c9ae58f3ddc444d2b5579917270211858e606a4afd", size = 485841, upload-time = "2026-06-07T21:07:19.555Z" }, - { url = "https://files.pythonhosted.org/packages/3b/b9/57ed8eaf596321c2ad747bd480fb1700dbd7177c60dfc9e4c187f629662e/aiohttp-3.14.1-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:7fb4bdf95b0561a79f259f9d28fbc109728c5ee7f27aff6391f0ca703a329abe", size = 492088, upload-time = "2026-06-07T21:07:21.581Z" }, - { url = "https://files.pythonhosted.org/packages/78/c0/5ebe5270a7c140d7c6f79dcb018640225f14d406c149e4eec04a7d82fe71/aiohttp-3.14.1-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:1b9748363260121d2927704f5d4fc498150669ca3ae93625986ee89c8f80dcd4", size = 501564, upload-time = "2026-06-07T21:07:23.388Z" }, - { url = "https://files.pythonhosted.org/packages/75/7f/8cdaa24fc7983865e0915153b96a9ac5bcdd3548d64c5a27d17cecccad2d/aiohttp-3.14.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:86a6dab78b0e43e2897a3bbe15745aa60dc5423ca437b7b0b164c069bf91b876", size = 751998, upload-time = "2026-06-07T21:07:25.046Z" }, - { url = "https://files.pythonhosted.org/packages/b2/f4/c4227aacfacc5cb0cc2d119b65301d177912a6842cd64e120c47af76064f/aiohttp-3.14.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:4dfd6e47d3c44c2279907607f73a4240b88c69eb8b90da7e2441a8045dfd21da", size = 510918, upload-time = "2026-06-07T21:07:27.28Z" }, - { url = "https://files.pythonhosted.org/packages/ab/01/a2d5f96cd4e74424864d30bc0a7e44d0a12dacdcfa91b5b2d1bd3dca6bf3/aiohttp-3.14.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:317acd9f8602858dc7d59679812c376c7f0b97bcbbf16e0d6237f54141d8a8a6", size = 508657, upload-time = "2026-06-07T21:07:29.252Z" }, - { url = "https://files.pythonhosted.org/packages/e8/ed/3c0fb5c500fdd8e7ebc10d1889c04384fffa1a9163eac1356088ca9da1b1/aiohttp-3.14.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bd869c427324e5cb15195793de951295710db28be7d818247f3097b4ab5d4b96", size = 1757907, upload-time = "2026-06-07T21:07:31.03Z" }, - { url = "https://files.pythonhosted.org/packages/0b/ab/d4c924d9bd5be3050c226612413ce68cb54c70d2c31b661bfc8d9a5b6a70/aiohttp-3.14.1-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:93b032b5ec3255473c143627d21a69ac74ae12f7f33974cb587c564d11b1066f", size = 1737565, upload-time = "2026-06-07T21:07:33.031Z" }, - { url = "https://files.pythonhosted.org/packages/19/2a/37326821ff779084020cdc33224d20b19f42f4183a500ff92022a739eda7/aiohttp-3.14.1-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:f234b4deb12f3ad59127e037bc57c40c21e45b45282df7d3a55a0f409f595296", size = 1799018, upload-time = "2026-06-07T21:07:35.003Z" }, - { url = "https://files.pythonhosted.org/packages/b3/4f/6e947ba73e4ce09070761c05ed3a8ceb7c21f5e46798671d8b2aac0e4626/aiohttp-3.14.1-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:9af6779bfb46abf124068327abcdf9ce95c9ef8287a3e8da76ccf2d0f16c28fa", size = 1894416, upload-time = "2026-06-07T21:07:36.956Z" }, - { url = "https://files.pythonhosted.org/packages/9d/6e/dbf1d0625dc711fb2851f4f3c3055c39ed58bae92082d8c627dbe6013736/aiohttp-3.14.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:faccab372e66bc76d5731525e7f1143c922271725b9d38c9f97edcc66266b451", size = 1783881, upload-time = "2026-06-07T21:07:39.063Z" }, - { url = "https://files.pythonhosted.org/packages/44/c2/5e25098a67268ed369483ae7d1a58bd0a13d03aab860d2a0e4a6eb25b046/aiohttp-3.14.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:f380468b09d2a81633ee863b0ec5648d364bd17bb8ecfb8c2f387f7ac1faf42c", size = 1587572, upload-time = "2026-06-07T21:07:41.058Z" }, - { url = "https://files.pythonhosted.org/packages/2a/bd/cf9cee17e140f942a3de73e658a543aa8fbf35a5fc67a9d2538d52d77f0b/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:97e704dcd26271f5bda3fa07c3ce0fb76d6d3f8659f4baa1a24442cc9ba177ca", size = 1722137, upload-time = "2026-06-07T21:07:43.014Z" }, - { url = "https://files.pythonhosted.org/packages/89/6d/5684f8c59045c96f81a18cefbc1fbbd79d25b88f1c622f2a5c5c08fcb632/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:269b76ac5394092b95bc4a098f4fc6c191c083c3bd12775d1e30e663132f6a09", size = 1755953, upload-time = "2026-06-07T21:07:45.933Z" }, - { url = "https://files.pythonhosted.org/packages/a8/40/35caf3170f8359760740a7d9aa0fff2e344bef98e1d1186f5a0f6dec17e6/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:5c0b3e614340c889d575451696374c9d17affd54cd607ca0babed8f8c37b9397", size = 1766479, upload-time = "2026-06-07T21:07:48.047Z" }, - { url = "https://files.pythonhosted.org/packages/6d/a1/b0c61e7a137f0d81de49a82023a6df73c3c16d6fefb0f8e4a93d21639002/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:5663ee9257cfa1add7253a7da3035a02f31b6600ec48261585e1800a81533080", size = 1580077, upload-time = "2026-06-07T21:07:50.069Z" }, - { url = "https://files.pythonhosted.org/packages/0b/41/194ea4623693009fcefebef7aef63c141754f153e9cd0d39d3b9e36c175c/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:603a2c834142172ffddc054067f5ec0ca65d57a0aa98a71bc81952573208e345", size = 1791688, upload-time = "2026-06-07T21:07:52.106Z" }, - { url = "https://files.pythonhosted.org/packages/ba/45/4de841f005cfe1fd63e2a2fe011262c515e2a62aa6994b15947e7d717ac9/aiohttp-3.14.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cb21957bb8aca671c1765e32f58164cf0c50e6bf41c0bbbd16da20732ecaf588", size = 1761094, upload-time = "2026-06-07T21:07:54.113Z" }, - { url = "https://files.pythonhosted.org/packages/e4/ae/dbce10533d3896d544d5053939ed75b7dc31a1b0973d959b1b5ae21028d6/aiohttp-3.14.1-cp313-cp313-win32.whl", hash = "sha256:e509a55f681e6158c20f70f102f9cf61fb20fbc382272bc6d94b7343f2582780", size = 452662, upload-time = "2026-06-07T21:07:56.06Z" }, - { url = "https://files.pythonhosted.org/packages/7b/d9/0bf1a19362c32f06229da5e7ddfcec91f93474d6307f7a2d3135e9c674dc/aiohttp-3.14.1-cp313-cp313-win_amd64.whl", hash = "sha256:1ac8531b638959718e18c2207fbfe297819875da46a740b29dfa29beba64355a", size = 479748, upload-time = "2026-06-07T21:07:58.319Z" }, - { url = "https://files.pythonhosted.org/packages/22/0a/62e7232dc9484fbec112ceb32efb6a624cc7994ec6e2b019286f17c4e8f2/aiohttp-3.14.1-cp313-cp313-win_arm64.whl", hash = "sha256:250d14af67f6b6a1a4a811049b1afa69d61d617fca6bf33149b3ab1a6dbcf7b8", size = 447723, upload-time = "2026-06-07T21:08:00.154Z" }, - { url = "https://files.pythonhosted.org/packages/c4/a1/5fafa04e1ca91ddb47608699d60649c1c6db3cf41c99e78fc4056f9513db/aiohttp-3.14.1-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:7c106c26852ca1c2047c6b80384f17100b4e439af276f21ef3d4e2f450ae7e15", size = 508531, upload-time = "2026-06-07T21:08:02.093Z" }, - { url = "https://files.pythonhosted.org/packages/fa/2e/bfa02f699d87ffc86d5959270b28f1cb410add3ccaced8ed2e0b8a5238fc/aiohttp-3.14.1-cp314-cp314-android_24_x86_64.whl", hash = "sha256:20205f7f5ade7aaec9f4b500549bbc071b046453aed72f9c06dcab87896a83e8", size = 514718, upload-time = "2026-06-07T21:08:04.476Z" }, - { url = "https://files.pythonhosted.org/packages/85/a5/9594ad6289eebbc97d167c44213d557807f90e59115caad24de21ad2c3b1/aiohttp-3.14.1-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:62a759436b29e677181a9e76bab8b8f689a29cb9c535f45f7c48c9c830d3f8c3", size = 487918, upload-time = "2026-06-07T21:08:06.377Z" }, - { url = "https://files.pythonhosted.org/packages/b4/61/16a32c36c3c49edec122a3dc811f2057df2f94d3b14aa107c8017d981618/aiohttp-3.14.1-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:2964cbf553df4d7a57348da44d961d871895fc1ee4e8c322b2a95612c7b17fba", size = 494014, upload-time = "2026-06-07T21:08:08.263Z" }, - { url = "https://files.pythonhosted.org/packages/9b/89/3ebcf96ed99c05bec9c434aaac6963fd3cbab4a786ae739908a144d9ce44/aiohttp-3.14.1-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:237651caadc3a59badd39319c54642b5299e9cc98a3a194310e55d5bb9f5e397", size = 502398, upload-time = "2026-06-07T21:08:10.244Z" }, - { url = "https://files.pythonhosted.org/packages/fd/3d/b74870a0c2d40c355928cd5b96c7a11fa821b8a40fc41365e64479b151fb/aiohttp-3.14.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:896e12dfdbbab9d8f7e16d2b28c6769a60126fa92095d1ebf9473d02593a2448", size = 758018, upload-time = "2026-06-07T21:08:12.447Z" }, - { url = "https://files.pythonhosted.org/packages/d3/66/f42f5c984d99e49c6cff5f26f590750f2e2f7ef1fcfb99966ab5be1b632e/aiohttp-3.14.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:d03f281ed22579314ba00821ce20115a7c0ac430660b4cc05704a3f818b3e004", size = 512462, upload-time = "2026-06-07T21:08:14.624Z" }, - { url = "https://files.pythonhosted.org/packages/e9/a7/248e1aebe0c7810b0271e021a0f2a5eb6e78a051885b3c9df49f42a5802d/aiohttp-3.14.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:07eabb979d236335fed927e137a928c9adfb7df3b9ec7aa31726f133a62be983", size = 512824, upload-time = "2026-06-07T21:08:16.572Z" }, - { url = "https://files.pythonhosted.org/packages/26/97/2aa0e5ba0727dc3bd5aaebb7ccbc510f7dfb7fb961ec87497cd496635ab1/aiohttp-3.14.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4fe1f1087cbadb280b5e1bb054a4f00d1423c74d6626c5e48400d871d34ecefe", size = 1749898, upload-time = "2026-06-07T21:08:18.635Z" }, - { url = "https://files.pythonhosted.org/packages/00/8d/e97f6c96c891d457c8479d92a514ba194d0412f981d72c70341ee18488ed/aiohttp-3.14.1-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:367a9314fdc79dab0fac96e216cb41dd73c85bdca85306ce8999118ba7e0f333", size = 1710114, upload-time = "2026-06-07T21:08:20.892Z" }, - { url = "https://files.pythonhosted.org/packages/6f/e6/aa8d7e863048c8fceb5cd6ce74017311cec3ead07847387e12265fb4444e/aiohttp-3.14.1-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:a24f677ebe83749039e7bdf862ff0bbb16818ae4193d4ef96505e269375bcce0", size = 1802541, upload-time = "2026-06-07T21:08:23.044Z" }, - { url = "https://files.pythonhosted.org/packages/83/a8/72193137de57fda4ebfae4563182d082c8856e3b6e9871d0b46f028fb369/aiohttp-3.14.1-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c83afe0ba876be7e943d2e0ba645809ad441575d2840c895c21ee5de93b9377a", size = 1875776, upload-time = "2026-06-07T21:08:25.288Z" }, - { url = "https://files.pythonhosted.org/packages/a0/18/938441025db6769a3464596b2410af3afde0b21eb2f204c6f766f68af4bd/aiohttp-3.14.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:634e385930fb6d2d479cf3aa66515955863b77a5e3c2b5894ca259a25b308602", size = 1760329, upload-time = "2026-06-07T21:08:27.363Z" }, - { url = "https://files.pythonhosted.org/packages/60/29/bf2496b4065e76e09fe48015aaffe5ce161d8f089b06ac6982070f653076/aiohttp-3.14.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:eeea07c4397bbc57719c4eed8f9c284874d4f175f9b6d57f7a1546b976d455ca", size = 1587293, upload-time = "2026-06-07T21:08:29.805Z" }, - { url = "https://files.pythonhosted.org/packages/49/a2/2136674d52123b1354bd05dd5753c318db47dc0c927cc70b27bab3755456/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:335c0cc3e3545ce98dcb9cfcb836f40c3411f43fa03dab757597d80c89af8a35", size = 1714756, upload-time = "2026-06-07T21:08:32.094Z" }, - { url = "https://files.pythonhosted.org/packages/a7/b9/e5fd2e6f915503081c0f9b1e8540947037929c70c191da2e4d54b31a21a1/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:ae6be797afdef264e8a84864a85b196ca06045586481b3df8a967322fd2fa844", size = 1721052, upload-time = "2026-06-07T21:08:34.167Z" }, - { url = "https://files.pythonhosted.org/packages/63/5a/2833e324a2263e104e31e2e91bc5bbee81bc499afd32203faee048a883f0/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:8560b4d712474335d08907db7973f71912d3a9a8f1dee992ec06b5d2fe359496", size = 1766888, upload-time = "2026-06-07T21:08:36.95Z" }, - { url = "https://files.pythonhosted.org/packages/57/fa/dea6511870913162f3b2e8c42a7614eb203a4540b8c2da43e0bfb0548f3c/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:2b7edd08e0a5deb1e8564a2fcd8f4561014a3f05252334671bbf55ddd47db0e5", size = 1581679, upload-time = "2026-06-07T21:08:39.292Z" }, - { url = "https://files.pythonhosted.org/packages/14/bd/3cf0d55e71784b33534e9710a67d382d900598b4787fbce6cc7317f8c42a/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:b6ff7fcee63287ae57b5df3e4f5957ce032122802509246dec1a5bcc55904c95", size = 1782021, upload-time = "2026-06-07T21:08:41.407Z" }, - { url = "https://files.pythonhosted.org/packages/c1/af/14bb5843eccbe234f4dfb78ab73e549d99727247e62ae5d62cbd22eaf5b0/aiohttp-3.14.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6ffbb2f4ec1ceaff7e07d43922954da26b223d188bf30658e561b98e23089444", size = 1742574, upload-time = "2026-06-07T21:08:43.795Z" }, - { url = "https://files.pythonhosted.org/packages/f2/1e/fbeb7af9210a67ac0f9c9bec0f8f4568497924e33137a3d5b48e1cf85f3f/aiohttp-3.14.1-cp314-cp314-win32.whl", hash = "sha256:a9875b46d910cff3ea2f5962f9d266b465459fe634e22556ab9bd6fc1192eea0", size = 457773, upload-time = "2026-06-07T21:08:46.168Z" }, - { url = "https://files.pythonhosted.org/packages/f0/2b/13e8d741a9ec5db7d900c060554cf8352ab85e44e2a4469ebb9d377bda17/aiohttp-3.14.1-cp314-cp314-win_amd64.whl", hash = "sha256:af8b4b81a960eeaf1234971ac3cd0ba5901f3cd42eae42a46b4d089a8b492719", size = 485001, upload-time = "2026-06-07T21:08:48.401Z" }, - { url = "https://files.pythonhosted.org/packages/df/30/491acfa2c4d6c3ff59c49a14fc1b50be3241e25bbb0c84c09e2da4d11395/aiohttp-3.14.1-cp314-cp314-win_arm64.whl", hash = "sha256:cf4491381b1b57425c315a56a439251b1bdac07b2275f19a8c44bc57744532ec", size = 453809, upload-time = "2026-06-07T21:08:50.7Z" }, - { url = "https://files.pythonhosted.org/packages/34/e3/19dbe1a1f4cc6230eb9e314de7fe68053b0992f9302b27d12141a0b5db53/aiohttp-3.14.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:819c054312f1af92947e6a55883d1b66feefab11531a7fc45e0fb9b63880b5c2", size = 793320, upload-time = "2026-06-07T21:08:52.775Z" }, - { url = "https://files.pythonhosted.org/packages/7f/20/1b7182219ba1b108430d6e4dc53d25ae02dcfcf5a045b33af4e8c5167527/aiohttp-3.14.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:10ee9c1753a8f706345b22496c79fbddb5be0599e0823f3738b1534058e25340", size = 529077, upload-time = "2026-06-07T21:08:55Z" }, - { url = "https://files.pythonhosted.org/packages/b9/c8/14ce60ec31a2e5f5274bb17d383a6f7a3aabca31ac04eee05585bbadab16/aiohttp-3.14.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:1601cc37baf5750ccacae618ec2daf020769581695550e3b654a911f859c563d", size = 532476, upload-time = "2026-06-07T21:08:57.176Z" }, - { url = "https://files.pythonhosted.org/packages/7e/02/9ac85e081e53da2e061b02fa7758fe0a12d17b8ce2d1f5e6c7cb76730328/aiohttp-3.14.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4d6e0ac9da31c9c04c84e1c0182ad8d6df35965a85cae29cd71d089621b3ae94", size = 1922347, upload-time = "2026-06-07T21:08:59.563Z" }, - { url = "https://files.pythonhosted.org/packages/c0/3e/d3ba07a0ab38b5389e10bec4362d21e10a4f667cba2d79ba30837b3a5059/aiohttp-3.14.1-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9e8f2d660c350b3d0e259c7a7e3d9b7fc8b41210cbcc3d4a7076ff0a5e5c2fdc", size = 1786465, upload-time = "2026-06-07T21:09:01.909Z" }, - { url = "https://files.pythonhosted.org/packages/0b/cb/e2ee978a00cfb2df829704a69528b18154eba5939f45bc1efa8f33aee4c5/aiohttp-3.14.1-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:4691802dda97be727f79d86818acaad7eb8e9252626a1d6b519fedbb92d5e251", size = 1909423, upload-time = "2026-06-07T21:09:04.357Z" }, - { url = "https://files.pythonhosted.org/packages/73/5d/1430334858b1022b58ae50399a918f0bd6fe8fa7fa183598d657ff61e040/aiohttp-3.14.1-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:c389c482a7e9b9dc3ee2701ac46c4125297a3818875b9c305ddb603c04828fd1", size = 2001906, upload-time = "2026-06-07T21:09:06.722Z" }, - { url = "https://files.pythonhosted.org/packages/66/4e/560c7472d3d198a23aa5c8b19a5115bf6a9b77b7d3e4bb363da320430ad2/aiohttp-3.14.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fc0cacab7ba4e56f0f81c82a98c09bed2f39c940107b03a34b168bdf7597edd3", size = 1877095, upload-time = "2026-06-07T21:09:09.011Z" }, - { url = "https://files.pythonhosted.org/packages/0d/f1/4745806578d447db4a784a8591e2dae3afdfc2bcb96f8f81271b13df6543/aiohttp-3.14.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:979ed4717f59b8bb12e3963378fa285d93d367e15bcd66c721311826d3c44a6c", size = 1676222, upload-time = "2026-06-07T21:09:11.461Z" }, - { url = "https://files.pythonhosted.org/packages/6a/c9/48255813cca749a229ef0ab476004ec623728ad79a9c0840616f6c076325/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:38e1e7daaea81df51c952e18483f323d878499a1e2bfe564790e0f9701d6f203", size = 1842922, upload-time = "2026-06-07T21:09:14.118Z" }, - { url = "https://files.pythonhosted.org/packages/3d/c0/bbd054e2bee909f529523a5af3891052606af5143c09f5f183ec3b234676/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:4132e72c608fe9fecb8f409113567605915b83e9bdd3ea56538d2f9cd35002f1", size = 1825035, upload-time = "2026-06-07T21:09:16.447Z" }, - { url = "https://files.pythonhosted.org/packages/a8/ae/90395d4376deceb74e09ec26b6adf7d2015a6f8802d6d84446af860fef04/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:eefd9cc9b6d4a2db5f00a26bc3e4f9acf71926a6ec557cd56c9c6f27c290b665", size = 1849512, upload-time = "2026-06-07T21:09:18.742Z" }, - { url = "https://files.pythonhosted.org/packages/93/bd/fb25f3049957553d4ce0ba6ae480aa2f592a6985497fca590837d16c1be0/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:b165790117eea512d7f3fb22f1f6dad3d55a7189571993eb015591c1401276d1", size = 1668571, upload-time = "2026-06-07T21:09:21.458Z" }, - { url = "https://files.pythonhosted.org/packages/3f/22/7f73303d64dd567ff3addca90b556690ed1233a47b8f55d242fb90af3681/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:ed09c7eb1c391271c2ed0314a51903e72a3acb653d5ccfc264cdf3ef11f8269d", size = 1881159, upload-time = "2026-06-07T21:09:23.813Z" }, - { url = "https://files.pythonhosted.org/packages/44/be/0474c5a8b5640e1e4aa1923430a91f4151be82e511373fe764189b89aef5/aiohttp-3.14.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:99abd37084b82f5830c635fddd0b4993b9742a66eb746dacf433c8590e8f9e3c", size = 1841409, upload-time = "2026-06-07T21:09:26.207Z" }, - { url = "https://files.pythonhosted.org/packages/7b/3c/bb4a7cba26956cb3da4553cc2056cf67be5b5ff6e6d8fa4fbdff73bfb7ae/aiohttp-3.14.1-cp314-cp314t-win32.whl", hash = "sha256:47ddf841cdecc810749921d25606dee45857d12d2ad5ddb7b5bd7eab12e4b365", size = 494166, upload-time = "2026-06-07T21:09:28.505Z" }, - { url = "https://files.pythonhosted.org/packages/8a/84/ec80c2c1f66a952555a9f86df6b33af65108a6febfa0471b69013a12f807/aiohttp-3.14.1-cp314-cp314t-win_amd64.whl", hash = "sha256:5e78b522b7a6e27e0b25d19b247b75039ac4c94f99823e3c9e53ae1603a9f7e9", size = 530255, upload-time = "2026-06-07T21:09:30.843Z" }, - { url = "https://files.pythonhosted.org/packages/2a/71/6e22be134a4061ada85a92951b842f2657f17d926b727f3f94c56ae963d6/aiohttp-3.14.1-cp314-cp314t-win_arm64.whl", hash = "sha256:90d53f1609c29ccc2193945ef732428382a28f78d0456ae4d3daf0d48b74f0f6", size = 469640, upload-time = "2026-06-07T21:09:33.028Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/58/d9/22ce5786ac0c1653ae8b6c23bded02c1686d11f0dbb45b31ce128e0df985/aiohttp-3.14.3.tar.gz", hash = "sha256:9491196535a88924a60afd5b5f434b5b203b6cc616250878dbdb223a8f7844bc", size = 7971213, upload-time = "2026-07-23T01:57:27.037Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/57/be/5afd201cc0ab139029aadb75392efe85a293403d9dd3a3226161c21ce00c/aiohttp-3.14.3-cp313-cp313-android_21_arm64_v8a.whl", hash = "sha256:2e9878ae68e4a5f1c0abe4dd497dbc3d51946f5837b56759e2a02e78fa90ef86", size = 506269, upload-time = "2026-07-23T01:54:49.075Z" }, + { url = "https://files.pythonhosted.org/packages/22/09/dec8189d62b45ade009f6792a2264b942a90cb88aeaf181239933cd72c3c/aiohttp-3.14.3-cp313-cp313-android_21_x86_64.whl", hash = "sha256:f3d2669fe7dec7fc359ecdb5984b29b50d85d5d00f8c1cb61de4f4a24ee42627", size = 515166, upload-time = "2026-07-23T01:54:51.894Z" }, + { url = "https://files.pythonhosted.org/packages/28/24/2854869d29ed8a8b19d74f9ec6629515f7e04d02dd329d9d179201e58e47/aiohttp-3.14.3-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:cc7cb243a68167172f48c1fd43cee91ec4b1d40cefd190edd43369d1a6bc9c82", size = 486263, upload-time = "2026-07-23T01:54:54.223Z" }, + { url = "https://files.pythonhosted.org/packages/d4/dd/57187c8be2a35aea65eaee3bd2c3dcbbcf0204f5106c89637e3610380cd1/aiohttp-3.14.3-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:78253b573e6ffab5028924fc98bc281aae05445969982a10864bc360dea2016c", size = 492299, upload-time = "2026-07-23T01:54:56.236Z" }, + { url = "https://files.pythonhosted.org/packages/b9/11/06ae6ed8f0d414edf4068861e233d8fe23ee699bfd4b3ceb8663db948a62/aiohttp-3.14.3-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:7041d52c3a7fa20c9e8c182b534704abb19502c8bdcbde7ab23bfda6f642394f", size = 502235, upload-time = "2026-07-23T01:54:58.377Z" }, + { url = "https://files.pythonhosted.org/packages/7e/a3/559639c34a345d2cf7c52dff6838119f2eaf29eb508227b5b83f573af813/aiohttp-3.14.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:ac74facc01463f138b0da5580329cfcc82818dea5656e83ddcd11268fc12ff80", size = 750883, upload-time = "2026-07-23T01:55:00.65Z" }, + { url = "https://files.pythonhosted.org/packages/91/cd/41e131f13afd1e7b0172a9d9eda085ef90eb8439f41f0d279db81ed3ae60/aiohttp-3.14.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d6218d92e450824e9b4881f44e8c09f1853b490f9a64130801024a4793b1b3b0", size = 508473, upload-time = "2026-07-23T01:55:02.945Z" }, + { url = "https://files.pythonhosted.org/packages/bc/6b/e7f13410d391c6e55b4c007a8de024355389d7d459e3d64c42b2d33617e5/aiohttp-3.14.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:11fb37ef075669eee52ab1928fbf6e1741fada40409fa309ebde9607a962aebf", size = 509190, upload-time = "2026-07-23T01:55:05.173Z" }, + { url = "https://files.pythonhosted.org/packages/97/21/6464573e53d69672cc1eada3e5c5cb2d2efa82701e8305a0f2047a576967/aiohttp-3.14.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:55bdcc472aafe2de4a253045cc128007a64f1e0264fb675791e132ea5edaa3bd", size = 1761478, upload-time = "2026-07-23T01:55:07.383Z" }, + { url = "https://files.pythonhosted.org/packages/1a/81/d217043a4c17fbce360905e3b2bdd20139ebc9a2de836d035d179c4da006/aiohttp-3.14.3-cp313-cp313-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:c39846c3aad97a8530c89d7a3869a8f8e9e3762c6ac0504481e5c80948f7e807", size = 1735092, upload-time = "2026-07-23T01:55:09.803Z" }, + { url = "https://files.pythonhosted.org/packages/a1/66/e13a02d0eeb1a9a502402a977abb4e4abff9fe4051c26f80558c57a7c975/aiohttp-3.14.3-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:5895ef58c4620afe02fa16044f023dc4dafec08158f9d08874a46a7dbc0341b8", size = 1800546, upload-time = "2026-07-23T01:55:12.012Z" }, + { url = "https://files.pythonhosted.org/packages/26/5e/57d42fca1d18cb5acc1cad945d017fabc5d6ae71d8a08ad66be8dc3ee544/aiohttp-3.14.3-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:fa9467a8113aa69d3d7c55a70ef0b7c636010a40993f3df9d9d0d73b3eb7ef24", size = 1895250, upload-time = "2026-07-23T01:55:14.357Z" }, + { url = "https://files.pythonhosted.org/packages/ca/1c/7da8d08e74d56f00070822f9638ff3f1c563f8ad87d1efa996c87bfc8644/aiohttp-3.14.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d7d2deec16eeedf55f2c7cf75b521ea3856a5177e123844f8fd0f114ce252cb5", size = 1789289, upload-time = "2026-07-23T01:55:16.668Z" }, + { url = "https://files.pythonhosted.org/packages/cd/0f/cf16bcf56896981c1a0319f5d5db9337994b5165730c48a8fa07e9b34be6/aiohttp-3.14.3-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:dd54d0e8717de95939766febac482ac0474d8ac3b048115f9f2b1d23a16e7db4", size = 1586706, upload-time = "2026-07-23T01:55:18.913Z" }, + { url = "https://files.pythonhosted.org/packages/fe/6f/76eac12a7f2480e1e304f842efdb07db33256b0d9165b866b6ef0806c202/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:df82f3787c940c94986b34222d59c9e38843fba85139f36e85255a82ad5355a9", size = 1724652, upload-time = "2026-07-23T01:55:21.296Z" }, + { url = "https://files.pythonhosted.org/packages/39/b6/19c8c592baeeb94b75f966547d40c02ac7590902306ec5863d5c027cf506/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:42a67efc36300d052fb4508a53e8b6901b9284b599ae63945c377569c5fcc1e1", size = 1756239, upload-time = "2026-07-23T01:55:23.705Z" }, + { url = "https://files.pythonhosted.org/packages/dc/c9/4e9383150296f97f873b680c4de8fb2cd88608fb9f48c79edcb111611abc/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:7a75aa63cbf9b21cfaf60dc2657e19df2c2867d91707d653fee171ffeedd1371", size = 1769161, upload-time = "2026-07-23T01:55:26.082Z" }, + { url = "https://files.pythonhosted.org/packages/aa/1e/147bdc6cc5de5f3ab011be8bf5d6e786633249f22c20bae06f85e45f5387/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:e92eb8acc45eb6a9f4935071a77edf5b85cc6f8dfad5cd99e97653c26593cdde", size = 1578759, upload-time = "2026-07-23T01:55:28.846Z" }, + { url = "https://files.pythonhosted.org/packages/fd/31/78388a9d6040ece2e11df62ea229a822cf5e52d238374b220ae9975b2623/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:b014a6ed7cf912e787149fdc529166d3ceabac23f26efeea3158c9aba2354e7e", size = 1792025, upload-time = "2026-07-23T01:55:31.457Z" }, + { url = "https://files.pythonhosted.org/packages/03/51/a3d29fdf2c25d796746af8ad6fe56a45d6256c38b0a8a2ed752e1160b3a2/aiohttp-3.14.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3d4f72af88ac2474bb5bca640030320e3d38a0163a1d7533500e87be458eef71", size = 1768477, upload-time = "2026-07-23T01:55:33.87Z" }, + { url = "https://files.pythonhosted.org/packages/29/a6/442e18b5afeade534d877a2dc3c3e392aff8d49787890b0cf84790410267/aiohttp-3.14.3-cp313-cp313-win32.whl", hash = "sha256:5f08ec777f35ee70720233b8b9811d3bb5d728137f30ac91b7457709c3261ac0", size = 451069, upload-time = "2026-07-23T01:55:36.121Z" }, + { url = "https://files.pythonhosted.org/packages/9d/69/3d876ac02659f271cf7f6769f14a8e3de5b6e888ed8b5a7e998086a4cec8/aiohttp-3.14.3-cp313-cp313-win_amd64.whl", hash = "sha256:dff9461ec275f22135650d5ba4b4931a11f3958df7dfbb8db630000d4dee0883", size = 476518, upload-time = "2026-07-23T01:55:38.303Z" }, + { url = "https://files.pythonhosted.org/packages/b2/0e/50d6e6471cd31edce8b282bdec59375a3a69124d8a989a0b1313355cae52/aiohttp-3.14.3-cp313-cp313-win_arm64.whl", hash = "sha256:ddcac3c6b382e81f1dd0499199d4136b877beb4cb5ef770bbbfba56c4b8f55d2", size = 447676, upload-time = "2026-07-23T01:55:40.451Z" }, + { url = "https://files.pythonhosted.org/packages/c8/20/887fdcf832326571b370ffc347b3e70abe101096f3720126aac161b1d872/aiohttp-3.14.3-cp314-cp314-android_24_arm64_v8a.whl", hash = "sha256:49f7325beb0f85ef4aef5f48f490269575f83e6e2acad00a1d80b807eb027062", size = 509067, upload-time = "2026-07-23T01:55:42.618Z" }, + { url = "https://files.pythonhosted.org/packages/ad/a3/92cec936f78cc4bf0fa5554ebe593b73459d94e3c62303e1902a4cccb6f7/aiohttp-3.14.3-cp314-cp314-android_24_x86_64.whl", hash = "sha256:e3be98a7c30b8c25d573dafba7171d66dfb05ee6a9070fc46535464ff97700a6", size = 514774, upload-time = "2026-07-23T01:55:44.937Z" }, + { url = "https://files.pythonhosted.org/packages/29/ba/2a0c38df3fc557620b6a5acd98364af050053b6285b4dc7ee74100c63c18/aiohttp-3.14.3-cp314-cp314-ios_13_0_arm64_iphoneos.whl", hash = "sha256:614c61d478b83953e261d02bb2df750f17227cd33ef8002945bf5aebbde21919", size = 488134, upload-time = "2026-07-23T01:55:47.135Z" }, + { url = "https://files.pythonhosted.org/packages/48/d6/d51b7d4bf309af3693940d8ffd2b9ed0b682434ef85959b7c9c137f60cf8/aiohttp-3.14.3-cp314-cp314-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:1caa7b0d05f3e3a36f87788c59e970a7ee1cefcfcbb924a9f138c4a6551c9cb7", size = 494201, upload-time = "2026-07-23T01:55:49.451Z" }, + { url = "https://files.pythonhosted.org/packages/3f/5a/8f624384e5f1efabb5229b94157eb966b021e97bdb188c62860c2ae243c2/aiohttp-3.14.3-cp314-cp314-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:dfa68deb2a443bdaa3ea5297b0699c1464f08aef3812b486d1348eee61b07dc0", size = 502766, upload-time = "2026-07-23T01:55:51.656Z" }, + { url = "https://files.pythonhosted.org/packages/a6/26/4ff0164370deec18fb19254ee4ab10b7a73304ac0c860b13f5f84663759b/aiohttp-3.14.3-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:e72ee89e28d907a18f46959b4eb0bb06701cc7f8cf4366e00029e2ccfaaf5924", size = 756557, upload-time = "2026-07-23T01:55:53.964Z" }, + { url = "https://files.pythonhosted.org/packages/97/a3/7056b86dc0d9ec709ea9777eae3b0161428f943372f8b98c01c11593b682/aiohttp-3.14.3-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:ad4c8b7488d745d2ca4838ebd8ae5ba9b56341d30b1da43640e4ce87f9f49646", size = 510168, upload-time = "2026-07-23T01:55:56.22Z" }, + { url = "https://files.pythonhosted.org/packages/85/ed/0357a015892fd68058bf2d39d3fd1958e459b997a7db30aaa6aaa434ae96/aiohttp-3.14.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:db332af25642007330fca8be5c4d194caf2bea7a7fc84415aff3497af5dfee6b", size = 512957, upload-time = "2026-07-23T01:55:58.437Z" }, + { url = "https://files.pythonhosted.org/packages/47/d1/8aba53f15ccb2238405f5e9d30e2a8ca44f93878c26e7165ade00d374b1c/aiohttp-3.14.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:25bd2708db6bdf6a6630dd37bdcdfcb47c4434d22ac69c64665b802910140b30", size = 1750149, upload-time = "2026-07-23T01:56:00.856Z" }, + { url = "https://files.pythonhosted.org/packages/49/bd/40c3fee327529284375c6701cbb0fa4600cc2e8432af1378f897e2ef7d3a/aiohttp-3.14.3-cp314-cp314-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:cef89a58e628c4efcac3275c2d68083f82426dcdc89c1492a6f654f9f7ea6ab9", size = 1707685, upload-time = "2026-07-23T01:56:03.371Z" }, + { url = "https://files.pythonhosted.org/packages/2a/a3/ca0cc6724cca8114b05694abd916060758c79894c3aa5b012cdadc1bc28e/aiohttp-3.14.3-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c23ec8ee9d5ab2f5421f9c7fffce208435607af27fd46d4a44e031954352838f", size = 1803911, upload-time = "2026-07-23T01:56:05.817Z" }, + { url = "https://files.pythonhosted.org/packages/95/b5/85b099c299c3ffd38ad9b3e43694c8a346934e4a30c88c4fd5a841234f77/aiohttp-3.14.3-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:e2667f0bbe7eb6c74eae5e9691441ad186e5845ca3cff63230fc09c4e7514f5d", size = 1876929, upload-time = "2026-07-23T01:56:08.413Z" }, + { url = "https://files.pythonhosted.org/packages/d5/b7/1da684a04175473fa4cddbf9a2f572e79514c3fd27a74597f43057d4f3da/aiohttp-3.14.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18cb43369747b2ae007bd2655fb8e63a099c2ff1d207962943636dac989b3147", size = 1761112, upload-time = "2026-07-23T01:56:10.918Z" }, + { url = "https://files.pythonhosted.org/packages/d1/16/bc4b55e3e5cb175fd69c53c90d60d2f47797cb343da5106e23863dc4dba4/aiohttp-3.14.3-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d77640cc618c1d99fc4f8589c0f24a730adfa54eb1e57ef7bf0c8dfb78da898c", size = 1583500, upload-time = "2026-07-23T01:56:13.613Z" }, + { url = "https://files.pythonhosted.org/packages/2a/e8/13a9d957a1ee40837f46aa30f0f4c657e673ad86a2e6362a9f9be20d26d9/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:53e5179d8abb5710f8e83ba207c41c8d1261fcffd4616500e15ca2b7a33be10a", size = 1713940, upload-time = "2026-07-23T01:56:15.969Z" }, + { url = "https://files.pythonhosted.org/packages/38/05/d33c680c1bcf1c7e130f9cbfc1fc02fe8bb0c4af2a94a53dd5fb56131e5c/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:cd817772b2fcf2b8c0905795318485f9ec16eae60b29feb7f4c77085311637f0", size = 1724413, upload-time = "2026-07-23T01:56:18.591Z" }, + { url = "https://files.pythonhosted.org/packages/85/1d/af798d306f7a74b6a632dbcabcf62a4c91391b7582d2a8c6d7712e2cc54e/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:4e3ac92d90e92773b2362d506068e9a948192bd553e743c5b2429e28527c8661", size = 1770748, upload-time = "2026-07-23T01:56:21.074Z" }, + { url = "https://files.pythonhosted.org/packages/a8/92/ad720d472556a995049206867765e9410969684f86ee09423ff9969044c1/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:3f42e9b78301f11c8f861746175d8b9c1ccef713fcad9eab396e2f6db8ed4a22", size = 1577564, upload-time = "2026-07-23T01:56:23.475Z" }, + { url = "https://files.pythonhosted.org/packages/60/ad/0ed7586cbef7a884e23a752fa2bb987a122e6a5dd50dab109258d0a95193/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9d9edccfe496b476db5f398d97b865e9a6752bcf8aec4eef8390ce20fb64bb41", size = 1782080, upload-time = "2026-07-23T01:56:25.994Z" }, + { url = "https://files.pythonhosted.org/packages/97/ea/dbaed0d73e8a69aad653b045dab451c67c2454bb731a37b45a86593e9422/aiohttp-3.14.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:1c5ec8fb1bcc31a8466f74aaf26c345d5c386fa4bd08a3f0eb9c7a4a3fe8b5bf", size = 1745813, upload-time = "2026-07-23T01:56:28.604Z" }, + { url = "https://files.pythonhosted.org/packages/81/1b/6893d4bc57e434fc93a6c9217c637d967a0b651d989f6e3265179375754a/aiohttp-3.14.3-cp314-cp314-win32.whl", hash = "sha256:38901a84da3ce22249f6e860bf8f90d141bcab7da090cc398f8bb58c0e44b7da", size = 455872, upload-time = "2026-07-23T01:56:31.031Z" }, + { url = "https://files.pythonhosted.org/packages/f5/8b/c7baa1ba1eda4db6989baefe5de6d99834921b84ebd7918624febcb9f290/aiohttp-3.14.3-cp314-cp314-win_amd64.whl", hash = "sha256:8b3b60de05f3dcb6f6a00f818bb2ec781cee4de0645f59ccaf99b1d1823b6100", size = 481030, upload-time = "2026-07-23T01:56:33.365Z" }, + { url = "https://files.pythonhosted.org/packages/22/8c/c29d067df825a2df88ca432db848aa2fe8199598359cc06c12b09320cac9/aiohttp-3.14.3-cp314-cp314-win_arm64.whl", hash = "sha256:1576145bdceeb92382d899751e12743a3a5b8e460a841e3e50543859e54864dc", size = 453669, upload-time = "2026-07-23T01:56:35.731Z" }, + { url = "https://files.pythonhosted.org/packages/6a/a4/9c033beb355d39b6147980597ec9645e4729243f686ee4dc73945de72030/aiohttp-3.14.3-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:8800c996b01c2772a783e3e46f3e1abd5823029adca0df54231960de9bfefa5b", size = 791403, upload-time = "2026-07-23T01:56:37.972Z" }, + { url = "https://files.pythonhosted.org/packages/80/ca/87c32a0a7704583cfc49660bd817889bae5b830bf53b5dcb4e92145ac2da/aiohttp-3.14.3-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:ebe8e504f058fe91223351cecd2d9d6946c9d241bb0250d898ffbdf584cc72b0", size = 526413, upload-time = "2026-07-23T01:56:40.523Z" }, + { url = "https://files.pythonhosted.org/packages/9e/d8/8ec0e471248c500acdce2be3f46db8fb62b5eb60efef072529cc85ee1d26/aiohttp-3.14.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:30402d03a7c0ff52bce290b57e564e9079fd9d0cb545c8aba73f86a103162d2e", size = 532135, upload-time = "2026-07-23T01:56:42.876Z" }, + { url = "https://files.pythonhosted.org/packages/fe/45/f8919fd936e8b79fcd9bda7b6d8e62613462a713f4f17987fd7c34399142/aiohttp-3.14.3-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9fc7b5bfec6573f3ae844f457fdde5adeb713f8b8e4a81ad64fc207b49383716", size = 1922742, upload-time = "2026-07-23T01:56:45.528Z" }, + { url = "https://files.pythonhosted.org/packages/f6/ec/9ca76b28a27525b0cc53e20842e0228b022f301ce1f436b7d814b4aaf2df/aiohttp-3.14.3-cp314-cp314t-manylinux2014_armv7l.manylinux_2_17_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:8a5fd34f7f7410d1730d5c2ba873cacb2eed3fede366feb268a70ba22581ed8f", size = 1787371, upload-time = "2026-07-23T01:56:48.045Z" }, + { url = "https://files.pythonhosted.org/packages/b1/04/6acdbf17315f7b55f1937e3387acb89a3cddeb4995689553d064af8e92ab/aiohttp-3.14.3-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:270d3dace9ca2f10f0da5d8ebe519b7a310fc6112ed916e32df5866df0888553", size = 1912623, upload-time = "2026-07-23T01:56:50.605Z" }, + { url = "https://files.pythonhosted.org/packages/86/e6/438b0c79ca6f45eb9fd9817dd4c01a91919a38c0de5ee9e05e2b4dc0ece7/aiohttp-3.14.3-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:3ae5b3a59436d089b5395d910121a390feed4d00578eb95a0fd1a329fe963100", size = 2005515, upload-time = "2026-07-23T01:56:53.153Z" }, + { url = "https://files.pythonhosted.org/packages/bb/6b/62cbd6577758699525f5c712d1ddef57d9875fbab0ae8d5f5a202fd598f8/aiohttp-3.14.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2498f0fe69ead802f9675beca44a7c21c62fdaa4ec5145ea1c3ad6edbee29f85", size = 1879906, upload-time = "2026-07-23T01:56:55.818Z" }, + { url = "https://files.pythonhosted.org/packages/00/95/18bcbf830a21dc3aae24d8f6b6feaf3db1d2090242d00a7868db2ffb0b67/aiohttp-3.14.3-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a0dc483c00da8b673abbb367eb6f8d8f4bcec30eb58529ea13cb42e7fd2dfa33", size = 1675849, upload-time = "2026-07-23T01:56:58.861Z" }, + { url = "https://files.pythonhosted.org/packages/a9/19/47f4968659c5e23606c3790c80fc624e691c153d036148449ee84d31b287/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:c7d3a97c678d34fc5b59da671ee9cd630096ddc643e7b5a30d54a2a6f3574d3f", size = 1843496, upload-time = "2026-07-23T01:57:01.591Z" }, + { url = "https://files.pythonhosted.org/packages/64/af/38c33c4dd82fddcb4e56c4653b6f1072a8edbc6b7fa15809f14932c41e2d/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:f8fb78a83c9e5f741ca3a68cfb455c1f5bb83b4e7249a3848b3cd78d0a8563b0", size = 1827746, upload-time = "2026-07-23T01:57:05.131Z" }, + { url = "https://files.pythonhosted.org/packages/a1/9d/0537cda4885ac8f5b7053d164dd06312f4c483a4edcb8ee5b8aaf2a989bf/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:74ab5b6a9fb13e873e5a90946588baecaf488745e1db1a4a5c433f971f035098", size = 1853810, upload-time = "2026-07-23T01:57:08.043Z" }, + { url = "https://files.pythonhosted.org/packages/19/fe/26f9c5e6458385aa86497836b0dea6fb2f027827d63f37c7856cce9286ee/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:bd52f811e65f6fb634b1047159657c98f52b407f8efec907bcfc09da9a4c0a25", size = 1668895, upload-time = "2026-07-23T01:57:10.837Z" }, + { url = "https://files.pythonhosted.org/packages/ec/4c/618b1db9b9ba079b8875d2cdf78e7c4a3bf72903bd5850fee7dd9544600a/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_s390x.whl", hash = "sha256:f0f177d1b195b9e06376cfd7d308d8a1b920909a609d03ac82a8c73bbb16d3b9", size = 1883833, upload-time = "2026-07-23T01:57:13.672Z" }, + { url = "https://files.pythonhosted.org/packages/94/c6/bd959bd1e4771f9fd944e9e436224c48c77b018b73b519b5aad346335bcc/aiohttp-3.14.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:498c6c623134f8e09a3c4e60bcd607a0b4590dd7dbf08dd40851b27cbb520ccb", size = 1844251, upload-time = "2026-07-23T01:57:16.593Z" }, + { url = "https://files.pythonhosted.org/packages/5e/19/08d41839658bdd44a0ed2480f3891705ecb487ce28c0dde62c9040c997e0/aiohttp-3.14.3-cp314-cp314t-win32.whl", hash = "sha256:b304db572b4368edd8dda8a2274f73156fe15558fca4a917cb8a09fc47af5963", size = 474180, upload-time = "2026-07-23T01:57:19.306Z" }, + { url = "https://files.pythonhosted.org/packages/99/5d/3cd6ef0a2b2851f7ab913b5b079334781bd50ff56a323e4454063377a080/aiohttp-3.14.3-cp314-cp314t-win_amd64.whl", hash = "sha256:b20032766aedf6261c7a566585a40867d092ac03a0d81592d5370ef9b054f99b", size = 500528, upload-time = "2026-07-23T01:57:21.762Z" }, + { url = "https://files.pythonhosted.org/packages/a4/37/cfd1ed540a4d318da025590d96b728e63713c09e9377950fc655dadeb856/aiohttp-3.14.3-cp314-cp314t-win_arm64.whl", hash = "sha256:2e1161602f45a54de2ce0905243a95f58cb42dcd378402f3697f5e0b21e9d2e7", size = 469280, upload-time = "2026-07-23T01:57:24.241Z" }, ] [[package]] @@ -613,52 +613,52 @@ wheels = [ [[package]] name = "cryptography" -version = "49.0.0" +version = "50.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cffi", marker = "platform_python_implementation != 'PyPy'" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/1f/99/d1c90d6041656cc6ee229dc99cd67fd0cd5aec3c5f7d72fffc27cc750054/cryptography-49.0.0.tar.gz", hash = "sha256:f89660a348f4f78a92366240a61404e337586ef7f5909a2fef59ca88ef505493", size = 854345, upload-time = "2026-06-12T20:02:30.512Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/9b/22/adf66990e63584a68dfb50c24f48a125c07b1699899381c8151e63ed458c/cryptography-49.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:966fe0e9c67490071f14c0d2b1cb2dfb3023c5ce39457343931415f08382f2db", size = 4032100, upload-time = "2026-06-12T20:02:32.143Z" }, - { url = "https://files.pythonhosted.org/packages/09/41/3797cfaf69cae04a13ee78ebd83f0678d9c02b4779d21ce24445326f1a69/cryptography-49.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:36d1709f992593689b45bda411498d62c6e365f2ca00b84657d4dadd24de16db", size = 4692978, upload-time = "2026-06-12T20:01:21.305Z" }, - { url = "https://files.pythonhosted.org/packages/e6/8b/43011f7ebe515a8aa20d61f290a326cd890c2e738e16e59eaff8d9c3a412/cryptography-49.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0e959b578856a3924bc0cbb710fc12c387b9412a951389f3ca61704a9e25f325", size = 4716422, upload-time = "2026-06-12T20:01:48.566Z" }, - { url = "https://files.pythonhosted.org/packages/4a/91/01ce7303a4579e6d3a6abef01bd322848e9ea7a219adcabc5048b9033571/cryptography-49.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:53ecee2e23f7169b6117e99fc8a944e5e50f79e69758a83b52a00cb98ab2b2d2", size = 4700503, upload-time = "2026-06-12T20:02:47.091Z" }, - { url = "https://files.pythonhosted.org/packages/62/99/a2c95cf8293f07491e9e27c20cc4dcd18176d944e674679adeb1d0173fd6/cryptography-49.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:2eda353d8a27bcbcaa4cbed18994a74ab4d19a2ca897db188ea269ab9b71419b", size = 5309779, upload-time = "2026-06-12T20:02:08.987Z" }, - { url = "https://files.pythonhosted.org/packages/20/2c/0622f20ff02b2ef32558733443805dc82fd4c275be01b2d19d14676f3a1b/cryptography-49.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:2afe9051da7ae7bd5905da5a949280c7d2bb75682e188f650a9d0f2756b834c6", size = 4749683, upload-time = "2026-06-12T20:02:03.335Z" }, - { url = "https://files.pythonhosted.org/packages/a3/5b/c5246635d5fd3b64e0d45ae10e99fd32fe9676a79915ccfe5a61ba9af1a5/cryptography-49.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:0b82e28ee398a386f0807bba7884d30f25218855690f45115831bcce5d90822c", size = 4337874, upload-time = "2026-06-12T20:02:54.323Z" }, - { url = "https://files.pythonhosted.org/packages/6d/88/05563c7fe2e914e87d1a536d06fe83e66b4e1d95cb593e05aea375531da8/cryptography-49.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:ccac2bfebc306b862133e3bb71f3f6ee8bb525240089b2d952e4144b3a6d5da7", size = 4700283, upload-time = "2026-06-12T20:01:34.822Z" }, - { url = "https://files.pythonhosted.org/packages/c4/b6/d7696e4e890d6ae1469935164c9e5215c557671cb78d6e3f458ccceaa632/cryptography-49.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:d0527ce944105f257f605a827d6ebead966c752038b6e8656abb9c5edee6fc68", size = 5265844, upload-time = "2026-06-12T20:01:24.09Z" }, - { url = "https://files.pythonhosted.org/packages/a9/3c/f3ad17eecc1a57b0ba236dc01f90e783c51f4a2f35f64777cc4f47a184b2/cryptography-49.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:cbc77da8c523d5abd028635ba850a6966fcee2c82e2bf65a41d1d8afe0f98be9", size = 4749290, upload-time = "2026-06-12T20:01:30.848Z" }, - { url = "https://files.pythonhosted.org/packages/4f/01/339573cf1023163a400b0b5d16f6d507de413b9f60be6fd1b77feeaf6737/cryptography-49.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:b87e65d263b3e5d3bb92a57e2a6638e2f31110fa7aa890c7b2dbba42248d0a3f", size = 4834612, upload-time = "2026-06-12T20:01:29.246Z" }, - { url = "https://files.pythonhosted.org/packages/71/fd/577302e213a1be9468f92d1afef66fcf1ef83d516819d9992ca547f592bd/cryptography-49.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:66ec79c3904820572d7e987abdf304281f141d37ad9a489b8e97066e7b9b6459", size = 4980804, upload-time = "2026-06-12T20:01:42.853Z" }, - { url = "https://files.pythonhosted.org/packages/1f/09/f42b1d190c5ba75f72062a387f8030d1d75f6ab035788f1d9c4b01de6525/cryptography-49.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:e5dfc1e64de5677cec922ffa8da89c546d0415bf6efdf081842e5d44c84e1f0e", size = 3810026, upload-time = "2026-06-12T20:02:39.262Z" }, - { url = "https://files.pythonhosted.org/packages/ec/9e/db72b3ae7fc9cfad53e630e56c6ae83b9b6ff0bf3718ffb8012d20b3aabf/cryptography-49.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:73a205dce83953d131a4aa1e0fd917a2fd1c5b1eef251e9d7152efefcbf5caf7", size = 4013892, upload-time = "2026-06-12T20:02:10.735Z" }, - { url = "https://files.pythonhosted.org/packages/86/12/c48a424f38db03027be9f7ed5c7dc5de9933dbee992865f98b13727a009d/cryptography-49.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:196ecd6a36e4e9aa10270393bb98d8df88fccee0bf1e5128b91ae4eb4375896d", size = 4678835, upload-time = "2026-06-12T20:02:48.743Z" }, - { url = "https://files.pythonhosted.org/packages/68/28/8a3ad4653662c93fc44dc4e5d8fd374c25c42e07b34bbfbadf49cf57a5a8/cryptography-49.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7abcee80084cda3f7691f3eb1ce480d8df49cec637b429aa35986c1de71738aa", size = 4697239, upload-time = "2026-06-12T20:02:56.03Z" }, - { url = "https://files.pythonhosted.org/packages/a8/b2/2193fc74f81aee4f9b62733133b73b5176718932ed8f2e4b03fa040480a6/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:4ae387c9cb68ea569ca17e490d66d8142b81c3cc814bf179974b7d146e490bbb", size = 4685593, upload-time = "2026-06-12T20:02:50.666Z" }, - { url = "https://files.pythonhosted.org/packages/47/f1/1d3eaa243bfc5de4a187b22aa8c048b3e4980bfbe830ac46e6bac2e66947/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:f37d847238971164fdbc68ade6f6574aecc9c0af714190e2083429ff68f4ce9d", size = 5289961, upload-time = "2026-06-12T20:01:46.468Z" }, - { url = "https://files.pythonhosted.org/packages/58/39/2d51306721330c486495853eda1c567880ff036de15a14c4b74f399934af/cryptography-49.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:c2bc30226390d60ea19d9f82b19db005fe0452154a23c1c410c12ea801e43561", size = 4731145, upload-time = "2026-06-12T20:02:16.832Z" }, - { url = "https://files.pythonhosted.org/packages/17/50/983e838c7fd0d87fd8c969bcdd328edaf5f756e38df5281637424c155873/cryptography-49.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:07cab27cc7b7e0fd28e5e26bb9eeedde5c135c868b46de4a27845abe94af6122", size = 4321719, upload-time = "2026-06-12T20:02:52.611Z" }, - { url = "https://files.pythonhosted.org/packages/a7/f5/8f571d7e27c55bce9f76f026143bcb1e040a4233149ecca0bea5fa5dd5f7/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:b20133d204d2bb56ba047642199603876c872026ca53e79c35b83772ab2cc505", size = 4685209, upload-time = "2026-06-12T20:02:07.282Z" }, - { url = "https://files.pythonhosted.org/packages/e7/84/0e27016a6fc5a0886f797018b26aa42f40c09a82332bff77822a451deaaa/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:b970c6da94d5bb18629db453d14f2a1300f6bf59b61e9b82377931ef95504866", size = 5246285, upload-time = "2026-06-12T20:01:32.439Z" }, - { url = "https://files.pythonhosted.org/packages/11/2d/5e1fb307cb5931881516b464c98774b3f2c36b5d4bb9a2830253cf553cad/cryptography-49.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:d8ecde755e2e91bf773fc94e8c9d730cd7f2007004cb492263a794ec3899a1c8", size = 4730441, upload-time = "2026-06-12T20:02:01.469Z" }, - { url = "https://files.pythonhosted.org/packages/e4/c0/bff5a02ee731d207d6a1ed51732549d8c53d2bc8da1d10ec6f2844201d68/cryptography-49.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e3fb64c420688e5319ae25113a354015abbd8dffbfbc41781a1ea66fc7622ac3", size = 4815869, upload-time = "2026-06-12T20:01:36.574Z" }, - { url = "https://files.pythonhosted.org/packages/b9/26/814681d14248d95d73d5c3eea0c39a94eb8302df966f670a2c60de90974b/cryptography-49.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:32703d93296f5c1f4b53349ad3a250c2cae0fdecd3a3dd5d47e616d8d616af27", size = 4960948, upload-time = "2026-06-12T20:02:18.688Z" }, - { url = "https://files.pythonhosted.org/packages/4c/fe/93ecac273d3738939d023612ad12cca9a3740a5345d69fda04134c43fd96/cryptography-49.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:33cd0565932807baddb67b96dbee92f2c374b5c89dee09fd74079aeb8c8dba61", size = 3799153, upload-time = "2026-06-12T20:01:39.059Z" }, - { url = "https://files.pythonhosted.org/packages/19/2a/5bb823f5bedcf80718cea7fbc95ec5515cca3769633c4b01a32be7f30e7c/cryptography-49.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ec5e529fb80935c94fe7b729f9972b50e351a0e6b50aa294fd5cabb109fcc29a", size = 4025947, upload-time = "2026-06-12T20:01:25.745Z" }, - { url = "https://files.pythonhosted.org/packages/3d/df/40577043ca124e17012f408ddddaeb213b856336ac82ddb3bc915f39e29f/cryptography-49.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f78ff2c9ed8dc2d036b0f4d640e22522213d047c1b14e61205a7e55c80a494d4", size = 4692429, upload-time = "2026-06-12T20:01:53.628Z" }, - { url = "https://files.pythonhosted.org/packages/2c/99/2d13299eb3dd27b02dcfaafcc91d6b5cb3329f7cbd6d8f51921acd566c1a/cryptography-49.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:35b151772baff2c74cba7fa290ceaff4c3b11c0c881eb93eb5dbc05a7cfbba18", size = 4700968, upload-time = "2026-06-12T20:02:45.383Z" }, - { url = "https://files.pythonhosted.org/packages/a5/4d/9c0cd02f95e2602dd5e563da149ee0830abef3537be8b34dc56281ebe27a/cryptography-49.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:0f21641cf4b30fca7aee061ced0ec7ad7b073518088b7c9969a297c0ae796c69", size = 4697758, upload-time = "2026-06-12T20:01:41.13Z" }, - { url = "https://files.pythonhosted.org/packages/24/01/186c825898477d77e2324d5360fefe622ff1d8d1963ec0554e2cada8ec77/cryptography-49.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:9e82dcc8e56052715fb18b2429e3bca4823b1629136a2084fc45a9a5cecb9b64", size = 5298863, upload-time = "2026-06-12T20:02:24.579Z" }, - { url = "https://files.pythonhosted.org/packages/b8/7b/62cbbab75d0659865bf0273790031544a0b16c8072d258f9428dcd8190dc/cryptography-49.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:6f2debedf9ca60cf1d5bd466475638af5130f89965605cd818484d19987d3a21", size = 4735983, upload-time = "2026-06-12T20:01:50.14Z" }, - { url = "https://files.pythonhosted.org/packages/6c/72/3e798c064bc39e471008075d0f9bc9daf77a80879c092e4a8e170c585ed4/cryptography-49.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:8c25ceb16df5b9435f3f6a9829204985b0e0cbee3b48aacd432c7d2c850b44d9", size = 4334173, upload-time = "2026-06-12T20:01:44.743Z" }, - { url = "https://files.pythonhosted.org/packages/f0/ee/6fca21d1ac73e06f8bef71940abfd4d2f6472b4bca284d770f32bd4086f6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:28d8b15e6275f12c8a207dc309dfa957903c927d08d0cc937ee3f63f200693cc", size = 4697298, upload-time = "2026-06-12T20:02:20.918Z" }, - { url = "https://files.pythonhosted.org/packages/67/d0/a5fcd3515f0bae49a7b6d0413cc1bdccdcc1fc0047037a0d480642cdc5d6/cryptography-49.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6fc361c34fb6aac015ce19435876635e5c6d21db31998b0920f675f131e043b8", size = 5254338, upload-time = "2026-06-12T20:02:22.737Z" }, - { url = "https://files.pythonhosted.org/packages/a0/84/84fe36f19caf857d61cb7fc9c63035a47ffabd84ea12d1d393148efa3615/cryptography-49.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:2400ef9c9e2299a25614eb1dea3db54a69b1349efd043bfac9c67630d136df36", size = 4735650, upload-time = "2026-06-12T20:02:41.389Z" }, - { url = "https://files.pythonhosted.org/packages/6c/a0/db537264e234f7273a73ec020873d6d6b39dfd8a53db78b550ca8320440e/cryptography-49.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:67e1d20ad9ef3a563c59ef22e7a8a0b8210bd26604369ea4a30a7c66aefe504e", size = 4834820, upload-time = "2026-06-12T20:01:51.847Z" }, - { url = "https://files.pythonhosted.org/packages/93/77/8df9eb486495979bccecd1062e2eaf435250e84437040295b57d09048b0b/cryptography-49.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:42b0684e0e40cf26122427802486f6d93aea593612603a94fbf260c7eb1e9c1b", size = 4967968, upload-time = "2026-06-12T20:02:12.524Z" }, - { url = "https://files.pythonhosted.org/packages/c2/e6/f60198ea8d9dfa15fff9ed4ca02ce362f6eadd9ba757dcc50634c4257b63/cryptography-49.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:026ac7423e6fa66872d3bf889be5974507da3944f866f704fa200eadacd00001", size = 3785547, upload-time = "2026-06-12T20:02:26.847Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/de/41/6cbdcf9142d00fe82836fbb51e503e58088575cf7a0fe1dbff6695bf0840/cryptography-50.0.0.tar.gz", hash = "sha256:eeac2acb5a20ed25e0ad6d1df9891a520b78b404266b6d11778f25d5d691a6c9", size = 880201, upload-time = "2026-07-31T14:25:10.11Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c5/5c/59086b4aac5e879d38ddbcf74e4be7ade89cebc3eb199a55da998c3bb46a/cryptography-50.0.0-cp311-abi3-macosx_11_0_arm64.whl", hash = "sha256:031e2d5dd4bb9caa3ca9c82e5a197fd8ae680232cee62603d1a813f3f07e3d03", size = 4001252, upload-time = "2026-07-31T14:23:33.331Z" }, + { url = "https://files.pythonhosted.org/packages/57/ef/8f2df13c7216bcad3e1c74e07f6e193d93e998e114f524a53877c9af27ad/cryptography-50.0.0-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fd9192b7b70c573d7f214eb1ae35e00d359f6f5e4b27c7e21e30de1fc6204645", size = 4719554, upload-time = "2026-07-31T14:23:35.611Z" }, + { url = "https://files.pythonhosted.org/packages/d9/41/029086c34d91052fc3b88bcc8056f709a7c915c7a23b235a54eb800b1c97/cryptography-50.0.0-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:06a32a980526a6ab9a4b9bf8f7385800791e2bb960903cb6b530e4817509a3b7", size = 4702130, upload-time = "2026-07-31T14:23:37.635Z" }, + { url = "https://files.pythonhosted.org/packages/7d/ff/b6ce0954962e7f7b969f850a883744197bb3910bdfd7b6da162eab7d9f68/cryptography-50.0.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:a1b30560f2acc95aa8b2e06e716a13dbfc97314747b80d9707e307f77b40d6b3", size = 4725244, upload-time = "2026-07-31T14:23:39.471Z" }, + { url = "https://files.pythonhosted.org/packages/06/1e/63a1027cb7fec360a182208e1b7767d5aa1fe57be3d6aa856e69a321edc0/cryptography-50.0.0-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:8d89f3976b10b4ce31118de72329025f70d2c6ead14a8217c5514dd2c6d5a78f", size = 5342265, upload-time = "2026-07-31T14:23:41.286Z" }, + { url = "https://files.pythonhosted.org/packages/6b/72/a1116d683a6d7ece94590013882515de087edf9ef0e6292aae615a44df73/cryptography-50.0.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:b42a28c1844fd9de8f3f7d540e36b66f3a9c83fceac7170ebc7a6a19edd9dcae", size = 4734609, upload-time = "2026-07-31T14:23:43.139Z" }, + { url = "https://files.pythonhosted.org/packages/15/37/36a9c479bbe49acea2636c7fd3360d20f7b7e079c300352011c44850b181/cryptography-50.0.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:900131fafd8aead39ac7dd3a7e833be754c17a95cfd91221636949fe4eb0aa8a", size = 4356517, upload-time = "2026-07-31T14:23:44.939Z" }, + { url = "https://files.pythonhosted.org/packages/32/98/8a151d64367204cbc63ec65d37502f1d9c53cf4bfc6ec3c532614dbec60d/cryptography-50.0.0-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:07949c449a1abcf60d1ee6e88956d89404c7df3c8258f46589e912988e551987", size = 4724529, upload-time = "2026-07-31T14:23:46.93Z" }, + { url = "https://files.pythonhosted.org/packages/22/f6/ec13b470172126464a86bf54d2294a46d29837fc51ba3e45d4047946fb5e/cryptography-50.0.0-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:f89831ef99dd7dd169ab06d63a831adb9e20a87aac6d380266bbda5823349169", size = 5299852, upload-time = "2026-07-31T14:23:48.851Z" }, + { url = "https://files.pythonhosted.org/packages/da/3a/f05e32c99d440c9bb891ea0e36c9091891e36be5a9a87ab2ee6ea20729f6/cryptography-50.0.0-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:82148ec5bddac30b51a5b3c1945075f896fa022cb93f8e4a01e9f6ee95292c5f", size = 4734462, upload-time = "2026-07-31T14:23:50.861Z" }, + { url = "https://files.pythonhosted.org/packages/ca/dc/bd72b26be8953f80625f63151efd38eee71c76ca6cf591c08ff34615a79e/cryptography-50.0.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:1489e263a8048bb8b6a8bac662eb2d402ea5d2b7b4699b72f385f1e2772db105", size = 4852708, upload-time = "2026-07-31T14:23:52.715Z" }, + { url = "https://files.pythonhosted.org/packages/27/20/c930314a2ab476d15dec966ec87e2e9637bb02b06106b12c0396c57bb603/cryptography-50.0.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:7cec5b856506da6defb290f30c9ee687d5f5e8cb0bd3f6459dde43b0b4fa40ef", size = 5004179, upload-time = "2026-07-31T14:23:54.887Z" }, + { url = "https://files.pythonhosted.org/packages/32/2e/c9db68a0c4bfa28e310707527c0ee3a2bd254104d2e02e68f368e197aa4c/cryptography-50.0.0-cp311-abi3-win_amd64.whl", hash = "sha256:bd1c592e4d5974f0d08d4888e432157adba757c66da0246918e43677fafa2d30", size = 3840395, upload-time = "2026-07-31T14:23:56.677Z" }, + { url = "https://files.pythonhosted.org/packages/c3/fb/951032a3bf22a5697c83183fb6294a4843772947a70e616c57b3ff5f522e/cryptography-50.0.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:49e7d93abdbd2990caced757e5fade25302f719c3c8fb6e6fff2dde98999fc41", size = 3989258, upload-time = "2026-07-31T14:23:58.881Z" }, + { url = "https://files.pythonhosted.org/packages/d4/67/91eb047e69c5e845f2f14b8a2e4a1aab0f283cb885531e9e22c8adb176bc/cryptography-50.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:19736989797678c6af1e55cd49055cdbcb55d8f6b5583ac5335f933aba9101dc", size = 4700648, upload-time = "2026-07-31T14:24:00.702Z" }, + { url = "https://files.pythonhosted.org/packages/30/82/85f0f7425c856b9f96459411eb12e74ef72df9caf6f8f15bf23a33ff131f/cryptography-50.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:80b63928fa35083b33966ce1efb70e5b9607181e49dcd1c22c8c005e319f667f", size = 4682442, upload-time = "2026-07-31T14:24:02.538Z" }, + { url = "https://files.pythonhosted.org/packages/1a/28/b555a365adff1cca2fbe7b9e487d68a40de6bc67ff2cb587473eb43de0e7/cryptography-50.0.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:d58c3db7cd6eed54e6c06744db55456b65ebd7492ddeae9c1e93cfca7aa857d3", size = 4707596, upload-time = "2026-07-31T14:24:04.394Z" }, + { url = "https://files.pythonhosted.org/packages/72/d8/f52538140cc719df62a01cf87d1c7142318d235817109d6f4054d7c352d6/cryptography-50.0.0-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:df2a58a472f332225671c35b0a830208b86d004f82baa8530fa3782c85646533", size = 5314552, upload-time = "2026-07-31T14:24:06.31Z" }, + { url = "https://files.pythonhosted.org/packages/38/14/6120e5bd7c5aa022ad15424ba4d5c5269d0d9448ed4d55e492ea91e3c1c4/cryptography-50.0.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:11b74db56cdbe3cdee6e3f6982ecb70334fa10dce99ed58bf7894aaaa3b2a037", size = 4717113, upload-time = "2026-07-31T14:24:08.349Z" }, + { url = "https://files.pythonhosted.org/packages/fa/71/190bf38c3ee2e0f8efc9860ae100c9df4169742eef274b91e7aa1cb133b9/cryptography-50.0.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f59e38625469987d7ef6d495323c55e7db6c212eaf6112267e0d3b565a2e9c9f", size = 4338580, upload-time = "2026-07-31T14:24:10.227Z" }, + { url = "https://files.pythonhosted.org/packages/3a/63/504ccfbbe61fd8aa983f7f146399cdf034c72c2fc55f5b2dfdcdcdb20c99/cryptography-50.0.0-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:ecfed7367f965a0328cfbdd70da860f15441f002f613185668c6e6ebf5a0ac11", size = 4707038, upload-time = "2026-07-31T14:24:12.169Z" }, + { url = "https://files.pythonhosted.org/packages/01/77/2cf79bbfc4d12ca106437a6e170d6aaa01a373e93093118aaaef0e801bd4/cryptography-50.0.0-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:9aa87839c383bdbab6ef865787a1fb877af8dd03464c4400322726feaaadfc6d", size = 5273110, upload-time = "2026-07-31T14:24:14.38Z" }, + { url = "https://files.pythonhosted.org/packages/e5/45/8aae2972c520145377ea3559a605a899bebe227bf070b33cdb445929a9b9/cryptography-50.0.0-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:6ba6a53445bd3cfa809ef3ef5f1589aa6ba08784a1d962bf47d0940e871dab1c", size = 4716439, upload-time = "2026-07-31T14:24:16.415Z" }, + { url = "https://files.pythonhosted.org/packages/7b/20/4fe50b619a48c2525cc46e2dbc1ac490708d704be5d467bdaac6dc955682/cryptography-50.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:3f5735ffe4996d28b809371756219f5354864902a3b9e7c0b9ee87041209fc9c", size = 4837383, upload-time = "2026-07-31T14:24:18.553Z" }, + { url = "https://files.pythonhosted.org/packages/92/91/3a31366e183343d3703f8995c095f5734676bd6938118047e50fcf279eb4/cryptography-50.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1b4a266766514614f8aa60416e71f2fc6e575d36e7bdc90f644fadb2f4b75b95", size = 4985772, upload-time = "2026-07-31T14:24:20.385Z" }, + { url = "https://files.pythonhosted.org/packages/74/9a/02ffe35b2853d121689871eb5dce862092562b3a1ed5cc98f1aaed441506/cryptography-50.0.0-cp314-cp314t-win_amd64.whl", hash = "sha256:12b9c6996425c76ea6c457ace4f3073e715b8c545add07cd1a8f3a4f90691269", size = 3816291, upload-time = "2026-07-31T14:24:22.125Z" }, + { url = "https://files.pythonhosted.org/packages/03/37/73d005be173aff344af30e9fd2a576575cb2391a7101d9cd3842e1fa8cce/cryptography-50.0.0-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:ccdc4a71a4dabae05de219404f9f4abc38e3b58422177ff93d0da05967dafa07", size = 4036009, upload-time = "2026-07-31T14:24:24.122Z" }, + { url = "https://files.pythonhosted.org/packages/ff/c6/7a6202a534e32103a285b7834a120869557fe198d51d7cfe59754c8bda9c/cryptography-50.0.0-cp39-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:910e1d2668e7de9648f2bcee30e180db2a6b15c30f887d7c4c93ddf96e3992e3", size = 4745252, upload-time = "2026-07-31T14:24:26.118Z" }, + { url = "https://files.pythonhosted.org/packages/85/4f/0fa8c2f4428198f15d9ff8d63400e27afbf94ce833f6108da1eb3753f945/cryptography-50.0.0-cp39-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a91296cb61e8df6f86d0c19cc4068228da256bf59bf86049fbd821084565327f", size = 4728939, upload-time = "2026-07-31T14:24:27.994Z" }, + { url = "https://files.pythonhosted.org/packages/d1/63/54dd723490ba2dc09b299682c10b38db38f159728bcaae8c591b8af2f22d/cryptography-50.0.0-cp39-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:e722f16708d854fe924790e051061f6704a472c3bac347b6fd88033ea8dd0dc5", size = 4748483, upload-time = "2026-07-31T14:24:30.254Z" }, + { url = "https://files.pythonhosted.org/packages/1d/dd/7c77d26285cc7f6991efce64a0f5b4f9383bfa5dd8c5033003eaf7db4cdb/cryptography-50.0.0-cp39-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:d764dcf130c428ef66786f866dd750f53182bc608813489915e9fc106bb0c82f", size = 5367599, upload-time = "2026-07-31T14:24:32.457Z" }, + { url = "https://files.pythonhosted.org/packages/46/c9/f60aed34c013f317f92817b6c171c2d22a78270fa41109bd4b08af26b194/cryptography-50.0.0-cp39-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:105110f43a471dbd0060b9c9516cb8a6a79233631a04cc2ba16f28323ac6e025", size = 4762647, upload-time = "2026-07-31T14:24:34.599Z" }, + { url = "https://files.pythonhosted.org/packages/be/f3/f9a0173b139372c3a48ed98154b45cc6b9de17c789d5ab552e621c293609/cryptography-50.0.0-cp39-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:828743d939e9629bc267b8e2d08d8bb67cd4319c771a33d4b18b22dd8fb7440a", size = 4385197, upload-time = "2026-07-31T14:24:36.647Z" }, + { url = "https://files.pythonhosted.org/packages/d8/36/83bb81f6e569bc38e1e4a7bc80f29b46bb9601920bc455fc8e888f5d5742/cryptography-50.0.0-cp39-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:2a8183b489dc1f7f80f135780fadc1108f14b31b8a40411c7a5b17425f65f28b", size = 4748095, upload-time = "2026-07-31T14:24:39.493Z" }, + { url = "https://files.pythonhosted.org/packages/6b/16/d3008eff98c764979865834c3d386d4fd041b5f52e7f34fc29ac1a5eb515/cryptography-50.0.0-cp39-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:6e7d61120573a7f2cd94cc095f9e81f6967c61ccdf194285aa143ecec8e0b708", size = 5325948, upload-time = "2026-07-31T14:24:41.556Z" }, + { url = "https://files.pythonhosted.org/packages/9c/f8/d97f9603efda3888187bfdb893f26c41be4735c10631d05d284ee6b047c4/cryptography-50.0.0-cp39-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:37fdb0d0111f1e2ff07139dfb79f1b49531f8e213c46f1163dd7642979b58c47", size = 4762400, upload-time = "2026-07-31T14:24:43.636Z" }, + { url = "https://files.pythonhosted.org/packages/64/a2/4615c8f7d81a00b1d6e6afe19f694e1543582349fb5f4076f6cb5dc36485/cryptography-50.0.0-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:c87f62a3d3b9888ed0fdde100ec06aa61ca9cd44bad9057d1dff9a516b5f5bb9", size = 4878208, upload-time = "2026-07-31T14:24:45.522Z" }, + { url = "https://files.pythonhosted.org/packages/d2/1a/efcfb02f91407149a0dacffffab791f7e19bf6385f63b3666dc8b5e5c9c8/cryptography-50.0.0-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:65c2c3add92b45fd0709db8594536aea39c2a67af0e27ffcf049c498501140b7", size = 5037050, upload-time = "2026-07-31T14:24:47.697Z" }, + { url = "https://files.pythonhosted.org/packages/57/30/4a22984d4f1bdfb8c054f07a92bc176b97a3134cc1d6c4b3bffb1f3688b4/cryptography-50.0.0-cp39-abi3-win_amd64.whl", hash = "sha256:d24fead1d4d076e1bfb006dcec392074a3cd8d7b4fc8a595aa64073b2b7a96ba", size = 3874135, upload-time = "2026-07-31T14:24:50.085Z" }, ] [[package]]