Add codex and change cost calc - #3
Draft
Intron7 wants to merge 15 commits into
Draft
Conversation
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
for more information, see https://pre-commit.ci
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3 +/- ##
==========================================
+ Coverage 63.20% 73.13% +9.93%
==========================================
Files 22 25 +3
Lines 2829 3972 +1143
==========================================
+ Hits 1788 2905 +1117
- Misses 1041 1067 +26
🚀 New features to boost your workflow:
|
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
Provider usage or credit running out is an infrastructure failure, not evidence about the model, so it must not be scored as an agent failure. Runs that hit provider exhaustion get reason `provider_exhausted` and a `valid: false` result. The pass cancels only that provider's remaining cells, lets other providers finish their running and queued work, and exits non-zero. Invalid cells stay pending, so replenishing the credential and rerunning the same command retries them without `--no-resume`. Reports and `improve` refuse invalid evidence outright. Both adapters now preserve provider stderr on failure so the runner can classify exhaustion from the actual provider error text. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`acumen bench` with no arm flag covered only the baseline, so comparing a skill against it took one invocation per version and `--dry-run` never showed what the project as a whole implied. A bare `acumen bench` now covers every arm on disk — the baseline plus each version in `skills/` — benched one after another against a single prepared target, with per-arm counts, per-arm tallies and a combined total. Resume still applies per arm, so adding `skills/v3` and rerunning costs only v3. `--no-skill` / `--skill vN` restrict the pass to one arm as before, and `--dry-run` plans the same set for free. Skills load before target prep, so a version that will not load stops the pass at planning instead of being dropped from the comparison or failing part-way through a paid run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The teal ramp had luna and terra swapped: luna took the mid-dark step and terra the lightest, though terra is the more potent of the two. Published rates put the tiers at sol $5/$30, terra $2/$12, luna $0.20/$1.20, so the ramp now steps sol -> terra -> luna, darkest to lightest, matching the warm Anthropic ramp. _MODEL_ORDER follows, which also puts the legend and bar order in the same sequence config.yaml already scaffolds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Prices move, and each run's cost is frozen into its result.json when the run finishes and never recomputed. A table compiled into a release is therefore wrong from whatever date the providers next change a price, and that wrongness gets stored rather than corrected on the next run. So no rates ship with the package at all. They come from the providers' pricing pages, or from a `prices:` pin in config.yaml, which still wins: only the operator knows what a negotiated rate or a gateway markup is. A model neither layer covers is unpriced, which was already a first-class state throughout (cost_usd is None, never 0.0). bench resolves rates before it spends anything and fails the pass if the pages cannot be read: cost is a headline metric, and a pass that cannot establish rates has not earned the numbers it would print. draft, improve, tasks and ship fetch too but degrade to unpriced, since their cost line is progress reporting rather than stored evidence. That degradation warns about its one real consequence: Codex's max_usd cap is derived from these rates, so an unpriced Codex run has no enforceable budget cap. Each run now records price_source and price_rates_as_of alongside the rates themselves, so passes months apart stay individually attributable and one report can mix them. The report flags arms priced on different dates, whose cost gap contains any price change and not only the skill's effect. Two things this surfaced: - A dated snapshot (claude-haiku-4-5-20251001) now falls back to its family rate. Providers publish one rate per family and never list the snapshot IDs, and acumen init scaffolds one, so a default project would otherwise have benched an entirely unpriced model. The old table hid this by hardcoding both IDs. - Three CLI tests began reaching the live network once bench started fetching, and passed only because the machine happened to be online. An autouse fixture now fails any test that touches the network, which is what the suite's docstring already claimed. acumen prices reads the pages too, since there is no offline table left to print; --refresh now reports pins that have drifted from the published price, the only rates that can still go stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Intron7 <severin.dicks@icloud.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.