feat(pricing): in-process refresh + report how each price resolved by pitimon · Pull Request #92 · pitimon/TokenTracker

pitimon · 2026-07-24T22:36:57Z

Summary

Closes the "the dollar number is confidently wrong and nothing says so" class that produced the claude-opus-5 $0 incident.

The root cause was not the stale cache file. ensurePricingLoaded() returns early once state.loaded is true, so a running process never reloads — the fetcher's 24h TTL only ever chose which snapshot to load at startup. The dashboard LaunchAgent had been up 21 hours. Deleting the cache was incidental; a restart alone would have fixed it, and nothing short of a restart ever would have.

Background refresh on a lookup miss, or when the snapshot has aged past its TTL. Single-flight, never awaited by the request path, with a 5-minute cooldown so a model that is genuinely absent upstream cannot turn every row into a fetch. A failed reload keeps serving what it has.
getModelPricingMeta() returns the resolution tier next to the price. getModelPricing keeps its bare-numbers contract, so no existing caller changes.
One warning per unknown model per process, naming the model and stating that its cost is being counted as $0.
pricing_tier per model on the model-breakdown response, plus unpriced_models, fuzzy_priced_models, and the snapshot's age/source.
Dashboard prefers the server tier over its cost <= 0 guess, and surfaces fuzzy-priced models.

Two things worth knowing before reviewing

1. The "no UI badge" premise was wrong — half of this already existed. buildFleetData already computes pricingMissing and both HeroSummary and ProviderBreakdownCard already render a count chip. So during the incident the dashboard did show a missing-pricing chip; it was just weak enough that nobody looked. What was genuinely missing is that the client had to infer it from cost <= 0, which cannot distinguish an unpriced model from a genuinely free one — hence the isKnownZeroCostModel name-sniffing hack (includes("free"), hy3-preview, glm-*-flash). The server now says which it is, and the heuristic stays only as the fallback path for an older server response.

2. Fuzzy pricing had no surface at all, and it is the more dangerous half. matcher.js resolves acme-9-turbo against the literal substring key acme-9 and returns that price. The result is non-zero, so no cost <= 0 heuristic can ever flag it — a guessed price never looks wrong. curated-overrides.json already carries {"match": "kimi", "ref": "kimi-k2.5"}, and the file already needed a kimi-k2.6 exact entry to escape that net.

3. ttlMs: 0 does not force a refetch. The first implementation used it, and a test caught the reload silently re-reading the same snapshot: isFresh compares Date.now() - stat.mtimeMs < ttlMs, and mtime carries sub-millisecond precision that Date.now() does not — a cache file written moments earlier compares as "written in the future" and counts as fresh. Fixed with an explicit forceRefresh flag on loadLitellmData.

Test plan

Follow-ups not in this PR

The fuzzy tiers are now reported but still accepted. Requiring a family match (first-token equality) before accepting a step-6 substring hit would prevent the mis-price rather than label it — worth doing once there is data on how often the fuzzy tier actually fires in practice.
_meta.expiries enforcement and the loopback/lock hardening are fix: correct expired DeepSeek prices, enforce curated expiries, harden Host + sync lock #91.

Closes #90

The claude-opus-5 incident was not a stale cache file — it was that a running process never reloads. ensurePricingLoaded() returns early once state.loaded is true, so the fetcher's 24h TTL only ever chose which snapshot to load at startup. The dashboard LaunchAgent had been up 21 hours; it showed 3.87M tokens at $0.000000 with no signal, and nothing short of a restart would have changed that. Three defects, one fix, because lookupPricing already computes everything needed and getModelPricing was throwing it away. - Reload in the background when a lookup misses or the snapshot has aged past its TTL. Single-flight, never awaited by the request, with a 5-minute cooldown so a model that is genuinely absent upstream cannot turn every row into a fetch. A failed reload keeps the snapshot it has. - Add getModelPricingMeta(), which returns the resolution tier alongside the price. getModelPricing keeps its bare-numbers contract. - Warn once per unknown model per process, naming the model and saying its cost is being counted as $0. - Expose pricing_tier per model plus unpriced_models / fuzzy_priced_models and the snapshot's age and source on the model-breakdown response. - Dashboard: prefer the server's tier over the cost<=0 guess, which cannot tell an unpriced model from a genuinely free one, and surface fuzzy prices — those were invisible because a guessed price is never $0 and so never looked wrong. The heuristic stays as the fallback for an older server response. Also adds forceRefresh to loadLitellmData. `ttlMs: 0` looks like it forces a refetch and does not: mtime carries sub-millisecond precision that Date.now() lacks, so a cache file written moments earlier compares as "written in the future" and still counts as fresh. A test caught the reload silently re-reading the same snapshot. Closes #90

…tics endpoint

Independent QA pass (Codex, xhigh) found the reload path re-introducing the bug it was written to fix. loadLitellmData falls back on its own — upstream, then the stale disk cache, then the bundled seed. A background refresh took whatever came back and installed it. So with the disk cache gone (deleting it is the documented workaround for this very bug) and upstream unreachable, a refresh would replace good in-memory data with the OLDER seed. Reproduced: a model priced at $5/$25 dropped to $0 after one failed refresh. - Background reloads now only install data that actually came from upstream; anything else is reported and discarded. The initial load still accepts any source, because at that point there is nothing better to keep. - Report only the error CODE, never the message. fs and fetch errors carry absolute paths, and this string is served over HTTP to the dashboard. - Key the tier map by source+model like the lookup itself. Keyed by model alone, one provider's exact hit could hide another provider's miss — Antigravity normalises model names before lookup, so collisions are real. - The old failed-refresh test left the disk cache in place, so the fetcher re-read the same data and the test passed while proving nothing. It now deletes the cache, which is what makes the downgrade observable, and asserts the prices survive. Added a test that the reported error carries no path. - Mirror pricing_tier in the vite dev mock, or `dashboard:dev` cannot exercise the unpriced/fuzzy badges at all.

Independent QA pass (Codex, xhigh) on the merged preview of #91 + #92. - Refuse absolute-form request targets. Host said loopback while the target carried its own authority, and routing parses the absolute URL — so the allowlist and the router disagreed about which site the request was for. Not reachable from a browser (absolute-form only goes to proxies), but a parser differential is not something to leave open in the one guard that stands between a rebound page and the whole spend history. - Allow the fully-qualified loopback spelling. WHATWG URL canonicalises a trailing dot away for IPv4 literals but not for names, so `localhost.` got a spurious 403 while `127.0.0.1.` passed. - Refuse userinfo in a Host header. `evil.example@127.0.0.1` was accepted; the origin genuinely is loopback so this was never a bypass, but Host has no userinfo component and anything carrying one is malformed. - Scan _meta values, not key names, for unenforced dates. Matching only `*_expiry` meant `promo_cutover: "2026-08-31 — update the price"` sailed past and would have expired in silence — the exact failure the validator exists to prevent. Now any YYYY-MM-DD parked anywhere in _meta (including nested) must live in the expiries array.

…ource keying QA re-check found blocker 5 still open and blocker 3 untested. - `e.code || e.name` was interpolated straight into a field served over HTTP. Those are normally short symbols, but a rejected promise can carry any object — `{code: "/Users/alice/private/pricing.json"}` reached the dashboard verbatim. Accept only symbol-shaped values, fall back to a constant. - The sanitizer is unit-tested directly rather than through the reload. loadLitellmData recovers from a failed fetch on its own, so driving it end to end only ever exercises the "fell-back" branch: an integration test asserting "no slash" passed without touching the sanitizer at all. The end-to-end test stays, but it is no longer the proof. - Added the two-source collision test the re-check flagged as missing: the same model id resolving exactly for one source and missing for another must keep both verdicts. Reverting the source-aware keying now fails a test.

QA re-check broke the regex sanitizer in one line: `sk_live_` plus 24 characters is symbol-shaped too, so `refresh-failed:sk_live_AAAA…` was reachable. No pattern separates "a short error symbol" from "a short secret" — only an allowlist does. - Replace the regex with two closed sets: known fs/network error codes and error class names, and the four sources loadLitellmData can report. Anything else becomes "unknown". - Unit-test the allowlist against a token, a path, a GitHub-style key and an object with a lying toString; assert the real codes still pass through. - The end-to-end test now asserts the reported value is a member of the full allowed set rather than merely "contains no slash". - Removed a test for scheduleReload's catch branch. It could not be driven deterministically — loadLitellmData recovers internally and only throws when cachePath is falsy, which loadInto substitutes away — and the attempt made a real network call while asserting nothing. The gap is recorded as a comment instead of covered by a test that cannot fail.

…anch The QA gate flagged the comment as too strong: statSafe rethrows a non-ENOENT stat error, so an unusable cache path does reach scheduleReload's catch (probe observed refresh-failed:TypeError). Sanitization held, but the comment claimed the branch was practically unreachable. Record the real gap instead.

@localhost

…n Host + sync lock (#91) * fix(pricing): apply post-promo DeepSeek rates, machine-check curated expiries deepseek-v4-pro was still pinned to its 75%-off launch promo 55 days after the promo ended, so every DeepSeek row billed at 25% of true cost. The file documented the cutover date itself, in prose, and nothing read it. A stale price is worse than a missing one: a missing price shows $0 and looks broken, a stale price looks fine forever. - Correct deepseek-v4-pro to the standard rates (1.74 / 3.48 / 0.0145 / 1.74) - Replace the free-text _meta.*_expiry keys with a structured _meta.expiries array (id / expires_at / what / action), carrying the Sonnet 5 2026-08-31 cutover that was about to repeat the same failure - Add validate:curated-expiry to ci:local so a PR opened on or after an expiry date fails until a human applies the action and clears the entry; it also rejects any regression to free-text *_expiry keys - Stop duplicating price literals in model-breakdown's coverage test — it asserts the lookup path (aliases, prefixes, casing) and reads expected rates from the curated table, so a legitimate price fix touches one file Closes #87 * fix(serve,sync): reject rebound Host headers, heartbeat the sync lock Two small hardening fixes for windows that are cheap to close now and expensive to diagnose afterwards. Host header (#88): binding to loopback does not make the Host header trustworthy. Under DNS rebinding a browser sends Host: attacker.example to 127.0.0.1 and treats the response as same-origin, so CORS never applies. Mutations were already gated on a loopback Origin, but every GET /functions/* endpoint — full spend history, model mix, project names — was readable by any page the victim had open. Requests whose Host is not loopback now get a 403 before any routing. isLoopbackHostname is reused from local-api so the Host allowlist and the Origin allowlist cannot drift. The request handler moved out of cmdServe into createRequestHandler so the test boots a real server and asserts the API handler is never reached — a guard that exists but is never wired is exactly the failure being prevented. Sync lock (#89): the lock had a 5-minute staleness window and never refreshed its own mtime, while local-sync fires on an interval. Any sync longer than one tick — full-corpus rebuilds and migration reparses are — had its lock stolen, letting two writers interleave appends into queue.jsonl. A torn line is silently skipped by the reader, and a skipped retraction row is a permanent overcount. - Heartbeat the lock mtime every 30s (unref'd, cleared on release) and raise the stale window to 30 minutes: "stale" now means the holder died - Record pid/host/startedAt in the lock, and reclaim immediately when the recorded process is gone — faster recovery than the old window, not slower - Gate the stale takeover behind an atomic mkdir mutex. The previous check-then-act let two waiters both delete and both acquire; a rename-based claim was tried first and still failed a 4-way race test, because rename is atomic but does not bind the check to the act - Clean up a failed lock write instead of leaking the fd and an empty file Closes #88 Closes #89 * fix(serve,pricing): close QA findings on the Host guard and expiry check Independent QA pass (Codex, xhigh) on the merged preview of #91 + #92. - Refuse absolute-form request targets. Host said loopback while the target carried its own authority, and routing parses the absolute URL — so the allowlist and the router disagreed about which site the request was for. Not reachable from a browser (absolute-form only goes to proxies), but a parser differential is not something to leave open in the one guard that stands between a rebound page and the whole spend history. - Allow the fully-qualified loopback spelling. WHATWG URL canonicalises a trailing dot away for IPv4 literals but not for names, so `localhost.` got a spurious 403 while `127.0.0.1.` passed. - Refuse userinfo in a Host header. `evil.example@127.0.0.1` was accepted; the origin genuinely is loopback so this was never a bypass, but Host has no userinfo component and anything carrying one is malformed. - Scan _meta values, not key names, for unenforced dates. Matching only `*_expiry` meant `promo_cutover: "2026-08-31 — update the price"` sailed past and would have expired in silence — the exact failure the validator exists to prevent. Now any YYYY-MM-DD parked anywhere in _meta (including nested) must live in the expiries array. * fix(serve): treat empty userinfo as userinfo, refuse network-path targets QA re-check found two gaps in the previous round's guard. - "@localhost" and ":@localhost" parse to a falsy url.username, so checking the parsed fields let exactly the malformed forms through while the fully-spelled "user:pass@localhost" was refused. Test the raw header for "@" instead. - "//evil/x" and "/\\evil/x" start with a slash and so passed the origin-form check, but WHATWG URL resolves both against a foreign authority (new URL("/\\evil/x", "http://localhost").hostname === "evil"). Routing only reads url.pathname today, so nothing is exploitable now — but handing a handler a URL that points at someone else's origin is the same guard-vs-parser disagreement absolute-form creates. * fix(serve): refuse control characters in the Host header and request target QA re-check found the last hole in the prefix checks: WHATWG URL strips tab, LF and CR from its input BEFORE parsing, so "/<tab>//evil/x" becomes "//evil/x" and adopts a foreign authority after passing a startsWith("//") test. Same trick applies to the Host string. Verified over a socket that Node's own parser returns 400 for those bytes in a request-target before the handler ever runs, so this was not reachable through the real server — recorded in the test so the next reader does not have to re-derive it. Fixed anyway: a guard that holds only because a different layer happens to be strict is exactly the guard-vs-parser disagreement this function exists to prevent. --------- Co-authored-by: itarun.p <itarun.p@somapait.com>

…vability

Ships the QA-gated fixes from #91 and #92: corrected DeepSeek rates with a machine-checked expiry, Host-header and sync-lock hardening, and pricing that refreshes in-process and reports how each price resolved. prepublishOnly re-vendored the LiteLLM seed, which now carries claude-opus-5 — so a cold start prices it correctly even before the first background refresh.

Closes issue 106. README:32 already made this claim and the product did not compute it: it showed what usage WOULD cost at list price, and it showed a quota bar, but the comparison — the actual decision — was left as mental arithmetic against a plan price nothing ever asked for. Every input already existed: per-source cost per window and the pricing tier telling you how much to trust it, both already on the model-breakdown endpoint. The arithmetic is a subtraction. THE LABELLING IS THE HARD PART, which is why the logic lives in a pure module with its own tests rather than inside JSX. Three rules, all from the issue, each with a test that fails if a future edit drops it: 1. It never says "you saved". This is LIST-PRICE-EQUIVALENT, not a counterfactual bill — anyone actually on the API would use it differently. A test asserts the rendered card contains no "saved", "savings", "wasted", "worth it" or "bargain", and the disclaimer line always renders. 2. It inherits the pricing caveats from #92. If any model in the window is unpriced or fuzzy-matched, the figure is a FLOOR and says so, naming the models. `unattributed` from #94 counts as unpriced. One floor anywhere makes the whole roll-up a floor — averaging confidence would let a well-priced provider vouch for a badly-priced one. 3. Under-usage reads exactly like over-usage. "$3 of usage on a $20 plan" is a downgrade signal and is as useful as the other direction, so neither gets a colour or a verdict. Tested by asserting the below case renders the same sentence with no "over"/"under" language. The plan price is entered by the user and stored in localStorage. Never sent anywhere — no plan catalogue to maintain and go stale, and a plan price is exactly the kind of thing this product promises not to transmit. A test asserts the storage hook contains no fetch and imports no API client. Absent is never zero, in three places: no stored price means no comparison rather than a $0 plan (which is infinitely over by construction), clearing the field stores nothing rather than 0, and a provider the user has not priced is left out of the roll-up entirely rather than inflating the list-price side. I nearly shipped this without an input. The card rendered "enter what you pay" with nowhere to enter it, which is not a feature. The inputs render in BOTH states because in the empty state they are the only way out of it. README:32 rewritten to match what shipped rather than left overselling it — the issue's own instruction if the honest framing could not be made to fit. It can, so the claim now names the window, the list-price-equivalence, and the floor. 37 new tests: 14 on the pure comparison, 8 on storage (corrupt JSON, non-object, non-positive values, a write that throws), 14 on the card, plus a wiring guard. That last one is a source-fact test rather than a render test because there is no DashboardView harness — and this session already produced the lesson twice: a Vite build stayed green over a deleted identifier, and a notice went into a component nothing mounts. ci:local exit 0: 952 root tests, 302 dashboard. Co-authored-by: itarun.p <itarun.p@somapait.com>

itarun.p added 3 commits July 25, 2026 05:36

chore(docs): regenerate openwiki source facts for the pricing diagnos…

311a7db

…tics endpoint

itarun.p added 3 commits July 25, 2026 06:31

Merge remote-tracking branch 'origin/main' into feat/90-pricing-obser…

ed325b3

…vability

pitimon merged commit 5f5c163 into main Jul 25, 2026
1 check passed

pitimon deleted the feat/90-pricing-observability branch July 25, 2026 00:12

Navigation Menu

feat(pricing): in-process refresh + report how each price resolved - #92

feat(pricing): in-process refresh + report how each price resolved#92
pitimon merged 7 commits into
mainfrom
feat/90-pricing-observability

pitimon commented Jul 24, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

pitimon commented Jul 24, 2026

Summary

Two things worth knowing before reviewing

Test plan

Follow-ups not in this PR

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant