diff --git a/CLAUDE.md b/CLAUDE.md index 6f6512ce..a0492cfa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,7 +6,7 @@ Guidance for Claude Code working in this repository. Every line here is loaded i Token Tracker is a local-first AI token usage tracker. -- **CLI** (`src/`, CommonJS, Node ≥20) — entry `bin/tracker.js` → `src/cli.js`. `serve` runs a local HTTP server on `:7680`, `sync` parses logs into `~/.tokentracker/queue.jsonl`. +- **CLI** (`src/`, CommonJS, Node ≥20) — entry `bin/tracker.js` → `src/cli.js`. `serve` runs a local HTTP server on `:7680`, `sync` parses logs into `~/.tokentracker/tracker/queue.jsonl`. - **Dashboard** (`dashboard/`, React 18 + Vite 7 + TS strict + Tailwind) — built to `dashboard/dist/`, served by the CLI on localhost. Local-only: there is no hosted deployment. - **macOS app** (`TokenTrackerBar/`, Swift 5.9, XcodeGen) — menu bar + WidgetKit. `EmbeddedServer/` bundles the CLI runtime + built dashboard so the `.app` is self-contained. - **Windows app** (`TokenTrackerWin/`, .NET 8 WinForms + WPF + WebView2) — system-tray counterpart of the macOS app. Launches the bundled CLI `serve` on a dynamic loopback port (avoids the DoSvc-held `:7680`), hosts the dashboard in WebView2, registers the `tokentracker://` deep-link for OAuth. Built `EmbeddedServer/` (Node + CLI + dashboard) is bundled by `scripts/bundle-node.ps1` so the `.exe` is self-contained. Dashboard adaptations are gated behind `isNativeWindowsApp()` (`dashboard/src/lib/native-bridge.js`) so macOS/web paths are untouched. @@ -18,7 +18,7 @@ For the canonical list of supported providers, grep `parse*Incremental` in `src/ ## Frequently used commands ```bash -npm test # node --test test/*.test.js (97 files) +npm test # node --test test/*.test.js node --test test/.test.js # single test file npm run ci:local # tests + validations + builds npm run dashboard:dev # Vite dev server with local API mock (port 5173) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index eed73fa4..84f9a71d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,7 +5,7 @@ Thanks for considering a contribution! TokenTracker is a small project, so the p ## Setup ```bash -git clone https://github.com/mm7894215/TokenTracker.git +git clone https://github.com/pitimon/TokenTracker.git cd TokenTracker npm install @@ -25,7 +25,7 @@ node bin/tracker.js doctor # Health check ## Tests ```bash -npm test # Full suite (96 test files, node --test) +npm test # Full suite (node --test over test/*.test.js) node --test test/rollout-parser.test.js # A single test file npm run ci:local # Tests + validations + builds (everything CI runs) ``` @@ -50,11 +50,11 @@ npm run validate:copy # Validate copy registry completene This is the most common kind of contribution. The pattern: -1. **Add a parser to `src/lib/rollout.js`** — most tools write JSONL or SQLite logs. The parser should normalize tokens into the canonical shape: `{input_tokens, output_tokens, cached_input_tokens, cache_creation_input_tokens, total_tokens, model, source, hour_start}`. +1. **Add a parser to `src/lib/rollout.js`** — most tools write JSONL or SQLite logs. The parser should normalize tokens into the canonical shape: `{input_tokens, output_tokens, cached_input_tokens, cache_creation_input_tokens, total_tokens, model, source, hour_start}`. This file is large; don't read it top to bottom — find the closest existing tool (`parseDroidIncremental`, `parseZedIncremental`, …) and copy its shape. Each parser exports a `resolve*`/`list*` pair for finding the tool's files and a `parse*Incremental` for reading them. 2. **Add a hook installer in `src/commands/init.js`** — most tools support a config file or hook script you can patch. Make it idempotent (safe to re-run). 3. **Add a status check in `src/commands/status.js`** — show whether the hook is installed and whether data has been collected. -4. **Add a parser test in `test/rollout-parser.test.js`** — use a real (anonymized) sample log fixture. -5. **Update `README.md` Supported AI Tools table** with the new row. +4. **Add a parser test in its own `test/-parser.test.js`** — use a real (anonymized) sample log fixture. Recent tools each got their own file (`droid-parser.test.js`, `zed-parser.test.js`, `goose-parser.test.js`); only the older ones share `rollout-parser.test.js`. +5. **Add the tool to the Supported tools list in `README.md`** — the blockquote under "🔌 Supported tools". Leave the "20+" count alone; a hard number in prose has no validator and goes stale silently. Look at how Claude Code, Codex, or Gemini are wired in for reference — they're the simplest examples. diff --git a/README.md b/README.md index e1ff1d10..920a03fd 100644 --- a/README.md +++ b/README.md @@ -4,7 +4,7 @@ ### Know exactly what you're spending on AI — across every CLI -Auto-collect token usage from **22 AI coding tools**, aggregate it locally, and read real cost trends in one dashboard. No account or API key required to start — just one command. +Auto-collect token usage from **20+ AI coding tools**, aggregate it locally, and read real cost trends in one dashboard. No account or API key required to start — just one command. [![npm version](https://img.shields.io/npm/v/@ipv9/tokentracker-cli.svg?color=blue)](https://www.npmjs.com/package/@ipv9/tokentracker-cli) [![npm downloads](https://img.shields.io/npm/dm/@ipv9/tokentracker-cli.svg?color=brightgreen)](https://www.npmjs.com/package/@ipv9/tokentracker-cli) @@ -24,6 +24,19 @@ Auto-collect token usage from **22 AI coding tools**, aggregate it locally, and --- +## 🤔 Why not just read each provider's billing page? + +You can — that is the honest alternative, and for a single tool it is enough. TokenTracker earns its place once you use more than one: + +- **One number instead of six tabs.** Claude, Codex, Cursor, Gemini and Copilot each bill in their own dashboard, on their own reset schedule, in their own units. Nobody adds them up for you. +- **Subscriptions hide the number entirely.** A flat monthly plan shows you a quota bar, not what your usage would have cost. TokenTracker prices every token against public model rates, so you can see whether the plan is a bargain or a subsidy you have outgrown. +- **Per-project and per-model, not just per-account.** Billing pages answer "what do I owe this month". This answers "which repo, which model, and which hour" — the resolution you need to actually change something. +- **Quota chips before you hit the wall.** Live plan-limit usage sits on each provider's card, so a 5-hour window running out is something you see rather than something you discover. + +If you only use one tool and never care about per-project cost, the provider's own page is genuinely fine. This is for the rest. + +--- + ## ⚡ Quick Start > **Requires** Node.js **20+**. @@ -48,21 +61,34 @@ tokentracker doctor # health check --- +## 🖥️ Prefer an app? There's a desktop build + +If you'd rather not keep a terminal open, both native apps are on the [Releases page](https://github.com/pitimon/TokenTracker/releases/latest): + +| Platform | Download | What it adds | +|---|---|---| +| **macOS 12+** | `TokenTrackerBar.dmg` | A menu-bar app — live token count in the menu bar, launch at login, sync and update from a click, plus a desktop widget. | +| **Windows** | `TokenTracker-Setup.exe` | The same dashboard as a standalone app. | + +TokenTracker in the macOS menu bar + +Both bundle their own Node runtime, so there is nothing else to install. They share the same local data as the CLI — run either, or both. The desktop builds are cut less often than the npm package, so the latest release tag usually trails the npm version badge above. + +--- + ## ✨ What you get -- 🔒 **Private by design.** Runs entirely on your machine — token counts and timestamps only, never prompts, responses, or file contents. No account, no required API keys, no telemetry, no phone-home. Nothing ever leaves your laptop. -- 📊 **One calm web dashboard.** Your whole picture in the browser at a local URL (no login): total spend, usage trend, per-provider breakdown, context breakdown, and a GitHub-style activity heatmap — light or dark, auto-refreshing while the tab is open. +- 🔒 **Your usage data never leaves your machine.** Token counts and timestamps only — never prompts, responses, or file contents. No account, no telemetry, no analytics, no phone-home. TokenTracker does make a few outbound calls *on your behalf* (model prices, your own plan quotas); every one is named in the Privacy section below, and none of them carry your usage. +- 📊 **One calm web dashboard.** Your whole picture in the browser at a local URL, no login — light or dark, auto-refreshing while the tab is open. - 📈 **Quota at a glance, on every card.** Live plan-quota usage (used %, e.g. 5h + weekly) as color-coded chips right on each provider's card — see how close you are to your limits without leaving the overview. Full windows + reset countdowns on the Limits page. Covers Claude, Codex, Cursor, Gemini, Kimi, Z.AI, Kiro, Copilot, and Antigravity. -- 💰 **Cost you can trust.** 2,200+ models priced from [LiteLLM](https://github.com/BerriAI/litellm) (refreshed daily) with a bundled offline snapshot, so USD totals are right even without a network. Cross-provider records are de-duplicated to match each provider's own billing. -- 🔌 **22 tools, zero config.** Claude Code, Codex, Cursor, Gemini, Copilot, Antigravity, OpenCode, Kiro, Zed, Goose, and more — auto-detected, hooks auto-install on first run. Zero to dashboard in ~30 seconds. +- 💰 **Cost you can trust — and a price tag when it can't.** 2,200+ models priced from [LiteLLM](https://github.com/BerriAI/litellm) (refreshed daily) with a bundled offline snapshot, so USD totals are right even without a network. A model too new to have a price is badged **pricing missing** rather than quietly counted as $0, and prices refresh in the background instead of waiting for a restart. Cross-provider records are de-duplicated to match each provider's own billing. +- 🔌 **20+ tools, zero config.** Claude Code, Codex, Cursor, Gemini, Copilot, Antigravity, OpenCode, Kiro, Zed, Goose, and more — auto-detected, hooks auto-install on first run. Zero to dashboard in ~30 seconds. - 🧩 **Skills tab.** Syncs 250+ public skills across your tools. --- ## 📊 The dashboard -A calm, single-screen readout — a hero total paired with a usage-trend chart, a provider breakdown stacked above a per-provider context breakdown, and a GitHub-style activity heatmap. - | Dark | Light | |---|---| | Dashboard — dark | Dashboard — light | @@ -83,7 +109,7 @@ Auto-refresh runs only while the tab is visible (`Off` / `30s` / `60s` / `120s`, Auto-detected on first run — no manual plugin or hook wiring: -> **Claude Code · Codex CLI · Cursor · Gemini CLI · GitHub Copilot · Antigravity · Kiro · OpenCode · OpenClaw · Every Code · Hermes · Kimi Code · CodeBuddy · Grok Build · oh-my-pi · pi · Craft Agents · Kilo CLI · Kilo Code · Roo Code · Zed Agent · Goose** +> **Claude Code · Codex CLI · Cursor · Gemini CLI · GitHub Copilot · Antigravity · Kiro · OpenCode · OpenClaw · Every Code · Hermes · Kimi Code · CodeBuddy · Grok Build · Droid · oh-my-pi · pi · Craft Agents · Kilo CLI · Kilo Code · Roo Code · Zed Agent · Goose** Each tool is connected one of three ways, all automatic: a **SessionEnd/notify hook** (Claude Code, Codex, Gemini, Every Code, CodeBuddy, Grok Build), a **bundled plugin** linked via the tool's own CLI (OpenCode, OpenClaw), or a **passive reader** that only reads files the tool already writes — SQLite, JSONL, OTEL exports (Cursor, Kiro, Copilot, Zed, Goose, and the rest). @@ -96,16 +122,16 @@ Rate-limit providers are auto-detected where possible. For Z.AI / GLM Coding Pla ## 🏗️ How it works ``` -AI CLI tools → hooks / passive readers → local SQLite → dashboard - (logs) (token counts only) (30-min buckets) (your browser) +AI CLI tools → hooks / passive readers → local queue file → dashboard + (logs) (token counts only) (30-min buckets) (your browser) ``` 1. Your AI tools write logs during normal use. -2. Lightweight hooks (or passive file readers) pick up token counts locally — never prompt or response content. -3. Counts are aggregated into 30-minute UTC buckets in a local SQLite snapshot. -4. The dashboard reads that snapshot and renders it in your browser's timezone. +2. Lightweight hooks (or passive file readers) pick up token counts locally — never prompt or response content. Some tools keep their logs in SQLite (Cursor, Kiro, Zed and friends); TokenTracker only ever *reads* those. +3. Counts are aggregated into 30-minute UTC buckets and appended to one plain-text file: `~/.tokentracker/tracker/queue.jsonl`. +4. The dashboard reads that file and renders it in your browser's timezone. -Nothing leaves your machine. There is no account, no upload, and no server to sign in to. +No account, no upload of your usage, and no server to sign in to. --- @@ -114,9 +140,20 @@ Nothing leaves your machine. There is no account, no upload, and no server to si | Protection | What it means | |---|---| | **No content** | Only token counts and timestamps. Never prompts, responses, or files. | -| **Local only** | All data stays on your machine. There is no upload path at all. | -| **Auditable** | Open source — read [`src/lib/rollout.js`](src/lib/rollout.js); it's just numbers and timestamps. | -| **No telemetry** | No analytics, no crash reporting, no phone-home. | +| **Your usage stays local** | Every count TokenTracker collects is written to one file on your disk and read back by a server on your own machine. There is no endpoint it uploads usage to. | +| **Auditable in one command** | You don't have to take our word for it — the store is an append-only text file you can open yourself: `cat ~/.tokentracker/tracker/queue.jsonl`. It's numbers and timestamps. | +| **No telemetry** | No analytics, no crash reporting, no phone-home, no account. | + +**Outbound calls, on your behalf only.** TokenTracker is local-first, not network-free. It talks to the internet in exactly these cases, and none of them carry your usage data: + +| When | Where | Why | +|---|---|---| +| Pricing refresh (daily) | `raw.githubusercontent.com` | Downloads the public [LiteLLM](https://github.com/BerriAI/litellm) price list. Anonymous — no credentials, nothing sent. Works offline from a bundled snapshot. | +| Quota chips + Limits page | `api.anthropic.com`, `chatgpt.com`, `cursor.com`, `cloudcode-pa.googleapis.com`, `api.kimi.com`, `api.z.ai`, `api.github.com` | Asks *your* provider about *your* plan limits, using credentials already on your machine. Only for providers you actually use. | +| Token refresh | `auth.openai.com`, `oauth2.googleapis.com`, `auth.kimi.com` | Renews those same provider credentials when they expire. | +| Profile avatars | Allowlisted avatar CDNs | Fetched server-side so your browser doesn't contact them directly. | +| IP check page | `ip.net.coffee` | Only if you open that page. | +| `npx` startup | npm registry | How `npx` works — it downloads the package. A global install avoids it. | --- @@ -147,29 +184,30 @@ Browser auto-open is opt-in: `tokentracker serve --open`. Background services an --- -## 🛠️ Development +## ⏱️ Always-on, without a terminal (macOS) + +If you want the dashboard up all the time but don't want the desktop app, the repo ships a launchd installer. It registers two LaunchAgents — the dashboard on port `7680`, and a periodic background sync — both pinned to a specific published version: ```bash git clone https://github.com/pitimon/TokenTracker.git cd TokenTracker -npm install +./scripts/install-local-service.sh # remove later with ./scripts/uninstall-local-service.sh +``` -# build the dashboard, then run the CLI -npm run dashboard:build -node bin/tracker.js +macOS only; it uses `launchd` directly. On Linux, the same effect is a small systemd user unit running `tokentracker serve --sync --no-open`. -npm test # root tests -npm run ci:local # full local gate (build + tests + validators) -``` +--- + +## 🛠️ Development -## 📚 Code Documentation +```bash +git clone https://github.com/pitimon/TokenTracker.git +cd TokenTracker && npm install +npm run dashboard:build && node bin/tracker.js +npm run ci:local # the full gate: build + tests + validators +``` -Source-backed engineering documentation starts at -[`openwiki/README.md`](openwiki/README.md). Regenerate the local fact -ledger with `npm run docs:openwiki:extract`, validate it with -`npm run docs:openwiki:check`, and use `npm run docs:openwiki:verify` for the -independent read-only review. The model-backed update command expects credentials -from the caller's environment and never reads them from this repository. +Setup details, the test layout, and how to add a new tool integration are in [CONTRIBUTING.md](CONTRIBUTING.md). Source-backed engineering documentation starts at [`openwiki/README.md`](openwiki/README.md). --- @@ -185,7 +223,7 @@ tokentracker status # see each integration's state tokentracker doctor # deeper health check ``` -If a tool you use shows as not configured, run `tokentracker activate-if-needed` to re-run detection. Still missing? [Open an issue](https://github.com/pitimon/TokenTracker/issues/new) with the `doctor` output. +If a tool you use shows as not configured, run `tokentracker init` — it re-runs detection and installs anything missing. Still missing? [Open an issue](https://github.com/pitimon/TokenTracker/issues/new) with the `doctor` output. @@ -261,6 +299,14 @@ tokentracker uninstall Removes every hook TokenTracker installed across all detected tools, plus local config and data. Safe to re-run. +One thing it does **not** touch: if you set up the always-on macOS service yourself with `scripts/install-local-service.sh`, that LaunchAgent is installed outside the CLI and keeps restarting the dashboard. Remove it first: + +```bash +./scripts/uninstall-local-service.sh +``` + +(The CLI never installs a LaunchAgent, so if you have only ever run `npx`/`tokentracker`, there is nothing extra to clean up.) + --- diff --git a/dashboard/src/lib/model-breakdown.ts b/dashboard/src/lib/model-breakdown.ts index 5c45ea5a..48c2a9a6 100644 --- a/dashboard/src/lib/model-breakdown.ts +++ b/dashboard/src/lib/model-breakdown.ts @@ -25,6 +25,13 @@ function resolveModelName(model: any, fallback: any) { // price is plausible and therefore never looks wrong — worth flagging. const FUZZY_PRICING_TIERS = new Set(["curated:fuzzy", "litellm:fuzzy", "litellm:prefix-strip"]); +// Server tiers that mean "$0 because we have no price", as opposed to a model +// that genuinely costs nothing. "unattributed"/"empty" rows carry no model id at +// all, but their tokens still count toward the total, so the same caveat applies +// — they are excluded from the server's unpriced_models list (which exists to +// name models needing a curated price) but must not silently read as priced. +const UNPRICED_PRICING_TIERS = new Set(["miss", "unattributed", "empty"]); + function isKnownZeroCostModel(name: any) { const lower = String(name || "").toLowerCase(); return lower.includes("free") || lower.includes("hy3-preview") || /^glm-[\d.]+-flash(?![a-z])/.test(lower); @@ -92,7 +99,7 @@ export function buildFleetData(modelBreakdown: any, { copyFn }: AnyRecord = {}) // stays as the fallback for responses from an older server. const pricingTier = typeof model?.pricing_tier === "string" ? model.pricing_tier : null; const pricingMissing = pricingTier - ? pricingTier === "miss" && modelTokens > 0 + ? UNPRICED_PRICING_TIERS.has(pricingTier) && modelTokens > 0 : modelTokens > 0 && (modelCost == null || modelCost <= 0) && !isKnownZeroCostModel(name); const pricingFuzzy = Boolean(pricingTier && FUZZY_PRICING_TIERS.has(pricingTier)); return { diff --git a/docs/npm-publish-checklist.md b/docs/npm-publish-checklist.md index cc899959..d28e5cee 100644 --- a/docs/npm-publish-checklist.md +++ b/docs/npm-publish-checklist.md @@ -28,8 +28,22 @@ service scripts under `scripts/`. Dashboard pricing/UI release gate: -- `claude-sonnet-5`, `claude-fable-5`, and `claude-opus-4-8` must resolve to - non-zero pricing before publish. Use `node --test test/pricing.test.js`. +- Every model you actually used must resolve to non-zero pricing before publish. + Run `node --test test/pricing.test.js`, then check the live surface rather than + a hard-coded list of model names — a list in prose goes stale the week a new + model ships, which is the failure this gate exists to catch: + + ```bash + FROM=$(python3 -c 'import datetime;print(datetime.date.today()-datetime.timedelta(days=7))') + TO=$(python3 -c 'import datetime;print(datetime.date.today())') + curl -s "http://localhost:7680/functions/tokentracker-usage-model-breakdown?from=$FROM&to=$TO" \ + | python3 -c 'import json,sys; p=json.load(sys.stdin)["pricing"]; print(p["unpriced_models"], p["fuzzy_priced_models"])' + ``` + + `unpriced_models` should be empty. Anything listed needs an entry in + `src/lib/pricing/curated-overrides.json` before you publish. Entries in + `fuzzy_priced_models` are priced by partial match — plausible but possibly + wrong, so confirm them rather than assuming. - Collapsed provider cards must show model chips, top-cost signal, and a pricing-missing badge when a non-free model has tokens but zero cost. Use the focused `UsageOverview` and `model-breakdown` tests before relying on a diff --git a/docs/screenshots/leaderboard.png b/docs/screenshots/leaderboard.png deleted file mode 100644 index 86e42129..00000000 Binary files a/docs/screenshots/leaderboard.png and /dev/null differ diff --git a/openwiki/README.md b/openwiki/README.md index 4125f9cf..2b74a8ac 100644 --- a/openwiki/README.md +++ b/openwiki/README.md @@ -92,8 +92,8 @@ transformed between them, see the [Data flow](architecture/dataflow.md) view. ## Start here -- [Quickstart](quickstart.md): documentation workflow, source ledger, and local - update commands. +- [Working on this documentation](quickstart.md): the source ledger, the + regeneration commands, and what the fact checker enforces. - [Architecture](architecture/overview.md): runtime components and boundaries. - [Data flow](architecture/dataflow.md): how usage data moves from tool logs to the dashboard, with a leveled data-flow diagram. diff --git a/openwiki/local-api.md b/openwiki/local-api.md index ce0c0e83..34053f52 100644 --- a/openwiki/local-api.md +++ b/openwiki/local-api.md @@ -30,6 +30,56 @@ The mutation endpoint checks local authorization. The skills endpoint has its ow method-specific behavior. Do not expose either endpoint beyond the local server without re-evaluating that security model. +## Pricing diagnostics + +`/functions/tokentracker-usage-model-breakdown` is the one endpoint that reports +how much to trust its own numbers. Each model in `sources[].models[]` carries a +`pricing_tier`, and the response's `pricing` object carries a snapshot of what +the pricing layer knows it guessed at or missed. Both come from +`getPricingDiagnostics()` in `src/lib/pricing/index.js`; read that function +before relying on the exact field set. + +The tiers come from the resolution ladder in `src/lib/pricing/matcher.js` — that +function is the authority; the grouping below is what each rung means for +trusting the number. The distinctions matter because three different situations +all produce a `$0` cost, and because a *guessed* price is never `$0` and so +cannot be spotted by looking for zeros. + +**Resolved exactly** — the id matched, trust the price: + +| Tier | Rung | +| --- | --- | +| `curated:exact` | Matched a key in `curated-overrides.json`. Curated always wins over LiteLLM. | +| `curated:exact-dot`, `litellm:exact-dot` | Matched exactly after rejoining dash-separated numerics (`glm-5-1` → `glm-5.1`), for providers that dash-normalize version numbers. | +| `litellm:exact` | Matched a LiteLLM key. | +| `curated:alias` | A deliberate curated mapping, e.g. Cursor's `auto` → `composer-1`. Intentional, not inferred. | +| `litellm:strip` | Matched the base model after removing a reasoning-effort suffix (`-high`, `-xhigh`, `-fast`, …). Reasoning effort changes how many tokens you spend, not the per-token rate, so the base price is the right one. | + +**Guessed** — plausible, possibly another model's price. These are what +`fuzzy_priced_models` reports: + +| Tier | Rung | +| --- | --- | +| `curated:fuzzy` | A curated substring rule matched. | +| `litellm:prefix-strip` | A provider-qualified key ended with the bare model name. Where several providers expose the same model, the lexicographically smallest key wins — deterministic, but the chosen provider's rate may not be yours. | +| `litellm:fuzzy` | Reverse substring, longest key first: the model id *contains* a known key. | + +**Not priced** — all cost `$0`, for three different reasons: + +| Tier | Rung | +| --- | --- | +| `miss` | Nothing matched. Needs an entry in `curated-overrides.json`; this is what `unpriced_models` lists. | +| `unattributed` | The row had no model id and is stored under the placeholder `unknown`. Also `$0`, but there is nothing to add a price for — deliberately excluded from `unpriced_models`. | +| `empty` | No model id passed at all. | + +`unpriced_models` lists only `miss` models — it is a work list of ids needing a +curated price, so placeholders are deliberately excluded from it. +`fuzzy_priced_models` lists the guessed ones. A `miss`, or a snapshot older than +its TTL, triggers a single-flight background refresh; `refreshing`, `stale`, and +`last_refresh_error` report that machinery. A refresh that cannot reach upstream +is discarded rather than installed, so a failed refresh never replaces good +prices with an older snapshot's. + ## Related modules - `src/lib/pricing/` resolves model pricing used by local aggregations. diff --git a/openwiki/quickstart.md b/openwiki/quickstart.md index 08f8f540..35a7f297 100644 --- a/openwiki/quickstart.md +++ b/openwiki/quickstart.md @@ -1,18 +1,8 @@ -# TokenTracker OpenWiki +# Working on this documentation -TokenTracker is a local-first token-usage tracker. The Node CLI parses supported -tool logs into local queue files, serves a dashboard on loopback, and can be -bundled by the macOS and Windows desktop applications. - -Start with [architecture](architecture/overview.md) for the data flow. Use these -pages for change-oriented source maps: - -- [CLI and operations](cli-and-operations.md) -- [Parsers and sync](parsers-and-sync.md) -- [Local API](local-api.md) -- [Dashboard routes](dashboard-routes.md) -- [Native app boundaries](native-app-boundaries.md) -- [Testing and release](testing-and-release.md) +How to keep these pages true: where the authority lives, which commands +regenerate it, and the entry points worth knowing before you edit. For the map +of the pages themselves, start at [the OpenWiki index](README.md). ## Source of truth @@ -26,6 +16,20 @@ npm run docs:openwiki:extract npm run docs:openwiki:check ``` +The check runs two different ways over two different file sets: + +- **Every reference must resolve** — across `openwiki/**`, plus `README.md` and + `CONTRIBUTING.md`. A CLI command, `/functions/*` endpoint, dashboard route, or + `parse*Incremental` symbol named in any of them must exist in the ledger. The + front-door docs are included because that is where readers copy commands from: + `README.md` spent a long time telling users with a broken install to run a + repair command the CLI has never had, while this checker passed every time + without ever looking at that file. There is deliberately no way to silence a + finding — if a page needs to name something that does not exist, describe it + instead of quoting it. +- **Every contract must be documented** — `openwiki/**` only. The README is a + front door, not a manifest; it is not required to list every endpoint. + ## Local flow ```text diff --git a/scripts/openwiki-check-facts.cjs b/scripts/openwiki-check-facts.cjs index fae9d04e..c2ef502a 100644 --- a/scripts/openwiki-check-facts.cjs +++ b/scripts/openwiki-check-facts.cjs @@ -21,7 +21,12 @@ function findLine(content, index) { return content.slice(0, index).split("\n").length; } -function collectFindings({ facts, files, root = ROOT }) { +// `files` are scanned for references that must resolve (an unknown command in +// any doc is a defect). `coverageFiles` are the subset that must *also* be +// complete — every real command/endpoint/route documented somewhere. Only +// openwiki/ carries that obligation: the README is a front door, not a manifest, +// and requiring it to list every endpoint would be the wrong kind of pressure. +function collectFindings({ facts, files, coverageFiles = files, root = ROOT }) { const findings = []; const commandNames = new Set(facts.cli.commands.map((command) => command.name)); const endpointNames = new Set(facts.local_api.endpoints.map((endpoint) => endpoint.path)); @@ -30,14 +35,16 @@ function collectFindings({ facts, files, root = ROOT }) { const documentedCommands = new Set(); const documentedEndpoints = new Set(); const documentedRoutes = new Set(); + const coveragePaths = new Set(coverageFiles.map((file) => file.path)); for (const file of files) { const relative = path.relative(root, file.path); + const countsForCoverage = coveragePaths.has(file.path); for (const match of file.content.matchAll(/(?:npx --yes @ipv9\/tokentracker-cli|(? file.content).join("\n"); + const allDocumentation = coverageFiles.map((file) => file.content).join("\n"); for (const route of routeNames) { if (allDocumentation.includes(`\`${route}\``)) documentedRoutes.add(route); } @@ -92,12 +99,25 @@ function checkFacts({ root = ROOT } = {}) { if (JSON.stringify(saved) !== JSON.stringify(current)) { findings.push("openwiki-facts/source-facts.json is stale; run npm run docs:openwiki:extract"); } - const files = readMarkdownFiles(path.join(root, "openwiki")); - if (files.length === 0) { + const coverageFiles = readMarkdownFiles(path.join(root, "openwiki")); + if (coverageFiles.length === 0) { findings.push("openwiki/ contains no Markdown documentation"); return findings; } - return findings.concat(collectFindings({ facts: current, files, root })); + // The front-door docs are checked for unresolvable references too. They are + // what users actually run commands from: README.md told broken installs to run + // `tokentracker activate-if-needed`, a command that has never existed, and + // this check would have caught it on the day it was written had it been + // looking. Scoped to the two root docs a reader is told to follow. + const files = coverageFiles.concat(readRootDocs(root)); + return findings.concat(collectFindings({ facts: current, files, coverageFiles, root })); +} + +function readRootDocs(root) { + return ["README.md", "CONTRIBUTING.md"] + .map((name) => path.join(root, name)) + .filter((filePath) => fs.existsSync(filePath)) + .map((filePath) => ({ path: filePath, content: fs.readFileSync(filePath, "utf8") })); } function main() { @@ -109,4 +129,4 @@ function main() { if (require.main === module) main(); -module.exports = { checkFacts, collectFindings }; +module.exports = { checkFacts, collectFindings, readRootDocs }; diff --git a/src/lib/pricing/index.js b/src/lib/pricing/index.js index b91a2444..d4dc4416 100644 --- a/src/lib/pricing/index.js +++ b/src/lib/pricing/index.js @@ -51,6 +51,11 @@ const RELOAD_COOLDOWN_MS = 5 * 60 * 1000; // wrong, unlike a $0 one. const FUZZY_SOURCES = new Set(["curated:fuzzy", "litellm:fuzzy", "litellm:prefix-strip"]); +// Placeholder ids that stand in for "this row has no model", so they resolve to +// the "unattributed" tier instead of being looked up and recorded as a miss. +// Closed set on purpose: a real model id must never be silently un-priced here. +const UNATTRIBUTED_MODEL_IDS = new Set(["unknown"]); + // `last_refresh_error` is served over HTTP to the dashboard, so it is built // from CLOSED sets, never from an arbitrary value. A previous version accepted // anything symbol-shaped, which a QA pass broke immediately: a 32-character @@ -204,11 +209,28 @@ function resolveLookupSource(opts) { return null; } +// A row whose model id could not be determined is stored and aggregated under +// the literal id "unknown" — persisted into the queue by src/commands/sync.js +// and src/lib/claude-categorizer.js, then coalesced to it again at read time by +// src/lib/local-api.js. That is a placeholder for "no model", not a model, so +// pricing it turned a missing *attribution* into a missing *price*: it recorded +// a permanent miss, listed "unknown" in unpriced_models as though a real model +// needed a curated price, and logged advice to add it to curated-overrides.json +// where it could never match anything. +// Returns the tier to report, or null when this is a real model id to look up. +function resolvePlaceholderTier(model) { + if (!model) return "empty"; + const normalized = String(model).trim().toLowerCase(); + if (!normalized) return "empty"; + return UNATTRIBUTED_MODEL_IDS.has(normalized) ? "unattributed" : null; +} + // Returns the price AND how it was resolved. getModelPricing keeps the old // bare-numbers contract for the many existing callers; anything that wants to // show the user how much to trust the number uses this. function getModelPricingMeta(model, opts = {}) { - if (!model) return { pricing: ZERO_PRICING, tier: "empty" }; + const placeholderTier = resolvePlaceholderTier(model); + if (placeholderTier) return { pricing: ZERO_PRICING, tier: placeholderTier }; const lookupSource = resolveLookupSource(opts); const cacheKey = lookupSource ? `${lookupSource}\0${model}` : model; diff --git a/test/model-breakdown.test.js b/test/model-breakdown.test.js index eb5b716c..1c9782ea 100644 --- a/test/model-breakdown.test.js +++ b/test/model-breakdown.test.js @@ -718,3 +718,40 @@ test("without pricing_tier the cost<=0 heuristic still applies (older server res assert.deepEqual(provider.missingPricingModels.map((m) => m.name), ["brand-new-model"]); assert.deepEqual(provider.fuzzyPricingModels, []); }); + +test("an unattributed row still carries the unpriced caveat", async () => { + const mod = await loadDashboardModule("dashboard/src/lib/model-breakdown.ts"); + const { buildFleetData } = mod; + + // The server excludes the "unknown" placeholder from unpriced_models — that + // list names models needing a curated price, and a placeholder is not one. + // The dashboard chip answers a different question: "are these tokens counted + // at $0?" For an unattributed row they are, so it must still be flagged. On + // this machine every such row currently has 0 tokens and is dropped earlier, + // so only a synthesized row exercises the path. + const [provider] = buildFleetData({ + sources: [ + { + source: "claude", + totals: { billable_total_tokens: 2000, total_cost_usd: "5" }, + models: [ + { + model: "unknown", + model_id: "unknown", + pricing_tier: "unattributed", + totals: { billable_total_tokens: 1000, total_cost_usd: "0" }, + }, + { + model: "acme-1", + model_id: "acme-1", + pricing_tier: "litellm:exact", + totals: { billable_total_tokens: 1000, total_cost_usd: "5" }, + }, + ], + }, + ], + }); + + assert.deepEqual(provider.missingPricingModels.map((m) => m.name), ["unknown"]); + assert.deepEqual(provider.fuzzyPricingModels, []); +}); diff --git a/test/openwiki-facts.test.js b/test/openwiki-facts.test.js index 655913cb..a7ae317b 100644 --- a/test/openwiki-facts.test.js +++ b/test/openwiki-facts.test.js @@ -1,8 +1,10 @@ const assert = require("node:assert/strict"); +const fs = require("node:fs"); +const path = require("node:path"); const { test } = require("node:test"); const { extractFacts } = require("../scripts/openwiki-extract-facts.cjs"); -const { collectFindings } = require("../scripts/openwiki-check-facts.cjs"); +const { checkFacts, collectFindings, readRootDocs } = require("../scripts/openwiki-check-facts.cjs"); test("OpenWiki facts expose TokenTracker's public command, API, route, and parser contracts", () => { const facts = extractFacts(); @@ -24,3 +26,48 @@ test("OpenWiki fact checker rejects unsupported concrete claims", () => { }); assert.equal(findings.filter((finding) => finding.includes("unknown")).length, 4); }); + +test("the fact checker reads the front-door docs, not only openwiki/", () => { + // README.md told users with a broken install to run a repair command the CLI + // has never had. The checker that would have caught it existed the whole time; + // it just wasn't looking at the file users actually follow. + // + // Asserted through readRootDocs rather than by writing a fake command into the + // real README: a killed process or a CI timeout would skip the restore, and + // the recovery path is `git add README.md` — which would silently re-commit + // the exact falsehood this check exists to prevent. + const scanned = readRootDocs(process.cwd()).map((file) => path.basename(file.path)); + assert.deepEqual(scanned.sort(), ["CONTRIBUTING.md", "README.md"]); + assert.deepEqual(checkFacts(), [], "repo docs are clean as committed"); + + const facts = extractFacts(); + const findings = collectFindings({ + facts, + root: process.cwd(), + files: [{ path: `${process.cwd()}/README.md`, content: "Run `tokentracker activate-if-needed` to fix.\n" }], + }); + assert.ok( + findings.some((f) => f.startsWith("README.md:") && f.includes("activate-if-needed")), + `expected the fake command to be rejected, got: ${JSON.stringify(findings)}`, + ); +}); + +test("the front-door docs do not owe openwiki's completeness obligation", () => { + // README is a front door, not a manifest. Documenting a command there must not + // satisfy openwiki's "every command is documented" check, or extending the + // scan would have quietly weakened the coverage half of the same validator. + const facts = extractFacts(); + const findings = collectFindings({ + facts, + root: process.cwd(), + files: [ + { path: `${process.cwd()}/README.md`, content: "`tokentracker doctor`\n" }, + { path: `${process.cwd()}/openwiki/example.md`, content: "nothing documented here\n" }, + ], + coverageFiles: [{ path: `${process.cwd()}/openwiki/example.md`, content: "nothing documented here\n" }], + }); + assert.ok( + findings.includes("openwiki/ missing CLI command 'doctor'"), + "a command documented only in README must still count as undocumented in openwiki", + ); +}); diff --git a/test/pricing-observability.test.js b/test/pricing-observability.test.js index 2f55404c..f096ddd4 100644 --- a/test/pricing-observability.test.js +++ b/test/pricing-observability.test.js @@ -343,3 +343,91 @@ test("one provider's exact hit cannot hide another provider's miss", async () => const diagnostics = pricing.getPricingDiagnostics(); assert.deepEqual(diagnostics.unpriced_models, ["unknown-only-here"]); }); + +// --- the "unknown" placeholder is not a model ------------------------------- +// Rows whose model id could not be determined are stored under the literal id +// "unknown" (src/commands/sync.js:1250,1675,1768) and coalesced to it again at +// read time (src/lib/local-api.js:210,434,1121,1355). Pricing it turned a +// missing *attribution* into a missing *price*, which is a different problem +// with a different fix — the live dashboard reported unpriced_models: +// ["unknown"] and advised adding it to curated-overrides.json, where it could +// never match anything. + +test("the \"unknown\" placeholder resolves as unattributed, not as an unpriced model", async () => { + const payload = { current: { "acme-1": entry(1e-6, 2e-6) } }; + await loadWith(payload, tmpCachePath("unattributed")); + + const { result: meta, lines } = captureWarnings(() => + pricing.getModelPricingMeta("unknown", { source: "claude" }), + ); + + assert.equal(meta.tier, "unattributed"); + assert.deepEqual(meta.pricing, pricing.ZERO_PRICING, "still costs nothing, as before"); + assert.deepEqual(lines, [], "no advice to add a placeholder to curated-overrides.json"); + + const diagnostics = pricing.getPricingDiagnostics(); + assert.deepEqual( + diagnostics.unpriced_models, + [], + "unpriced_models names models needing a curated price — a placeholder is not one", + ); + assert.deepEqual(diagnostics.fuzzy_priced_models, []); + assert.equal( + pricing.__getStateForTests().reloadPromise, + null, + "a placeholder must not burn an upstream refresh looking for a price it can never have", + ); +}); + +test("a model id that is only whitespace is empty, not a placeholder-shaped miss", async () => { + const payload = { current: { "acme-1": entry(1e-6, 2e-6) } }; + await loadWith(payload, tmpCachePath("blank")); + + // `!model` misses this, so before the trim it reached the lookup and leaked + // into unpriced_models as " " — the same defect wearing different characters. + assert.equal(pricing.getModelPricingMeta(" ").tier, "empty"); + assert.deepEqual(pricing.getPricingDiagnostics().unpriced_models, []); +}); + +test("case and padding do not smuggle the placeholder past the check", async () => { + const payload = { current: { "acme-1": entry(1e-6, 2e-6) } }; + await loadWith(payload, tmpCachePath("placeholder-variants")); + + for (const variant of ["UNKNOWN", " Unknown ", "unknown\n"]) { + assert.equal( + pricing.getModelPricingMeta(variant).tier, + "unattributed", + `${JSON.stringify(variant)} is the same placeholder`, + ); + } + assert.deepEqual(pricing.getPricingDiagnostics().unpriced_models, []); +}); + +test("a real model that merely contains \"unknown\" is still priced or missed normally", async () => { + // The exemption is a closed set, not a substring rule: silently un-pricing a + // real model would recreate the $0 bug this whole surface exists to expose. + const payload = { current: { "unknown-labs-v2": entry(3e-6, 6e-6) } }; + await loadWith(payload, tmpCachePath("real-model")); + + assert.equal(pricing.getModelPricingMeta("unknown-labs-v2").tier, "litellm:exact"); + // Contains the priced key, so it resolves the same way any other model would. + assert.equal(pricing.getModelPricingMeta("unknown-labs-v2-preview").tier, "litellm:fuzzy"); + assert.equal(pricing.getModelPricingMeta("mystery-model").tier, "miss"); + assert.deepEqual(pricing.getPricingDiagnostics().unpriced_models, ["mystery-model"]); +}); + +test("cost for an unattributed row is unchanged by the exemption", async () => { + const payload = { current: { "acme-1": entry(1e-6, 2e-6) } }; + await loadWith(payload, tmpCachePath("unattributed-cost")); + + // The row was already costing $0 (no price to apply). The fix changes how it + // is *reported*, and must not quietly change anyone's total. + const cost = pricing.computeRowCost({ + model: "unknown", + source: "claude", + input_tokens: 1_000_000, + output_tokens: 1_000_000, + }); + assert.equal(cost, 0); + assert.deepEqual(pricing.getModelPricing("unknown"), pricing.ZERO_PRICING); +});