Skip to content

Track multi-provider spend and model usage - #2759

Open
Chipagosfinest wants to merge 56 commits into
steipete:mainfrom
Chipagosfinest:agent/cumulative-spend-converged
Open

Track multi-provider spend and model usage#2759
Chipagosfinest wants to merge 56 commits into
steipete:mainfrom
Chipagosfinest:agent/cumulative-spend-converged

Conversation

@Chipagosfinest

@Chipagosfinest Chipagosfinest commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this changes

CodexBar Usage & Spend now works as a true multi-provider overview instead of stopping at Claude and Codex.

  • Every enabled or configured service stays visible, including providers without spend history yet.
  • The Overview leads with combined estimated spend, tracked tokens, and a clear reporting-provider count.
  • Compact native rows and smooth scrolling keep large provider rosters usable while Codex remains first.
  • Copy Stats and Copy Image produce a shareable local summary with honest partial-coverage disclosure.
  • Top models preserve meaningful variants such as Claude Fable, Opus, and Sonnet.

Unknown spend is never turned into $0. Estimates use ~, currencies remain separate, and incomplete coverage stays explicit.

Screenshots

All values shown here are synthetic.

Combined Overview

CodexBar Overview with combined estimated spend, token total, and six providers

Share card

CodexBar local share card with six tracked services, partial spend coverage, and top models

Wide settings · Narrow settings · Proof notes

OpenRouter scope

OpenRouter stays inside the existing plugin boundary on JavaScriptCore and QuickJS. An ordinary API key continues to report credits and key quota.

Management /activity enrichment is deliberately deferred to a separate follow-up. Removing it restores the existing single optional-request timeout budget and avoids adding a provider-specific history payload or a broader async QuickJS host contract to this PR.

Final integration fixes

  • Merged current main at bff43f8a2cc736a9dd248f6fb275f594b3a472c7.
  • Changed share copy from "connected services" to truthful "tracked services" terminology.
  • Preserved the saved overview ordering while enforcing explicit custom provider caps.
  • Reconciled the exact-line provider architecture gatekeeper with the merged source.
  • Repaired a stale integration assertion to verify the requested history window is forwarded.

Verification

Exact head: 6e7b6178a3cc2ed86e1e9490b7e551b1eed997a8

  • Exact-head CI run 31452605577: all 9 checks passed, including both macOS Swift test shards, Linux x64/arm64, Linux musl, lint, and GitGuardian.
  • Provider architecture catalog: 389 exact anchors validated with zero mismatches after the merge.
  • SwiftFormat and git diff --check passed for the follow-up fixes.
  • Local screenshot proof covers the production Overview, share window, responsive Settings layouts, and Copy Stats flow with synthetic provider data.

This is the current-main successor to #2469 and includes the provider-runtime convergence from #2758. OpenRouter /activity is preserved separately for a clean follow-up after this lands.

Stacked follow-up

#2855 adds the next focused layer: a provider-colored known-token allocation strip and approximate same-row cost per 1M tokens in the Overview summary card.

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. labels Aug 8, 2026
@clawsweeper

clawsweeper Bot commented Aug 8, 2026

Copy link
Copy Markdown

Codex review: needs real behavior proof before merge. Reviewed August 11, 2026, 1:46 AM ET / 05:46 UTC.

ClawSweeper review

What this changes

This PR expands Usage & Spend into a coverage-aware multi-provider dashboard, menu summary, and local share card with 7-, 30-, and 365-day periods.

Merge readiness

Blocked until stronger real behavior proof is added - 7 items remain

Keep open: current main still caps an unset Overview selection at six providers, so this is a distinct feature, but replacing that default needs owner approval and the changed Codex RPC timeout recovery still lacks live proof. Likely related people: Peter Steinberger (high confidence) and Yuxin Qiao (medium confidence).

Priority: P2
Reviewed head: 6e7b6178a3cc2ed86e1e9490b7e551b1eed997a8
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The UI evidence and CI are useful, but real timeout recovery proof and an explicit compatibility decision remain merge gates.
Proof confidence 🦐 gold shrimp (3/6) Needs stronger real behavior proof before merge: Prepared synthetic screenshots convincingly show the native UI and share flow, but not the changed live Codex RPC timeout/EOF recovery; add a redacted terminal trace, then update the PR body for re-review.
Patch quality 🦐 gold shrimp (3/6) 1 actionable review finding remain.

Verification

Check Result Evidence
Real behavior Needs proof Needs stronger real behavior proof before merge: Prepared synthetic screenshots convincingly show the native UI and share flow, but not the changed live Codex RPC timeout/EOF recovery; add a redacted terminal trace, then update the PR body for re-review.
Evidence reviewed 7 items Current default behavior: Current main limits the merged Overview to six providers when no saved selection exists, so the proposed all-provider default changes existing behavior.
Current selection resolver: The current resolver returns the first six normalized active providers when no selection preference is stored.
Timeout path affected: Current main races the RPC body against a timeout that terminates the Codex subprocess; the PR changes this EOF/timeout ordering, which requires production-boundary recovery evidence.
Findings 1 actionable finding [P1] Preserve or explicitly approve the no-selection default
Security None None.

How this fits together

CodexBar gathers usage and cost snapshots from enabled providers, aggregates them by currency and coverage, then renders settings, menu, and share-card summaries. This PR changes both the aggregation horizon and which providers are surfaced to users.

flowchart LR
A[Enabled providers] --> B[Usage and cost snapshots]
B --> C[Coverage-aware aggregation]
C --> D[Settings dashboard]
C --> E[Menu overview]
C --> F[Share card]
Loading

Decision needed

Question Recommendation
Should users without a saved Overview selection move from the existing six-provider default to every active provider on upgrade? Preserve legacy default: Keep the six-provider fallback and require an explicit user selection or a separately approved migration for all-provider display.

Why: Current main deliberately caps this path at six, while the PR intentionally changes that behavior; the compatibility trade-off is product policy rather than a mechanical correction.

Before merge

  • Add real behavior proof - Needs stronger real behavior proof before merge: Prepared synthetic screenshots convincingly show the native UI and share flow, but not the changed live Codex RPC timeout/EOF recovery; add a redacted terminal trace, then update the PR body for re-review.
  • Preserve or explicitly approve the no-selection default (P1) - This replaces the existing six-provider fallback for users without a saved selection with every active provider. The PR discussion identifies this as requiring owner sign-off; preserve the fallback or record an explicit upgrade decision before landing.
  • Resolve merge risk (P1) - Users with no saved Overview selection would move from six visible providers to every active provider on upgrade.
  • Resolve merge risk (P2) - The screenshots do not exercise the changed live Codex subprocess timeout/EOF recovery path.
  • Resolve merge risk (P1) - The local partial clone could not hydrate the exact PR tree for a second complete source-level diff pass.
  • Complete next step (P2) - A maintainer must choose the upgrade policy, and the contributor must supply live timeout recovery proof; neither is a safe automated repair.

Findings

  • [P1] Preserve or explicitly approve the no-selection default — Sources/CodexBar/SettingsStore.swift:191
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch scope 74 files; +3,468 / -353 lines The feature crosses provider aggregation, menu rendering, settings, localization, documentation, and tests, so the two remaining merge gates apply broadly.

Root-cause cluster

Relationship: canonical
Canonical: #2759
Summary: This PR is the active successor for the multi-provider spend-dashboard work; related work is either superseded, merged test coverage, or a stacked follow-up.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

  1. Capture timeout recovery (recommended)
    Add a redacted live subprocess trace showing a forced Codex RPC timeout is reported as a timeout and the next probe recovers.
  2. Approve the upgrade behavior
    Explicitly accept the no-selection transition from six providers to all active providers before merge.
  3. Keep the existing fallback
    Restore the six-provider no-selection behavior if the broader menu is not an approved upgrade change.

Technical review

Best possible solution:

Land the all-provider default only after explicit upgrade-policy approval and a redacted live Codex app-server timeout trace that shows timeout classification and recovery.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this is a feature PR, not a report of a currently failing behavior.

Is this the best way to solve the issue?

No: the expanded default is not ready to land until its upgrade behavior is explicitly approved and the timeout-path change has production-boundary proof.

Full review comments:

  • [P1] Preserve or explicitly approve the no-selection default — Sources/CodexBar/SettingsStore.swift:191
    This replaces the existing six-provider fallback for users without a saved selection with every active provider. The PR discussion identifies this as requiring owner sign-off; preserve the fallback or record an explicit upgrade decision before landing.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.84

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8d113c642516.

Labels

Label justifications:

  • P2: This is a substantial but non-emergency Usage & Spend feature whose remaining blockers are compatibility direction and proof.
  • merge-risk: 🚨 compatibility: The PR changes the no-selection Overview default from six active providers to every active provider.
  • merge-risk: 🚨 availability: The patch changes Codex RPC timeout handling, and live subprocess recovery evidence is still absent.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: Prepared synthetic screenshots convincingly show the native UI and share flow, but not the changed live Codex RPC timeout/EOF recovery; add a redacted terminal trace, then update the PR body for re-review.
  • proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. Prepared synthetic screenshots convincingly show the native UI and share flow, but not the changed live Codex RPC timeout/EOF recovery; add a redacted terminal trace, then update the PR body for re-review.

Evidence

What I checked:

  • Current default behavior: Current main limits the merged Overview to six providers when no saved selection exists, so the proposed all-provider default changes existing behavior. (Sources/CodexBar/SettingsStore.swift:191, 8d113c642516)
  • Current selection resolver: The current resolver returns the first six normalized active providers when no selection preference is stored. (Sources/CodexBar/SettingsStore+Defaults.swift:871, 8d113c642516)
  • Timeout path affected: Current main races the RPC body against a timeout that terminates the Codex subprocess; the PR changes this EOF/timeout ordering, which requires production-boundary recovery evidence. (Sources/CodexBarCore/UsageFetcher.swift:1015, 8d113c642516)
  • Provided UI proof inspected: The prepared Overview, share-card, and wide/narrow Settings screenshots visibly demonstrate the intended synthetic multi-provider UI, but not subprocess timeout recovery. (6e7b6178a3cc)
  • Feature history: Current-main history includes the ProviderInstanceID seam and shared-cache token activity work that this dashboard expansion builds upon. (Sources/CodexBar/SpendDashboardController.swift, 521af81e1e10)
  • Release context: v0.49.2 is the latest release baseline; this open PR is not part of that shipped release. (330ae4384b18)

Likely related people:

  • Peter Steinberger: Introduced the ProviderInstanceID seam and shared-cache token activity work underlying the dashboard’s provider/account aggregation. (role: recent area contributor; confidence: high; commits: 15a855a2132a, 521af81e1e10; files: Sources/CodexBar/SettingsStore.swift, Sources/CodexBar/SpendDashboardController.swift)
  • Yuxin Qiao: Added current-main spend-coverage and dashboard token-activity regression coverage. (role: recent test contributor; confidence: medium; commits: f61744bad39f, 68f26b5d65c9; files: Tests/CodexBarTests/SpendDashboardTokenActivityIntegrationTests.swift)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Add a redacted live Codex app-server timeout-and-recovery trace.
  • Obtain owner approval for, or revert, the all-provider no-selection default.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (57 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-09T18:15:36.293Z sha a573a79 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:14:01.090Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:35:53.777Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T19:49:56.586Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-09T21:12:41.155Z sha 8bcaf4e :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T02:20:40.603Z sha 05b58b9 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T02:41:27.885Z sha 6e7b617 :: needs real behavior proof before merge. :: none
  • reviewed 2026-08-11T03:04:56.348Z sha 6e7b617 :: needs real behavior proof before merge. :: none

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b03a03bd1a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Sources/CodexBar/SpendDashboardModel.swift
Comment thread Sources/CodexBar/UsageStore+TokenCost.swift Outdated
Comment thread Sources/CodexBar/StatusItemController+OverviewSpend.swift Outdated
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

Thanks for the sharp review. Two clarifications on the product and architecture decisions:

  • All connected providers in Overview is intentional for both unset fresh and unset existing installs; preserving the six-provider fallback would keep the exact omission this PR is meant to fix. The no-selection path is covered with 20 providers, explicit saved subsets remain scoped, and long rosters use native scrolling.
  • The OpenRouter bridge is intentionally a narrow first-party exception, gated to .openrouter, bounded and validated in Swift, Codable-compatible, fail-open for optional enrichment, and covered across JavaScriptCore/QuickJS. We considered a generic token-history contract, but without a second consumer it would prematurely freeze shared accounting semantics that differ here (completed UTC days, routed models, reasoning-token inclusion, management-key scope).

I also fixed the two concrete hosted-Xcode regressions exposed by the previous run at 65327296f8850f00b96ce5885431f244b716ae43. I will request a fresh review after the exact-head CI is terminal. Pete: the remaining requested decision is whether this narrowly tested first-party plugin exception is acceptable as the durable boundary.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 8, 2026
@clawsweeper clawsweeper Bot removed the rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. label Aug 8, 2026
@clawsweeper clawsweeper Bot removed the rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. label Aug 9, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2194fc72c4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread Sources/CodexBar/ShareStatsPayload.swift Outdated
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. merge-risk: 🚨 auth-provider 🚨 Merging this PR could break OAuth, tokens, provider routing, model choice, or credentials. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 9, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 9, 2026
@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. and removed merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 9, 2026
@clawsweeper clawsweeper Bot removed the merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. label Aug 11, 2026
@Chipagosfinest

Copy link
Copy Markdown
Contributor Author

@Yuxin-Qiao this is ready for final review/merge when you have a moment.

Exact head 6e7b6178a3cc2ed86e1e9490b7e551b1eed997a8 is merged with current main, all 9 CI checks are green, GitHub reports MERGEABLE / CLEAN, and all 10 review threads are resolved.

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. label Aug 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature: ✨ showcase ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. P2 Normal priority bug or improvement with limited blast radius. proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant