Skip to content

Add opt-in ccusage fallback for incomplete Codex history - #2817

Open
Yoroin wants to merge 3 commits into
steipete:mainfrom
Yoroin:feat/codex-ccusage-fallback
Open

Add opt-in ccusage fallback for incomplete Codex history#2817
Yoroin wants to merge 3 commits into
steipete:mainfrom
Yoroin:feat/codex-ccusage-fallback

Conversation

@Yoroin

@Yoroin Yoroin commented Aug 9, 2026

Copy link
Copy Markdown

Summary

  • Add an opt-in Codex-only ccusage bridge for native history scans whose coverage is not yet established.
  • Preserve multi-account isolation by forwarding the selected CODEX_HOME.
  • Keep native results on missing helpers, timeouts, invalid JSON, non-zero exits, or lower-token fallback output.
  • Add optional universal helper packaging, architecture checks, signing, tests, and MIT provenance documentation.

Changes after review

  • Preserve native completeness independently: fallback results now keep historyCoverageIsEstablished=false while exposing historyFallbackCoverageIsEstablished=true.
  • Dashboard and CLI can use fallback coverage without claiming that the native archive catch-up has completed.
  • Packaged helpers now require an explicit version and SHA-256 digest, verify the digest before signing, and record verified provenance beside the helper.
  • Added regression coverage for native/fallback flags, dashboard behavior, CLI JSON, and digest mismatch handling.

Validation

  • CCUsageCodexBridgeTests: 6/6 passed.
  • CostUsageFetcherTests: 15/15 passed.
  • Targeted CLI and dashboard tests passed.
  • Package helper tests passed.
  • Shell syntax, git diff check, and SwiftFormat lint passed.
  • Real-archive smoke run completed against a real local Codex archive with ccusage 20.0.19. The uploaded evidence is intentionally redacted:
    • provider=codex
    • historyCoverageIsEstablished=false
    • historyFallbackCoverageIsEstablished=true
    • dailyDays=2
    • fallback totals present=true
  • No credentials, account identifiers, session identifiers, prompts, raw logs, or local filesystem paths are included here.

Design notes

  • CodexBar does not download the helper or search PATH. Use CODEXBAR_CCUSAGE_PATH locally or CODEXBAR_CCUSAGE_SOURCE when packaging.
  • Native catch-up status remains authoritative; the fallback is only used after an incomplete native scan and a successful, non-regressive ccusage result.
  • This PR remains Draft so maintainers can decide whether the preferred long-term direction is this optional bridge, native semantic-stall recovery, or both.

Related to #2815.

@clawsweeper

clawsweeper Bot commented Aug 9, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal priority bug or improvement with limited blast radius. merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. labels Aug 9, 2026
@clawsweeper

clawsweeper Bot commented Aug 9, 2026

Copy link
Copy Markdown

Codex review: found issues before merge. Reviewed August 10, 2026, 9:02 AM ET / 13:02 UTC.

ClawSweeper review

What this changes

This PR adds an opt-in ccusage executable fallback for incomplete Codex history scans, plus separate fallback-coverage signals for the dashboard and CLI and optional helper packaging.

Merge readiness

⚠️ Needs maintainer review before merge - 9 items remain

Keep open: the real-archive evidence supports the incomplete-history scenario, but the branch still accepts aggregate-only fallback comparisons that can regress individual days and needs a maintainer decision on the executable-fallback boundary. Likely related people: steipete (high confidence) and xx205 (medium confidence).

Priority: P2
Reviewed head: 0a9c393f8cdf1a331a04366df84dd3a24c6ad7c5
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦐 gold shrimp (3/6) The branch has credible real behavior proof and broad focused tests, but the remaining daily-regression and compatibility blockers prevent a merge-ready rating.
Proof confidence 🐚 platinum hermit (4/6) Sufficient (live_output): The PR supplies redacted after-fix live output from a real local Codex archive showing fallback coverage while native coverage remains incomplete.
Patch quality 🦐 gold shrimp (3/6) Security review found an item that needs attention.

Verification

Check Result Evidence
Real behavior Verified Sufficient (live_output): The PR supplies redacted after-fix live output from a real local Codex archive showing fallback coverage while native coverage remains incomplete.
Evidence reviewed 8 items Current main remains native-only: Current main loads the local scan result directly into the snapshot and has no ccusage bridge or fallback-coverage field, so the central capability is not already implemented or released.
Aggregate check permits daily regression: The bridge accepts a fallback solely when its total tokens are at least the native total; it does not verify each native day remains present and non-regressive.
Accepted fallback replaces native daily report: After acceptance, the fetcher uses the fallback as the Codex daily report, making an omitted or lower fallback day visible to the dashboard and CLI.
Findings 3 actionable findings [P1] Reject fallbacks that regress an individual day
[P2] Require exact helper architecture matches
[P2] Use clearly synthetic model identifiers in fixtures
Security Needs attention Constrain the executable trust boundary: A chosen helper inherits CodexBar's process environment and receives the selected CODEX_HOME; package-time SHA-256 validation does not apply to a runtime path override.

How this fits together

CodexBar reduces local Codex archive data into usage snapshots for the Spend Dashboard and codexbar cost. When native catch-up is incomplete, this branch can invoke a supplied ccusage executable and publish its coverage separately.

flowchart LR
A[Codex archive] --> B[Native history scan]
B --> C{Native coverage complete?}
C -->|Yes| D[Usage snapshot]
C -->|No| E[Optional ccusage executable]
E --> F[Fallback validation]
F --> D
D --> G[Spend Dashboard]
D --> H[CLI cost JSON]
Loading

Decision needed

Question Recommendation
Should CodexBar ship an opt-in external ccusage executable fallback for incomplete native history, or prioritize native semantic-stall recovery without this executable path? Adopt a bounded fallback: Keep the opt-in bridge after the daily-preservation and packaging defects are repaired, with its trust boundary documented as an intentional product contract.

Why: This introduces a new environment/package surface and a deliberate local-code-execution boundary; source review cannot determine the preferred long-term product and security contract.

Before merge

  • Reject fallbacks that regress an individual day (P1) - The total-only comparison accepts a fallback that is larger overall but omits or lowers a native daily entry, and the fetcher then replaces the native report with it. Compare each overlapping day or preserve native rows through a defined merge, and add that regression case.
  • Require exact helper architecture matches (P2) - The new helper install path relies on a substring test in verify_binary_arches; for example, arm64e x86_64 passes a requested arm64 x86_64 set. Tokenize and compare architecture names exactly, then cover the near-match rejection.
  • Use clearly synthetic model identifiers in fixtures (P2) - These fixtures retain plausible gpt-5.6-* model identifiers despite the repository policy and the outstanding review request. Use clearly fictitious fixture names or an explicit test pricing seam so tests cannot expose unverified model names.
  • Resolve security concern: Constrain the executable trust boundary - A chosen helper inherits CodexBar's process environment and receives the selected CODEX_HOME; package-time SHA-256 validation does not apply to a runtime path override.
  • Resolve merge risk (P2) - A fallback with a larger aggregate total can still omit or lower a native day, causing visible history regression after it replaces the native report.
  • Resolve merge risk (P1) - The new executable path runs a selected binary with CodexBar's environment and account-scoped archive location; maintainers need to explicitly accept that opt-in trust boundary.
  • Resolve merge risk (P2) - The helper architecture check can accept a similarly named but unsupported slice, leaving a packaged fallback unusable on a target architecture.
  • Complete next step (P2) - A maintainer must choose the executable-fallback product and security direction; the present branch also has concrete pre-merge fixes.

Findings

  • [P1] Reject fallbacks that regress an individual day — Sources/CodexBarCore/CostUsage/CCUsageCodexBridge.swift:292-299
  • [P2] Require exact helper architecture matches — Scripts/package_app.sh:421
  • [P2] Use clearly synthetic model identifiers in fixtures — Tests/CodexBarTests/CCUsageCodexBridgeTests.swift:81-84
  • [medium] Constrain the executable trust boundary — Sources/CodexBarCore/CostUsage/CCUsageCodexBridge.swift:80
Agent review details

Security

Needs attention: The deliberately opt-in helper execution path is a new security boundary that needs maintainer acceptance, although no hidden download or dependency-resolution path was found.

Review metrics

Metric Value Why it matters
Change surface 17 files; production +516/-19, tests +466, docs +37 The fallback spans runtime accounting, CLI and dashboard contracts, packaging, and a third-party executable boundary.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #2815
Summary: This PR is a candidate implementation for the reported incomplete Codex-history behavior.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

  1. Preserve daily data and validate exact architectures (recommended)
    Require fallback acceptance to preserve every native day and make helper-architecture matching exact, with focused regression coverage before merge.
  2. Explicitly accept the helper trust boundary
    If retaining runtime executable overrides, document that the chosen binary receives CodexBar's environment and scoped archive access as an intentional opt-in contract.
  3. Pause the executable fallback
    Keep the native catch-up path as the only supported recovery route if maintainers do not want this new executable boundary.

Technical review

Best possible solution:

Choose whether CodexBar should support an executable fallback, then require per-day non-regression, exact architecture matching, and policy-compliant model fixtures before merging it.

Do we have a high-confidence way to reproduce the issue?

Yes for the remaining defect: a fallback whose total exceeds native totals while omitting or lowering one native day passes the aggregate check and replaces the report. The supplied real-archive smoke additionally demonstrates the broader fallback path.

Is this the best way to solve the issue?

No. The approach needs per-day non-regression and exact architecture validation, and maintainers must first choose whether the executable fallback is the desired product direction.

Full review comments:

  • [P1] Reject fallbacks that regress an individual day — Sources/CodexBarCore/CostUsage/CCUsageCodexBridge.swift:292-299
    The total-only comparison accepts a fallback that is larger overall but omits or lowers a native daily entry, and the fetcher then replaces the native report with it. Compare each overlapping day or preserve native rows through a defined merge, and add that regression case.
    Confidence: 0.99
  • [P2] Require exact helper architecture matches — Scripts/package_app.sh:421
    The new helper install path relies on a substring test in verify_binary_arches; for example, arm64e x86_64 passes a requested arm64 x86_64 set. Tokenize and compare architecture names exactly, then cover the near-match rejection.
    Confidence: 0.94
  • [P2] Use clearly synthetic model identifiers in fixtures — Tests/CodexBarTests/CCUsageCodexBridgeTests.swift:81-84
    These fixtures retain plausible gpt-5.6-* model identifiers despite the repository policy and the outstanding review request. Use clearly fictitious fixture names or an explicit test pricing seam so tests cannot expose unverified model names.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.97

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against b093129a4f94.

Labels

Label justifications:

  • P2: This is a bounded but user-visible Codex usage-reporting feature with source-backed merge blockers.
  • merge-risk: 🚨 compatibility: The accepted fallback can replace native daily history and the packaging check can accept an incompatible architecture slice.
  • merge-risk: 🚨 security-boundary: The feature adds an opt-in executable invocation that inherits the app environment and selected CODEX_HOME.
  • rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🐚 platinum hermit and patch quality is 🦐 gold shrimp.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR supplies redacted after-fix live output from a real local Codex archive showing fallback coverage while native coverage remains incomplete.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR supplies redacted after-fix live output from a real local Codex archive showing fallback coverage while native coverage remains incomplete.

Evidence

Security concerns:

  • [medium] Constrain the executable trust boundary — Sources/CodexBarCore/CostUsage/CCUsageCodexBridge.swift:80
    A chosen helper inherits CodexBar's process environment and receives the selected CODEX_HOME; package-time SHA-256 validation does not apply to a runtime path override.
    Confidence: 0.95

What I checked:

Likely related people:

  • steipete: Git shortlog shows Peter Steinberger as the dominant contributor across the central fetcher, pricing, and packaging paths; current fetcher lines also blame to the v0.49.1 update. (role: long-running usage-scanner and packaging contributor; confidence: high; commits: ae1111e39912, b093129a4f94; files: Sources/CodexBarCore/CostUsageFetcher.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsagePricing.swift, Scripts/package_app.sh)
  • xx205: The release history credits xx205 for the SQLite Codex-history and append-linear catch-up work that this fallback would supplement. (role: adjacent native catch-up contributor; confidence: medium; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner+CacheHelpers.swift, CHANGELOG.md)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Preserve native daily entries when a fallback is incomplete or lower for any day.
  • Use exact architecture matching and cover a near-match rejection.
  • Replace plausible model fixture identifiers with clearly synthetic values.

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)
  • reviewed 2026-08-09T16:36:48.393Z sha ba573b6 :: needs real behavior proof before merge. :: [P1] Keep native scan completeness independent from fallback data | [P2] Verify the packaged helper's immutable provenance
  • reviewed 2026-08-09T17:20:42.062Z sha ba573b6 :: needs real behavior proof before merge. :: [P1] Preserve the native coverage state | [P2] Pin the packaged ccusage artifact
  • reviewed 2026-08-10T01:34:40.628Z sha dbcb003 :: needs maintainer review before merge. :: none
  • reviewed 2026-08-10T06:49:04.698Z sha dbcb003 :: found issues before merge. :: [P1] Preserve native daily entries before accepting fallback data

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels Aug 9, 2026
@Yoroin

Yoroin commented Aug 10, 2026

Copy link
Copy Markdown
Author

@clawsweeper re-review

Updated the PR body with redacted real-archive smoke evidence and pushed the follow-up fixes for native coverage separation and version/SHA-256 helper provenance.

@clawsweeper

clawsweeper Bot commented Aug 10, 2026

Copy link
Copy Markdown

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Aug 10, 2026
@Yoroin
Yoroin marked this pull request as ready for review August 10, 2026 06:46
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Aug 10, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dbcb0035ed

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

"cacheCreationTokens": 5,
"totalTokens": 955,
"totalCost": 1.25,
"modelsUsed": ["gpt-5.6-sol"],

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use clearly fictitious test model names

This fixture introduces gpt-5.6-sol (and repeats it in the helper fixture below), which looks like a plausible future model identifier rather than an obviously fake placeholder; please switch these new test values to something clearly synthetic such as test-model so fixtures cannot be mistaken for unreleased model names.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 Merging this PR could break existing users, config, migrations, defaults, or upgrades. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant