fix: keep Codex fork catch-up progress across appends

[xx205]

Summary

• Resume an appended unresolved-fork rollout from its validated cached offset instead of restarting
  a bounded full scan at byte zero.
• Charge only the appended suffix against per-file and per-refresh budgets, while preserving compact
  fork events and merging newly appended token snapshots.
• Restrict that resume path to ordinary unresolved forks; appended subagent buffers still take the
  correctness-preserving full-rescan path.
• Preserve unchanged same-size buffered retries as zero disk work for both ordinary forks and
  subagents, so an exhausted refresh budget cannot defer a pure in-memory replay.
• Keep a temporarily unresolved dependency key unset so a parent discovered in the same refresh can
  trigger the existing in-memory child retry.
• Accept the immediate predecessor parser producer because the cache schema and stored token-delta
  semantics are unchanged, avoiding a one-time 20+ GiB rebuild just to adopt this fix.

Problem

#2525 made oversized Codex histories accurate and resumable, and persisted compact unresolved-fork
events. One completed-fork append path was still too restrictive, however. Its in-memory retry was
eligible only when the cached parent dependency key was nil and the cached offset already
equaled the current file size.

As soon as an active fork appended, its old offset was smaller than the new size. A stable
missing-parent lookup could also produce a non-nil dependency key. Either condition disabled the
fast path. Ordinary append parsing intentionally excludes forks, so the scanner fell back to a
correctness-preserving bounded full rescan from byte zero.

That fallback is safe for accounting, but it replaces a later completed offset with the first
bounded slice. On a large active rollout this makes Spend Dashboard progress move backward and can
re-read the same prefix after later appends. Finish now makes the rollback more visible because it
runs catch-up passes continuously; it does not itself clear the cache.

There was a related transition bug when a previously missing parent appeared. If the first replay
ran before the newly discovered parent had been scanned, retaining the old missing-parent dependency
key suppressed the post-parent retry and left the child's buffered usage unpublished.

The appended-subagent safety boundary also needs a distinct same-size case. A fully scanned
subagent can be retried before its queued parent is available without reading JSONL, because the
unchanged file cannot contain new lineage. Treating that retry like an appended subagent reports the
whole file as pending work and can defer the in-memory replay behind an exhausted byte budget.

Implementation

Validated append resume

A completed buffered fork can resume at its previous offset only when all of these checks pass:

• the cache contains a fork ID and non-empty compact unresolved-fork events;
• the previous scan completed and has no partial JSONL-record resume state;
• the current file identity matches the cached identity;
• the previous parsed offset is positive, not past EOF, and equals the token-index anchor offset;
• the existing 64 KiB prefix-boundary anchor still matches.

The scanner then replays compact events in memory and reads only the suffix. Appended token snapshots
are merged with the existing snapshot index. A same-size dependency retry preserves the existing
snapshots because it performs no disk read.

The validation is split by operation: an unchanged same-size replay accepts either ordinary-fork or
subagent buffers, while an appended suffix resume requires an empty subagent buffer. Any failed
validation keeps the existing full-rescan behavior. In particular, a completed subagent rollout
that may be reclassified by newly appended lineage metadata still receives a full rescan.

Work accounting and parent transition

pendingCodexScanWorkBytes now reports currentSize - cachedOffset for a validated buffered fork,
so the suffix, not the whole file, is admitted against the 256 MiB / 512 MiB limits.
For a validated same-size buffered retry it reports zero, allowing the pure in-memory replay even
when the refresh byte budget is already exhausted.

During buffered replay, a newly unresolved dependency now remains nil instead of inheriting the old
missing-parent key. Once the queued parent finishes scanning, the existing same-refresh retry can
publish the child's compact events without rereading its JSONL.

Cache compatibility

The generated parser hash still rotates. The immediately preceding producer
codex:cu:p843ca061c36bbea1 is explicitly compatible because this change adds a resume path but does
not change the cache layout or token-delta semantics. Older pre-interleave-containment producers
remain incompatible.

Privacy-safe validation

A current local cache was searched without emitting identifiers for one completed ordinary fork
with a valid prefix anchor, a non-empty unresolved-fork buffer, no subagent buffer, and real bytes
appended after its cached offset. Exactly one entry matched. Its rollout and single cache entry were
copied into an isolated temporary root; the local-only harness and clone were removed from the
validation workspace afterward.

REAL_APPEND_PROOF source=real-local-rollout-copy
REAL_APPEND_PROOF cached_completed_offset=763410606
REAL_APPEND_PROOF file_size_after_append=773963499
REAL_APPEND_PROOF appended_bytes=10552893
REAL_APPEND_PROOF admitted_work=10552893
REAL_APPEND_PROOF after_fix_processed_offset=773963499
REAL_APPEND_PROOF after_fix_scan_complete=true
REAL_APPEND_PROOF warm_retry_offset=773963499
REAL_APPEND_PROOF warm_retry_stable=true
REAL_APPEND_PROOF privacy=paths-ids-dates-models-tokens-content-redacted

The scan slice was only 64 KiB larger than the 10,552,893-byte suffix and therefore could not cover
the 773,963,499-byte full file. Admitting exactly the suffix and reaching EOF proves the fixed path
resumed from the cached offset. An unchanged warm retry preserved the completed offset.

As supporting observation, the CodexBar 0.47.0 process did not restart while one account-scoped
cache indexing the same 20+ GiB corpus reported:

1,415,520,432 -> 823,980,119 -> 1,082,720,169 processed bytes

The exact regression case is synthetic and checked in as CodexForkAppendResumeTests:

Field	Before fix	After fix
Initial completed file	18,390 B	18,390 B
Appended suffix	391 B	391 B
Refresh budget	512 B	512 B
parsedBytes after append	452 B	18,781 B
codexScanComplete	false	true
token snapshots	0	2
pending work after append	18,781 B	391 B

Because the 512-byte budget cannot cover the 18,390-byte prefix, reaching 18,781 bytes proves that
the fixed path resumed from the cached offset rather than rereading the file.

The adjacent subagent boundary regression starts with a completed buffered subagent larger than the
512-byte budget. With the file unchanged and the dependency temporarily unresolved, pending work is
zero and an already exhausted budget still admits .allow(0). After appending lineage and token
events, pending work becomes the entire file and the bounded refresh remains incomplete at or below
512 bytes. This proves the same-size retry is free without weakening the appended-subagent full
rescan rule.

Verification

• make check
  • parser hash is current;
  • repository/package/locale checks passed;
  • SwiftFormat: 0 files require formatting;
  • SwiftLint: 0 violations in 1,762 files.
• Focused CostUsage validation:
  • 142 tests in 5 suites passed;
  • includes cache compatibility, performance gates, fork/subagent integration, scanner breakdown,
    missing-parent appearance, the append regression, and the subagent full-rescan boundary.
• GPT-5.6 pricing isolation check:
  • 2 tests passed when CFFIXED_USER_HOME pointed to a fresh temporary directory;
  • without isolation, those tests read the developer machine's live models.dev cache and fail
    four bundled-price assertions. The pricing production and test files are unchanged from base.
• Full make test with an isolated user cache and at most two Swift compile jobs:
  • 813/813 selections in 68/68 groups passed on the first attempt;
  • 0 failed groups, retries, or timeouts;
  • 1,361.4 seconds total (134.6 seconds discovery, 1,226.8 seconds execution).

Scope

• No UI strings, controls, accounting formulas, or catch-up duty-cycle settings change.
• This fixes repeated prefix work and progress rollback inside each cache. The default cache and an
  account-scoped Dashboard cache can still scan the same corpus independently; cross-cache work
  sharing is a separate optimization.
• Append reuse relies on Codex rollout files being append-only. File replacement, truncation, or a
  failed boundary anchor continues to invalidate the resume path.

Screenshots

Not applicable. This is a scanner/cache correctness and performance fix with no UI change; the
privacy-safe validation is represented by deterministic cache-state assertions.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 4, 2026, 3:59 PM ET / 19:59 UTC.

ClawSweeper review

What this changes

The PR resumes validated ordinary unresolved-fork scans from their cached offset after appends while keeping appended subagent buffers on full bounded rescans.

Regression provenance

Possible regression — probable (reproduction; reviewed change; known regression link). No predecessor PR is attributed.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

The final head resolves the prior subagent retry finding and retains guarded fallbacks, so this remains a strong merge candidate. Likely related person: xx205 (high confidence).

Priority: P1
Reviewed head: 72f2d12384d4d98cf7544002129ee2152a2781c0

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	The focused repair has strong real behavior evidence, scoped cache safeguards, resolved prior blockers, and green supplied checks.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (live_output): The PR includes redacted after-fix output from a real local rollout copy showing suffix-sized admission and EOF completion, with deterministic regression tests as supplemental proof.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (live_output): The PR includes redacted after-fix output from a real local rollout copy showing suffix-sized admission and EOF completion, with deterministic regression tests as supplemental proof.
Evidence reviewed	5 items	Final subagent boundary: The final commit validates same-size buffered replays separately from appended ordinary-fork resumes; only the latter permits suffix parsing, while a nonempty subagent buffer disqualifies the append path. Budget and fallback handling: Pending work is zero only for a fully validated same-size replay, suffix-sized only for an eligible ordinary-fork append, and otherwise charges a full scan; rejected incremental parsing releases the reservation before the full-rescan fallback. Focused regression coverage: New tests cover suffix-only admission for a completed unresolved fork and confirm that unchanged subagent retries consume zero budget while appended subagents force a bounded full rescan.
Findings	None	None.
Security	None	None.

How this fits together

CodexBar scans local Codex rollout JSONL files into a persisted usage cache for the Spend Dashboard. Cached fork metadata decides whether a refresh can replay buffered events, scan only an appended suffix, or perform a bounded full rescan.

flowchart LR
A[Codex rollout JSONL] --> B[Persisted usage cache]
B --> C[Identity and prefix validation]
C --> D{File state}
D -->|Same size| E[Memory-only fork replay]
D -->|Ordinary fork append| F[Read appended suffix]
D -->|Subagent append or invalid cache| G[Bounded full rescan]
E --> H[Usage cache]
F --> H
G --> H
H --> I[Spend Dashboard]

Loading

Before merge

• Resolve merge risk (P1) - Existing caches from the immediately preceding producer are deliberately reused; this relies on the reviewed claim that the persisted layout and token-delta semantics did not change.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	6 files affected; production +84/-23, tests +301	The implementation changes persisted scanner-resume and cache-compatibility behavior with focused regression coverage.

Merge-risk options

Maintainer options:

1. Merge the guarded resume path (recommended)
   Accept the immediately preceding cache producer because the branch retains identity and anchor validation and falls back to a bounded full rescan whenever validation or the subagent boundary fails.
2. Require an upgrade-cache fixture first
   Add a focused fixture containing a prior-producer fork cache before merging if direct serialized-cache upgrade coverage is preferred.

Technical review

Best possible solution:

Merge the guarded resume path with its regression coverage and retain the invalid-cache, truncation, replacement, and appended-subagent full-rescan fallbacks.

Do we have a high-confidence way to reproduce the issue?

Yes. The checked-in bounded-budget tests create a completed unresolved fork, append a suffix too small to reread its prefix, and verify EOF completion; they also verify the same-size and appended-subagent split.

Is this the best way to solve the issue?

Yes. The branch reuses only an unchanged cached prefix after file identity and anchor checks, charges only newly appended bytes, and preserves the existing full-rescan path when lineage can change.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a82f509ea8e7.

Labels

Label changes:

• add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

• P1: The patch repairs a Codex usage-refresh regression that can repeatedly scan large local rollout prefixes and make catch-up progress move backward.
• merge-risk: 🚨 compatibility: The patch changes reuse rules for persisted Codex caches produced by the immediately preceding parser hash.
• merge-risk: 🚨 availability: An incorrect resume eligibility decision could defer refreshes under byte budgets or trigger costly repeated rescans.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR includes redacted after-fix output from a real local rollout copy showing suffix-sized admission and EOF completion, with deterministic regression tests as supplemental proof.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR includes redacted after-fix output from a real local rollout copy showing suffix-sized admission and EOF completion, with deterministic regression tests as supplemental proof.

Evidence

What I checked:

• Final subagent boundary: The final commit validates same-size buffered replays separately from appended ordinary-fork resumes; only the latter permits suffix parsing, while a nonempty subagent buffer disqualifies the append path. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner+CacheHelpers.swift:1061, 72f2d12384d4)
• Budget and fallback handling: Pending work is zero only for a fully validated same-size replay, suffix-sized only for an eligible ordinary-fork append, and otherwise charges a full scan; rejected incremental parsing releases the reservation before the full-rescan fallback. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift:4253, 72f2d12384d4)
• Focused regression coverage: New tests cover suffix-only admission for a completed unresolved fork and confirm that unchanged subagent retries consume zero budget while appended subagents force a bounded full rescan. (Tests/CodexBarTests/CodexForkAppendResumeTests.swift:8, 72f2d12384d4)
• Feature provenance: The bounded fork scanner originated in the merged related change, which is the direct history context for this focused append-resume repair. (Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner+CacheHelpers.swift:1111, 2920019bc16d)
• Current-main and release check: The open PR head is neither an ancestor of current main nor of v0.47.0, so the requested behavior is not already implemented or released. (a82f509ea8e7)

Likely related people:

• xx205: The merged bounded fork-scanner change and this PR’s three commits are by xx205. (role: introduced the bounded fork-scanning feature and authored the follow-up; confidence: high; commits: 2920019bc16d, f1dbd40d0d2e, 54448a5f7817; files: Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner+CacheHelpers.swift, Sources/CodexBarCore/Vendored/CostUsage/CostUsageScanner.swift)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (8 earlier review cycles)

• reviewed 2026-08-04T12:33:46.808Z sha f1dbd40 :: needs real behavior proof before merge. :: [P1] Keep buffered subagent rollouts on the full-rescan path
• reviewed 2026-08-04T12:43:14.354Z sha f1dbd40 :: needs real behavior proof before merge. :: [P1] Keep buffered subagent rollouts on the full-rescan path
• reviewed 2026-08-04T15:02:24.864Z sha f1dbd40 :: needs real behavior proof before merge. :: [P1] Preserve full rescans for buffered subagent rollouts
• reviewed 2026-08-04T16:02:51.778Z sha 54448a5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-04T16:22:36.989Z sha 54448a5 :: needs maintainer review before merge. :: none
• reviewed 2026-08-04T16:30:00.915Z sha 54448a5 :: needs changes before merge. :: [P2] Preserve zero-cost same-size subagent retries
• reviewed 2026-08-04T16:54:08.939Z sha 54448a5 :: needs changes before merge. :: [P2] Preserve zero-cost same-size subagent retries
• reviewed 2026-08-04T18:16:09.607Z sha 72f2d12 :: needs maintainer review before merge. :: none

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 54448a5f78

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve zero-cost same-size subagent retries

When an unresolved subagent is fully rescanned before its newly queued parent, it is left with a nil dependency key and scheduled for a same-refresh, in-memory retry. This branch now recognizes only ordinary unresolved-fork buffers, so pendingCodexScanWorkBytes reports the subagent's entire file size instead of zero; after the preceding scans consume the refresh budget, that retry is deferred, and a later refresh can consume up to maxFileBytes despite rereading no JSONL. Preserve the validated zero-work case for same-size hasBufferedCodexForkRetryLines while using the suffix calculation only for ordinary fork appends.

Useful? React with 👍 / 👎.

[steipete]

Landed. Verification before merge (independent maintainer-agent review, line-by-line with a security lens):

• Truncation/rewrite safety: the append-resume path requires inode match, completed prior scan, no partial-resume state, valid offset bounds, token-anchor offset equality, and a SHA-256 match of the 64 KiB window ending at the cached offset. Truncate-then-regrow and in-place rewrites fail the hash and fall back to the bounded full rescan — strictly more validation than the old same-size retry path.
• Budget accounting: admission charges exactly size - offset for a validated resume using the same validators as the scan path, so admission and actual work cannot disagree. The checked-in test proves a 512-byte budget still reaches EOF on an 18 KiB-prefix file via resumption.
• Subagent boundary: append resume requires an empty subagent buffer; appended subagent files keep the full correctness-preserving rescan (test-pinned).
• Producer whitelist of the predecessor cache key verified justified by inspection: no stored schema or token-delta semantics change, spares large corpora a full rebuild.
• Parser hash reproduces deterministically via Scripts/regenerate-codex-parser-hash.sh.
• Local: 183 scanner/cache tests + broader CostUsage sweep of 483 tests in 38 suites, 0 failures; make check clean. CI green at merge.