perf: read verbs skip ingest on read; index bounded ts windows (all reads <100ms)#479
Conversation
Two changes that put every `burn` read command under 100ms on a large ledger (measured on a ~614MB burn.sqlite, `--since 24h`): 1. `summary` and `hotspots` no longer run a synchronous full-store `ingest_all` sweep before querying. That sweep re-stats every session file under every harness store (~44k+ OpenCode session files here), costing seconds. Both verbs now take an opt-in `--ingest` flag and are read-only by default, matching the existing `burn compare` precedent. Freshness stays the job of `burn ingest --watch` / the Claude Stop hook. 2. `query_turns` forces `idx_turns_ts` for a bounded `ts` window with no `session_id` filter. The statement ends in `ORDER BY rowid`, which made SQLite prefer a full `SCAN` of all ~97k rows even for a 24h window; the index seek + an in-memory rowid sort of the small matched set is far cheaper. Output is byte-identical (same rows, same order) — gated behind the new `TableFilters.prefer_ts_index`, set only by the full-window `query_turns` (the session-IN path and non-`turns` tables opt out). Results (median): hotspots 3088->76ms, summary 3298->62ms, compare 50->16ms, overhead 49->13ms. sessions/stamps/flow already ~4-5ms. All command golden snapshots remain byte-identical. Adds `scripts/bench-summary.py` (`pnpm run bench:summary`) as a timing gate, and unit/smoke tests for both changes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
|
Warning Review limit reached
More reviews will be available in 2 minutes and 12 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bae2989ae9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| stdout=subprocess.DEVNULL, | ||
| stderr=subprocess.DEVNULL, | ||
| env=env, | ||
| check=False, |
There was a problem hiding this comment.
When the measured burn subprocess exits non-zero (for example with a corrupt ledger, invalid --summary-args, or a CLI regression), check=False still records the elapsed time while stdout/stderr are discarded, so a fast failure can be included in the samples and make the budget gate report PASS. Check the return code (or use check=True) before accepting a sample.
Useful? React with 👍 / 👎.
Two causes, both fixed: 1. Stale ledger. `burn summary` is read-only now (#479) and the background `ingest --watch` doesn't keep up (manual ingest pulled in 3290 backlogged turns), so the polled totals didn't move. Replace the watch with a one-shot incremental `burn ingest` at the top of each poll (BurnLedger.ingest()) — a warm sweep is ~1–3s, so the poll cadence is now 2.5s. 2. Negative/clamped rate. The rate was the delta between two trailing-5-min window totals, which dips negative as old turns age out of the window and was clamped to 0 — reading as "no usage". Replace with a moving average: each poll queries the trailing 60s and divides, which is non-negative and robust to the window sliding and to late ingests. The cumulative line is now the integral of that rate (a monotonic session total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#482) * macOS: Settings cog (appearance + quit), move tab toggle into the top bar - Move the Usage/Live segmented toggle up into the header row, next to the Codex/Claude provider toggle. - Add a gear (settings) button on the right of that bar. It toggles a Settings tab with: - Appearance — Light / Dark / System (follow system), persisted in the "appearance" UserDefault and applied to the whole popover (chrome + content) by AppDelegate via popover.appearance, live as it changes. - Quit Burn (the global ⌘Q still works too). - Picking Usage/Live also dismisses Settings. This re-lands the settings work that never reached main (it was committed to the live-burn branch after #480 merged). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix live burn monitor showing no usage Two causes, both fixed: 1. Stale ledger. `burn summary` is read-only now (#479) and the background `ingest --watch` doesn't keep up (manual ingest pulled in 3290 backlogged turns), so the polled totals didn't move. Replace the watch with a one-shot incremental `burn ingest` at the top of each poll (BurnLedger.ingest()) — a warm sweep is ~1–3s, so the poll cadence is now 2.5s. 2. Negative/clamped rate. The rate was the delta between two trailing-5-min window totals, which dips negative as old turns age out of the window and was clamped to 0 — reading as "no usage". Replace with a moving average: each poll queries the trailing 60s and divides, which is non-negative and robust to the window sliding and to late ingests. The cumulative line is now the integral of that rate (a monotonic session total). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
Summary
Every
burnread command now returns in under 100ms on a large ledger. Two independent costs were dominating, both addressed here.1. Read verbs ran a synchronous full-store ingest sweep
summaryandhotspotscalledingest_allbefore querying. That sweep re-stats every session file under every harness store (~44k+ OpenCode session files on the test ledger) on each invocation — seconds of wall-clock, ~99% of the command's time.Fix: both verbs are read-only by default with an opt-in
--ingestflag, matching the existingburn compareprecedent. Freshness stays the job ofburn ingest --watch(FS-event driven) or the Claude Stop hook. When the sweep is skipped, an emptyIngestReportkeeps the banner / JSONingestblock byte-identical to a no-op sweep.2. Windowed turn queries full-scanned the table
query_turnsrunsSELECT record_json FROM turns WHERE ts >= ? ORDER BY rowid. The ledger hasidx_turns_ts, butORDER BY rowidmade the planner pick a fullSCAN turns— reading all ~97k rows to return the ~1,123 in a 24h window (EXPLAIN QUERY PLANconfirmed).Fix: force
INDEXED BY idx_turns_tsfor a boundedtswindow when nosession_idfilter is present (newTableFilters.prefer_ts_index, set only by the full-windowquery_turns; the session-IN path ridesidx_turns_sessionand opts out, as do non-turnstables). The matched set is sorted by rowid in memory, so output is byte-identical (same rows, same order).Results
Median, release binary, live ~614MB
burn.sqlite,--since 24h:hotspotssummarycompareoverheadstate statussessions/stamps/flowCorrectness
BURN_GOLDENcommand snapshots still pass byte-for-byte (summary, hotspots, overhead, compare, flow) — the index change only alters the access path.summary/hotspotsingest gate) +reader.rs::build_select_sql_tests(index-hint conditions).cargo fmt/clippy -D warningsclean; full workspace suite green.BURN_GOLDEN-gatedstate/state-jsonsnapshots pinschema version: 5whileSCHEMA_VERSIONis6— worth a separate snapshot refresh.Tooling
Adds
scripts/bench-summary.py(pnpm run bench:summary) — times the release binary end-to-end against a live ledger and exits non-zero over a budget, so it doubles as a regression gate.Behavior change to flag
summary/hotspotsno longer auto-freshen on read. Consumers that relied on that (e.g. pollingsummary --json) should runburn ingest --watchor pass--ingest.🤖 Generated with Claude Code