fix(server): paginate large thread history to stop the server running out of memory

[olafura]

Fixes the server-side root cause behind #2761 (and likely #996).

What Changed

The server used to load a thread's entire history — every tool activity, message, and checkpoint — into memory at once. On a busy database that's hundreds of MB, which exhausted the Node heap and crashed the server (and made large threads slow to load / churn reconnects, as reported in #2761).

Now the server loads only the most recent slice of a thread's activity and tells the client whether older history exists; the web and mobile apps fetch older pages on demand as you scroll up. Deep history is still fully reachable — it's paged in instead of loaded all at once.

• Server: the snapshot endpoint and the t3 CLI offline path stop loading every thread's full history (they only read the project list). Thread detail is bounded to the most recent activities, with a hasMoreActivities flag and a cursor-paginated getThreadActivities RPC for older pages. The older-page query is shaped to use the existing index instead of a full sort.
• Reconnect catch-up: shell/thread subscriptions replay only a recent gap (up to 1,000 global events); older cached cursors receive a fresh projection snapshot instead of traversing the full event log.
• Web / Mobile: lazy-load older history on scroll-up — prepend older pages, anchored so the viewport doesn't jump.

The commits remain separated by server, web, mobile, shared-client, reconnect-safety, and rebase-formatting concerns so each layer is independently reviewable.

Why

Programs in a thread emit a lot of tool activity over time. The server treated "open a thread" (and "load the snapshot") as "read all of it from SQLite into memory." For a long-lived thread or a busy database, that single read materialised more than the heap could hold, so the process ran out of memory and died — and even when it didn't crash, shipping one giant snapshot blocked the connection (#2761). Bounding the read + paging older history on demand keeps memory flat regardless of how much history a thread accumulates.

UI Changes

The web and mobile timelines gain a small "Load older history" affordance and a loading spinner when you scroll to the top of a thread longer than the window. It's a minor, additive interaction. I wasn't able to attach before/after screenshots or a scroll video here — happy to add them, or the server commit alone (no UI) carries the actual reliability fix if you'd rather review that first.

Checklist

• I explained what changed and why
• before/after screenshots for UI changes — to be added (web/mobile lazy-load affordance)
• video for the scroll-up interaction — to be added
• The server fix is independently reviewable (first commit, no UI)

🤖 Generated with Claude Code

---

Note

High Risk
Large cross-cutting change to projection queries, subscriptions, and client assumptions about complete activity lists; incorrect pagination or reshape detection could hide history or break prompts until paged in.

Overview
Bounds server memory by no longer loading full thread history on open: thread detail returns only the 500 most recent activities, sets hasMoreActivities, and adds getThreadActivities for cursor-paged older rows (sequence or legacy (createdAt, activityId) cursors). The orchestration snapshot HTTP path and t3 CLI use getCommandReadModel instead of the full snapshot so project-only callers avoid materializing huge per-thread tables.

Reconnect safety: subscribeThread replays at most 1,000 global events after the client cursor; stale cursors get a fresh thread snapshot instead of unbounded event replay.

Web and mobile share useOlderThreadActivities to merge lazy-loaded pages with the live window, drive scroll-up / “Load older history,” and derive pending approvals from mergedActivities (newest open prompt, not oldest). Sidebar thread pre-warming is removed.

Contracts: optional hasMoreActivities on threads; new RPC and error types for activity paging.

^{Reviewed by Cursor Bugbot for commit 034f493. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Paginate large thread history to prevent server memory exhaustion

• getThreadDetailById now returns at most 500 most-recent activities per thread with a hasMoreActivities flag; previously all activities were loaded into memory unconditionally.
• Adds a new getThreadActivities WebSocket RPC for cursor-based paging of older activities, supporting both sequence-based and (createdAt, activityId) keyset cursors.
• subscribeThread now bounds catch-up replay to 1,000 events; when the client cursor is stale it substitutes a fresh thread snapshot instead of replaying an unbounded event range.
• Client-side useOlderThreadActivities hook merges paged older activities with the live window; web and mobile UIs auto-trigger a load when scrolled to the top and show a 'Load older history' button or spinner.
• The sidebar prewarmer component (SidebarThreadDetailPrewarmer) is removed, stopping background thread-detail subscriptions for visible threads.
• Risk: threads with more than 500 activities will no longer have their full history returned in the initial snapshot; clients must page backwards explicitly.

^{Macroscope summarized 034f493.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: de39051b-b7bd-4d61-86f1-0d69eb174df9

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[olafura]

Thanks Bugbot — addressed in cddc7e1:

• Reconnect leaves stale older pages / Revert keeps lazy-loaded activities (both): the lazy-load state now resets when the live window's oldest activity id changes, not just on thread switch. That id is stable while activities only append, so it doesn't fire during a normal turn — but a reconnect re-snapshot or a checkpoint revert that drops live rows changes it, invalidating the prepended older pages. Applied to web and mobile.
• Web skips live activity dedup: web now dedups a new older page against both the already-loaded older pages and the live window (mobile already did this), so a boundary overlap can't produce duplicate ids.
• Unsequenced cursor hides sequenced history: I believe this is a false positive. Legacy NULL-sequence rows always sort as the oldest in both the detail window (sequence DESC, NULLs last) and the pager. So the oldest-loaded row becomes unsequenced only after every sequenced row is already in the loaded set — there are never sequenced rows older than an unsequenced one for the sequence IS NULL cursor to strand. Happy to add a test if useful.

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces significant new pagination infrastructure across server, contracts, and both clients—including a new RPC endpoint, a shared lazy-loading hook, and UI affordances for loading older history. The scope and architectural nature of these changes warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[olafura]

Addressed the latest review in 75d25e266 (rebased on current main):

• Stale in-flight load after reset — the reset now bumps a generation counter (olderActivitiesGenRef); the in-flight load captures it at dispatch and drops its result in .then()/.finally() if it changed, so a late page can't repopulate the cleared state. Replaces the thread-key-only staleness check (which was unchanged on a same-thread reshape).
• Revert may skip reset / Reconnect leaves stale pages — the reset now also fires when the live window shrinks (a checkpoint revert that drops rows without changing the oldest row), not just when the oldest id changes (reconnect). A pure append (same oldest, larger count) keeps the loaded pages. Applied to web and mobile.
• Empty page churns / loops — when dedup leaves nothing new, the handler returns without creating a new array reference and stops, so an unadvanced cursor can't loop identical requests.
• Web skips live dedup — dedups against the full merged set (older + live).
• Unsequenced cursor hides sequenced history — I believe this is a non-issue and added a server regression test to prove it: NULL-sequence rows always sort oldest, so when the oldest loaded row is unsequenced every sequenced row is already in the window; the sequence IS NULL cursor reaches all 103 older unsequenced rows in the test without stranding any of the sequenced ones.

Server/web/mobile typecheck, lint, and the projection + timeline suites pass.

[olafura]

Addressed the latest review round:

Cursor — "Append clears lazy-loaded history" (Medium) — The reset effect used liveThreadActivities[0]?.id as the live window's oldest boundary. The reducer's activityOrder sorts unsequenced rows to the end (a missing sequence is treated as newest) while the server snapshot lists legacy unsequenced rows first, so the first live append re-sorted the array and shifted index 0 — making a plain append look like a window reshape and wrongly clearing the user's scrolled-up history. Replaced it with an order-independent oldest sentinel, liveWindowOldestActivityId (min by createdAt, then id), which is stable across appends (always newest) and only changes on a genuine reshape (reconnect re-snapshot / revert). Applied to both ChatView and the mobile composer; the helper lives in client-runtime with unit tests.

macroscope — mixed-thread flash (Low, ChatView.tsx:1724) — Moved the older-pages reset from useEffect to useLayoutEffect so the cleared state commits before paint; otherwise a thread switch rendered one frame with the previous thread's lazy-loaded pages still merged in, flashing stale work-log/approval rows. Mirrored in mobile.

The two earlier macroscope Mediums (1703 staleness guard, 1762/1786 empty-fresh loop) are already ✅ resolved in the prior commit — the generation-ref guard supersedes the request-key check, and the empty-fresh path early-returns without churning the array. The High 1772 ("unsequenced cursor hides sequenced history") remains a false positive: the lazy-load cursor (threadActivities[0]) is only unsequenced when the whole thread fits the window, in which case hasMoreActivities is false and no sequenced rows are stranded.

pnpm --filter @t3tools/web typecheck, --filter @t3tools/mobile typecheck, the client-runtime reducer suite (24 passing incl. the new helper specs), and lint are green. Squashed into the lazy-load-hardening commit and rebased on upstream/main.

[olafura]

Latest round:

Cursor — "Wrong pagination cursor after live sort" (High). Valid consistency gap I introduced last round. I'd made the reshape sentinel order-independent (liveWindowOldestActivityId = chronological-oldest) but left the pagination cursor on threadActivities[0], which the reducer can fill with a newer sequenced row (it sorts unsequenced rows to the end). Those two now agree: extracted oldestActivityByChronology(activities) and made the cursor page from it in both ChatView and the mobile composer, so paging always continues from the genuinely-oldest loaded row (driving the unsequenced beforeCreatedAt/beforeActivityId cursor when that row is legacy-unsequenced, instead of a beforeSequence that could skip it). The sentinel now delegates to the same helper. Added a reducer test asserting the cursor-oldest is the unsequenced legacy row and that the sentinel agrees.

Cursor — "Stale hasMore after revert" (Medium). False positive. thread.reverted keeps activity.turnId === null || retainedTurnIds.has(activity.turnId) — it drops activities for the reverted (newer) turns, never the older history. hasMoreActivities describes whether older rows exist beyond the window, which a revert of newer turns doesn't change, so retaining the flag is correct. (And even if it were stale-true, the lazy-load already no-ops on an empty page: fresh.length === 0 clears hasMore without looping.)

@t3tools/web + @t3tools/mobile + @t3tools/client-runtime typecheck clean, reducer suite 26 passing, lint clean. Squashed into the lazy-load-hardening commit and rebased on upstream/main.

[olafura]

Self-review (10-angle, max effort) of 848cff64

Ran an adversarial multi-angle review and grounded the findings against the real 1.5 GB database (131,532 activities). Posting the results.

Key empirical finding

The projection_thread_activities.sequence column is 100% NULL in every database I can observe (both the 4 MB dev DB and the 1.5 GB realistic one), and 0 threads have a NULL-sequence row newer than a sequenced one. runtimeEventToActivities only sets sequence from event.sessionSequence, which no provider currently emits. So the beforeSequence/sequenced-row pathway is never exercised in production — all paging goes through the created_at keyset. This reframes the scary-looking findings as latent, not live.

Fixed

Sev	Finding	Fix
Low (hardening)	The internal SQL-input schemas typed beforeSequence/limit as bare Schema.Number, looser than the contract's NonNegativeInt. The WHERE clause (sequence < beforeSequence OR sequence IS NULL) is only equivalent to the old COALESCE(sequence,-1) < beforeSequence for non-negative cursors — a negative one silently returns only unsequenced rows.	Tightened both internal schemas to NonNegativeInt so any future non-RPC caller is validated, not just the RPC boundary. Typecheck + 11 pager tests green.

Latent / by-design (NOT changed — flagging for a decision)

• Window vs reducer NULL-ordering disagreement (highest-severity-if-triggered): the server window orders sequence DESC (SQLite → NULLs last/oldest) while the client reducer's activityOrder uses sequence ?? MAX_SAFE_INTEGER (NULLs newest). They only agree today because every row is NULL. If sequence is ever partially backfilled, a thread with mixed rows would (a) drop the newest unsequenced rows from the 500-window and (b) paginate in a different order than it displays. The robust fix is to order both sides by created_at consistently — but that's an index-strategy change (the covering index leads with sequence) and a no-op today, so I've left it. Recommend either ordering by created_at end-to-end, or guaranteeing sequence is all-or-nothing.
• Speculative sequence machinery: given the above, the entire sequenced-cursor arm (second SQL query, union input arm, "beforeSequence" in input branch, the sequence-leading index) is dead in practice. A single (created_at, activity_id) keyset would cover 100% of today's reality with ~half the surface. Kept as forward-looking, but worth a conscious call.
• web/mobile duplication (AGENTS.md "Duplicate logic across multiple files … should be avoided"): the ~120-line lazy-load state machine (gen guard, in-flight ref, reshape useLayoutEffect, dedup) is near-verbatim in ChatView.tsx and use-thread-composer-state.ts. Should become one shared hook — deferred because it's a cross-app React-architecture refactor, not a surgical fix.
• Asymmetry: activities are windowed to 500 but messages in the same getThreadDetailById are still unbounded, so the heap-OOM rationale is only half-applied.
• handleScroll/loadOlder identity churn on every live append (perf, not correctness); THREAD_DETAIL_ACTIVITY_WINDOW overloaded as window size + default page + max page.

Verified safe (no action)

getSnapshot→getCommandReadModel (only the CLI reads .projects); toReversed() (Node/lib supports it); mapThreadActivityRow typing; hasMoreActivities optional doesn't break fixtures/decoders; window slice/hasMore arithmetic; the empty-page stop guard; sequence===0 cursor; created_at tie-break cursor. No CLAUDE.md/AGENTS.md violations beyond the duplication note.

[olafura]

Addressed the latest review round (ead248b4)

Three threads — two fixed, one verified as a false positive. Both fixes applied to web (ChatView) and mobile (use-thread-composer-state) identically.

🟡 Cursor — "Duplicate page ends pagination early" (r3455736449) — Fixed. Paging now runs off an explicit olderCursorRef that advances to each page's oldest row (pages are ascending, so [0] is oldest) on every successful page — even one that dedupes to zero new rows. Because the server cursor is strict, the cursor strictly decreases, so an all-overlap page keeps paging (respecting the server's page.hasMore) without re-requesting the same cursor or looping. The old setOlderHasMore(false) early-stop is removed.

🟢 macroscope — "stale dedup seen" (r3455790594) — Fixed as suggested. Added a threadActivitiesRef (mobile: mergedActivitiesRef) assigned the merged array every render; the .then() dedup now builds seen from …Ref.current instead of the closure-captured snapshot, so a live append or a prior prepend that settles mid-flight is always reflected.

🟡 Cursor — "Stale hasMore after revert" (r3455516138) — False positive. thread.reverted keeps turnId === null || retainedTurnIds.has(turnId), dropping activities for the reverted (newer) turns and keeping older ones. hasMoreActivities describes whether rows exist older than the window, which a newer-turn revert doesn't change — so retaining it is correct. No code change.

These were latent (reachable only in the mixed sequenced+unsequenced case, which never occurs — sequence is 100% NULL in the real DB), but the fixes make paging correct-by-construction rather than relying on that invariant.

Verified: @t3tools/web + @t3tools/mobile typecheck clean, lint clean, reducer (26) / MessagesTimeline (12) / pager (11) suites green. Squashed into the lazy-load commit and rebased on upstream/main.

[carlulsoe]

Hi, i experienced the same issue:
Real-world validation for the reconnect-safety part of this PR:

• An Android client resumed its cached shell at approximately sequence 46,900 while the server projection was at approximately 48,450.
• subscribeShell({ afterSequence }) partially advanced, then stopped delivering newer shell state. The connection remained alive (heartbeats continued), but recent threads never appeared.
• The server was running 0.0.29-nightly.20260712.791.
• As an operational hotfix, I changed the stale-resume path to attach the live buffer, load an authoritative shell snapshot, then replay only events newer than that snapshot. After reconnecting, the Android client received the snapshot and immediately converged to the browser/server state.
• The unrelated Node runtime change was ruled out as the cause.

This cursor was about 1,550 global events behind, so the 1,000-event cutoff introduced here would have selected the fresh-snapshot path and avoided the failure. This is good evidence that bounding shell/thread catch-up and falling back to a current snapshot is needed in practice, not only for memory protection.

[olafura]

Hi, i experienced the same issue: Real-world validation for the reconnect-safety part of this PR:

• An Android client resumed its cached shell at approximately sequence 46,900 while the server projection was at approximately 48,450.
• subscribeShell({ afterSequence }) partially advanced, then stopped delivering newer shell state. The connection remained alive (heartbeats continued), but recent threads never appeared.
• The server was running 0.0.29-nightly.20260712.791.
• As an operational hotfix, I changed the stale-resume path to attach the live buffer, load an authoritative shell snapshot, then replay only events newer than that snapshot. After reconnecting, the Android client received the snapshot and immediately converged to the browser/server state.
• The unrelated Node runtime change was ruled out as the cause.

This cursor was about 1,550 global events behind, so the 1,000-event cutoff introduced here would have selected the fresh-snapshot path and avoided the failure. This is good evidence that bounding shell/thread catch-up and falling back to a current snapshot is needed in practice, not only for memory protection.

carlulsoe Thank your for testing it, glad it helps

[EMOEMOJAI]

Real-world confirmation that this PR targets the right root cause: hit the exact failure mode on 0.0.29-nightly.20260718.841 — backend child OOMs at the default V8 heap cap (~45 s after startup, RSS → 3.7 GB) while loading a 5.1 GB state.sqlite with 1.54M orchestration_events, producing an endless "cannot connect remote endpoint 127.0.0.1:3773" crash-loop.

Full diagnostics posted in #996 (comment). Notable data point for sizing the page limits: in-memory expansion was roughly 5× the raw JSON payload, and single threads can reach 165k+ events, so per-thread pagination (not just per-snapshot) matters. Would love to see this merged — currently the only user-side fix is manual SQLite surgery.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want fixes drafted automatically? Bugbot Autofix can create code changes for findings. A team admin can enable Autofix in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit e94ab70. Configure here.}

[danieliser]

Real-world macOS desktop validation: this PR targets the backend OOM path we captured.

The test profile had an untouched 4.43 GiB state.sqlite, approximately 1.24M orchestration events, and 432k projected activities. Before the patch, the packaged backend listened successfully and then OOM-aborted 35–85 seconds later at roughly 2.8 GiB RSS. The native stack passed through node::sqlite::StatementSync::All and V8 string creation.

Tracing showed cached task cursors 271k–591k global events behind. subscribeThread read the entire intervening global range and filtered by task only after SQLite decoded it. Electron’s first-ten sidebar detail prewarmer caused several catch-ups concurrently.

I tested all commits from this PR with two companion changes:

1. The stale-cursor snapshot path now emits synchronized when requestCompletionMarker is true, between the replacement snapshot and buffered live events. A focused regression test covers snapshot -> synchronized.
2. Electron’s hidden sidebar detail prewarmer was removed. Existing per-task atoms already have a five-minute idle TTL, so unopened sidebar rows should not retain detail streams indefinitely.

I also used an 8 GiB packaged-backend heap default as a safety rail while preserving explicit NODE_OPTIONS.

With the same database and no pruning or vacuuming, observed backend RSS stayed around 295k–437k KiB instead of climbing to ~2.8M KiB, port 3773 remained available for 15+ minutes, and no new crash report appeared.

This was a combined build, so the RSS result should not be attributed to #3510 alone. However, the SQLite trace directly confirms that its bounded replay, stale-snapshot fallback, and activity paging target the actual OOM path.

I recommend merging after fixing the missing synchronization marker. Recovering the aggregate-indexed per-task replay from closed PR #3825 would be a strong follow-up.