App Crash on heavy thread

[nassimna]

my t3code app is crashing after i ran a code review thread on a monorepo project it just crash and sometimes it finishes the work, but i could not start any new thread on that workspace without deleting that thread, and the only solution to get the app working well again is to clear the .t3 folder

• i have no attachment because i deleted the .t3 folder to fix the problem

[chuks-qua]

Any rough idea how large the thread was? also it could be due to some rerendering issues too. @juliusmarminge recently applied fixes for those but without proper context Can't really do much to help.

If you can provide a better estimate on the size of the thread that would be nice

[chuks-qua]

related to #980
Add this to ur desc please

[juliusmarminge]

i've pushed this pretty hard with long threads before so likely a regression. anything that helps debugging would be good

[nassimna]

I tried to replicate the problem, but I couldn't after deleting my .t3 folder and reseting every thing but reading the #980 issue, I confirmed that had more than 20 worktrees between all my projects, and my .t3 was over 8 GB. Since there is no UI to manage worktrees in the current release, i dont really know how to manage that

[chuks-qua]

I could not replicate the issue but I had claude investigate this deeply and eliminate the false positives too. here is what it found. the one that stood out the most is the snapshot which is likely the cause for #980. which is probably why @nassimna could access his workspace only after deleting the .t3 folder.

---

Investigation: Root Cause Analysis

I dug into the codebase to understand why this happens. Here's what I found.

How worktrees work in T3 Code
T3 Code creates real git worktrees (git worktree add) stored at ~/.t3/worktrees/{repoName}/{branch}`. The worktree path is tracked per-thread in the projection_threads SQLite table. Multiple threads can share one worktree.

The core problem: worktrees accumulate and never get cleaned up

Worktrees are only removed when a user explicitly deletes a thread from the sidebar. There is no automatic pruning, no count limit, no disk space check, and no startup cleanup. git worktree prune is never called anywhere in the codebase.

Worse, there are several ways worktrees become permanently orphaned:

• User declines the "Delete worktree too?" dialog when deleting a thread. The thread is already gone from the DB at that point, so the worktree can never be discovered again.
• App crashes between thread deletion and worktree removal. The thread is marked deleted in the database (line 612 of Sidebar.tsx) before worktree removal is attempted (line 636). If the app crashes in that gap, the worktree is orphaned with no recovery path.
• git worktree remove fails (permissions, timeout, disk full). A toast is shown, but the failed removal is never retried.
• Project deletion doesn't cascade to threads. The project.deleted event only sets deletedAt on the project. Threads and their worktrees are left behind.
• Database corruption or deletion orphans every worktree instantly, since the app never scans ~/.t3/worktrees/ on disk.

Why accumulated worktrees cause freezes

The getSnapshot() query (ProjectionSnapshotQuery.ts) loads ALL projects, threads, messages, activities, sessions, plans, and checkpoints in a single transaction with no WHERE deleted_at IS NULL filter. With many threads (including deleted ones that were never filtered out), this builds a large in-memory object graph on every snapshot refresh.

The client requests a snapshot refresh on every domain event (throttled to 100ms). So during active agent sessions
that emit many events, the snapshot query runs repeatedly with all that data.

Additionally, listBranches() calls git worktree list --porcelain on every invocation, which slows down as worktrees accumulate.

Each worktree with a setup script also gets its own node_modules (~1GB per worktree for a monorepo). This is the main source of the 8GB+ disk usage reported in #996.

What's missing

1. No automatic worktree cleanup (age-based, count-based, or on startup)
2. No git worktree prune call anywhere
3. No way to view or manage worktrees outside of deleting individual threads
4. No cascading cleanup when projects are deleted
5. No crash recovery for interrupted worktree removals
6. Snapshot query loads deleted threads into memory unnecessarily

---

@juliusmarminge i could whip up a pr (or multiple prs to match repo standards) attempting to fix these while you test when it's ready. what do you think?

[olafura]

@chuks-qua @nassimna Could you check out my branch to see if it fixes your problem

[EMOEMOJAI]

Reproduced with full diagnostics — backend OOM crash-loop, root cause confirmed

Hit what looks like the terminal form of this bug on 0.0.29-nightly.20260718.841 (macOS 26.5.2, M4 Max): app shows "Failed to connect. Reconnecting… / cannot connect remote endpoint 127.0.0.1:3773" and crash-loops forever (25 macOS crash reports in one morning).

Root cause: the backend child (apps/server/dist/bin.mjs) OOMs ~45s after every startup while loading thread history — consistent with the unfiltered snapshot load described in the investigation above:

FATAL ERROR: MarkCompactCollector: young object promotion failed
Allocation failed - JavaScript heap out of memory
[30531] 44149 ms: Scavenge 3641.2 (3719.6) -> 3624.6 (3724.4) MB ...

State at time of failure:

• ~/.t3/userdata/state.sqlite: 5.1 GB; orchestration_events: 1,541,085 rows (~1.6 GB JSON payload); projection_thread_activities: 1.4 GB; 304 thread streams
• Two single threads had 165k and 172k events each (~170 MB / ~150 MB payload) — long agent sessions
• Backend RSS climbs to ~3.7 GB in ~45 s, then aborts at the default V8 heap cap; the desktop shell respawns it → infinite loop. In-memory expansion is roughly 5× the raw JSON payload.
• NODE_OPTIONS=--max-old-space-size=... is not honored by the backend child (stripped), so there is no user-side escape hatch.

Workaround that fixed it (no full .t3 wipe needed): with the app quit, delete old/huge thread streams from orchestration_events + matching rows in projection_* / orchestration_command_receipts / checkpoint_diff_blobs, then VACUUM. DB went 5.1 GB → 323 MB; backend now stable at ~300 MB RSS with no crashes.

This is exactly the failure mode PR #3510 addresses — can confirm the pagination approach targets the right root cause. Happy to provide full crash reports or trace logs if useful.

[danieliser]

Desktop backend OOM on large profiles: stale global replay and eager task hydration

Summary

A packaged macOS arm64 desktop build was crash-looping every 35–85 seconds against a 4.43 GiB state.sqlite containing approximately 1.24 million orchestration events, 432,000 projected thread activities, and multiple very large archived tasks.

The backend child reached approximately 2,797,872 KiB RSS before aborting through node::OOMErrorHandler. The native stack passed through SQLite row extraction and V8 string creation.

The database was not pruned. The stable solution was to decouple retained history from resident memory.

Root cause

The failure was not initial server startup or archived history merely existing on disk.

Persisted client cursors were approximately 271,000–591,000 global events behind. The old subscribeThread resume path called:

readEvents(afterSequence, Number.MAX_SAFE_INTEGER)

It then filtered for the requested task only after SQLite had read and decoded every intervening global event.

Tracing recorded 15,544 pages of:

SELECT ...
FROM orchestration_events
WHERE sequence > ?
ORDER BY sequence ASC
LIMIT ?

The sidebar also mounted hidden detail subscriptions for up to ten visible tasks. Each subscription could initiate another global catch-up, multiplying SQLite decoding and allocations.

Opening a large task had another unbounded path: its complete activity history was materialized into the initial detail snapshot.

What worked

1. Integrated fix(server): paginate large thread history to stop the server running out of memory #3510:

   • Caps recent subscription replay at 1,000 global events.
   • Replaces stale-cursor replay with a fresh per-task snapshot.
   • Loads only the newest 500 activities initially.
   • Pages older activity through orchestration.getThreadActivities.
   • Adds on-demand older-history loading for web and mobile.

2. Removed Electron sidebar detail prewarming:

   • Unopened tasks remain lightweight shell metadata.
   • Messages, activities, checkpoints, and task streams load when the task is viewed.
   • Existing task-detail state becomes eligible for disposal after five minutes without subscribers.
   • The current task and tasks with active observers remain subscribed normally.

3. Fixed a correctness hole still present in fix(server): paginate large thread history to stop the server running out of memory #3510:

   • Its stale-cursor snapshot fallback omitted the requested synchronized marker.
   • The tested fix emits snapshot, then synchronized, then buffered live events, with a regression test.

4. Added an 8 GiB packaged-backend heap default as defense in depth:

   • Explicit NODE_OPTIONS still wins.
   • Development and WSL environment plumbing are unchanged.
   • This is a safety rail, not the primary performance fix.

5. Kept the earlier streaming improvements:

   • perf(server): skip shell refresh for streaming deltas #4009 prevents full shell refreshes for streaming-only deltas.
   • Increase Effect RPC ping timeout for large snapshots #3817 increases Effect RPC ping tolerance.
   • These improved streaming responsiveness but did not independently remove the unbounded resume scan.

Results

Measurement	Before	Patched build
Database	~4.43 GiB	~4.43 GiB, unchanged
Backend RSS	~2,797,872 KiB before abort	295,056–437,216 KiB during soak
Lifetime	35–85 seconds	15+ minutes and responsive
Port 3773	Repeatedly disappeared	Continuously listening
Crash reports	Roughly one per minute	None during patched soak
History deleted	N/A	None

Observed backend RSS fell approximately 84–89% without shrinking the database.

No event, activity, checkpoint, message, or archived-task rows were deleted. No VACUUM or log rotation was performed. At the end of verification the live database was 4,755,988,480 bytes and the existing log directory was still 6.6 GiB.

Verification

• 146 focused tests passed across desktop configuration, server projection/paging, stale-cursor recovery, shared lazy-history state, and web behavior.
• The existing 22 streaming projection tests also pass.
• Targeted type checks passed for contracts, client-runtime, web, desktop, mobile, and server.
• Targeted lint and formatting checks passed.
• An arm64 DMG was rebuilt and installed into /Applications.
• With the parent environment unset, the backend child received NODE_OPTIONS=--max-old-space-size=8192.
• Live use confirmed the packaged application is substantially more responsive.

Upstream status and remaining work

This is not fully solved on main.

• fix(server): paginate large thread history to stop the server running out of memory #3510 is the closest current fix. It is open and mergeable, but still needs the missing synchronization marker.
• perf(orchestration): bound in-memory read model and client per-thread state #4176 now addresses [Bug]: Orchestration read model and per-thread client/VCS state grow unbounded over uptime #4178’s separate O(1) command-state and deleted-task/client/VCS cleanup concerns while deliberately retaining archived tasks.
• Closed PR perf: reduce CPU usage in event replay, projections, and sidebar #3825 contains useful prior art for thread-indexed replay and hover-intent prewarming.
• Messages, proposed plans, and checkpoints remain unbounded when an individual task is opened.
• Recent resume still reads a global range before filtering, although that range is now capped at 1,000.
• Event and log storage continue growing on disk. Retention and rotation remain valid operational work, but they should not be required for heap safety.

The important result for this issue is that deleting .t3 or pruning old chats is a workaround, not a required fix. The same large database became stable once hydration and replay were bounded.