[orchestrator-v2] Fix shell cache hydration and multi-env project grouping

[mwolson]

Summary

• Ignore Grok ACP JSON-RPC responses with non-numeric ids before they reach the Effect RPC client.
• Persist and round-trip the latest live environment shell snapshot so projects and thread shells hydrate before live synchronization catches up.
• Keep shell enrichment non-blocking, re-emit identity refreshes on subscribe, and mark only roots whose identity resolution completed.
• Emit initial authoritative shell frames before concurrent enrichment refreshes.
• Accept authoritative server resets when a cached client sequence is ahead, while preventing stale enrichment snapshots from rolling back shell structure.
• Flush a stable final shell snapshot only after scoped stream workers stop.

Problem and Fix

Problem and Why it Happened	Fix
Grok can emit ACP JSON-RPC responses for agent-internal messages with non-numeric ids. Those are not valid Effect RPC responses and could fail numeric id decoding during startup.	Drop non-numeric response ids in the ACP transport wrapper before forwarding messages into the Effect RPC client.
Debounced shell cache writes could be lost on disconnect or shutdown, and runtime DateTime schemas could reject rows after crossing the JSON storage boundary.	Track the latest live shell independently, flush it on disconnect and scope close, and use JSON-compatible v2 cache envelope schemas.
Same-sequence identity enrichment could arrive while an older cache save was in flight, leaving the older write as the final IndexedDB value.	Re-check immutable snapshot identity after each save and persist the newest snapshot before returning.
Repository identity may finish resolving before a WebSocket subscriber attaches, so a resumed client could miss the completion and keep projects split across environment groups.	Emit one full snapshot at the start of every shell subscription, including afterSequence resumes, then continue with replay and live deltas.
A cached client can be ahead after the server database is rebuilt or reset. Lower-sequence HTTP and initial WebSocket snapshots were treated as stale enrichment, so all subsequent lower deltas were ignored indefinitely.	Treat HTTP and initial WebSocket snapshots as authoritative structural resets even at a lower sequence, and persist the corrected shell.
Null repository identity means either unresolved metadata or a completed no-remote result. Retaining every null made an old identity impossible to clear.	Add optional resolved-root metadata to enrichment snapshots. Only completed roots may authoritatively set identity to null; unrelated cold roots retain their prior identity.
A concurrent enrichment refresh could emit before the initial authoritative snapshot. If it observed a newer sequence, the later initial frame could roll structure backward and a disconnect could persist that intermediate shell.	Drain the complete initial snapshot prefix first, then merge queued enrichment with replay, completion, and live traffic. The enrichment subscription still attaches before loading so completions remain buffered.
Lower or equal-sequence enrichment snapshots could overwrite current projects, threads, archives, or sequence, and an empty-project shortcut bypassed sequence checks entirely.	Make same-or-lower enrichment identity-only and remove the empty-project shortcut. Higher-sequence snapshots still converge lagging clients.
The scope flush finalizer ran before scoped subscription fibers were interrupted, so a slow save could chase a continuously advancing snapshot and delay teardown indefinitely.	Register the flush finalizer before worker fibers so reverse finalizer order stops workers first, then saves a stable snapshot.

Defensive Fixes

Problem and Why it Happened	Fix
A project delta can temporarily carry null identity while background enrichment is still pending.	Retain prior identity for the same project and workspace root until a marked enrichment completion makes null authoritative.
Older mobile clients may connect to a newer server during rollout. A new shell stream union member would break closed-union decoding.	Keep the existing snapshot member and add optional metadata. Effect's legacy struct decoder accepts and strips the excess field, covered by a compatibility test.

Validation

• vp test run packages/contracts/src/orchestrationV2.test.ts packages/client-runtime/src/state/shellReducer.test.ts packages/client-runtime/src/state/shell-sync.test.ts apps/server/src/orchestration-v2/ShellStream.test.ts apps/server/src/project/ProjectEnrichmentService.test.ts: 49 tests pass
• Targeted vp fmt --check and vp lint for the touched files: pass, with pre-existing inline-schema warnings only
• Targeted typechecks for @t3tools/contracts, @t3tools/client-runtime, and t3: pass, with pre-existing Effect suggestions only
• Fresh Grok implementation and final review passes: no correctness findings
• Independent gpt-5.6-terra Codex review: no correctness findings

Note

Fix shell cache hydration and multi-env project grouping in orchestrator-v2

• Adds mergeShellSnapshotProjects to shellReducer.ts to merge incoming snapshots with prior state, preserving repositoryIdentity for unchanged roots in authoritative snapshots and avoiding structural rollbacks for enrichment snapshots at or below the cached sequence.
• Reworks EnvironmentShellState in shell.ts to track the latest live snapshot and re-save it if a newer same-sequence snapshot arrives while a prior save is in flight; flushes on disconnect and scope teardown without blocking state updates.
• Extends the WebSocket shell stream in ws.ts to always emit an authoritative unmarked initial snapshot first, optionally followed by a same-sequence marked snapshot containing resolvedRepositoryIdentityRoots, before any enrichment refresh can appear.
• Adds JSON-friendly schemas (*Json variants) in orchestrationV2.ts and updates orchestrationCache.ts to use them for persisted shell and thread snapshots, affecting date/timestamp serialization.
• Adds repositoryIdentityResolved to ProjectEnrichment in ProjectEnrichmentService.ts to distinguish resolved (including cached null) from pending/failed identity resolution.
• Risk: stored shell and thread snapshots now use JSON-encoded schemas; existing cached data with non-JSON date formats may fail to decode and be discarded.

^{Macroscope summarized 6764a41.}

---

Note

Medium Risk
Touches live shell sync, WebSocket ordering, and client cache persistence across server resets and same-sequence enrichment; mistakes could mis-group projects or serve stale shell state, but behavior is heavily test-covered.

Overview
Fixes orchestrator v2 shell sync so clients hydrate from cache reliably, group projects by git identity across environments, and stay correct when server sequence resets or enrichment races the initial snapshot.

Server: HTTP and WebSocket shell paths use non-blocking getAvailable for project enrichment and expose repositoryIdentityResolved. Shell subscribe always rehydrates a full snapshot (including afterSequence resumes), emits an unmarked authoritative frame first, optionally a same-sequence frame with resolvedRepositoryIdentityRoots, then merges batched enrichment refreshes only after that prefix via composeShellStreamWithEnrichment.

Client: mergeShellSnapshotProjects distinguishes authoritative resets (lower sequence still replaces structure) from enrichment-only updates (identity patches, no structural rollback). Shell state tracks the latest live snapshot, flushes on disconnect/scope close without blocking UI, and re-persists when a same-sequence enriched snapshot lands during an in-flight save. Orchestration cache v3 uses JSON-safe snapshot schemas for date fields.

ACP: Drops JSON-RPC responses with non-numeric ids before they hit the Effect RPC client (Grok agent-internal messages).

^{Reviewed by Cursor Bugbot for commit 6764a41. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 128d3b37-2ee6-42f4-9880-537c168f76d8

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Needs human review

3 blocking correctness issues found. This PR introduces substantial new logic for shell cache hydration and multi-environment project grouping. Unresolved review comments identify potential bugs including a high-severity issue where cache decoding may silently fail for projects, plus concerns about the persistence loop and identity clearing logic.

^{You can customize Macroscope's approvability policy. Learn more.}

Dismissing prior approval to re-evaluate d6a0e95

[macroscopeapp]

🟠 High src/orchestrationV2.ts:1601

OrchestrationV2ShellSnapshotJson overrides threads and archivedThreads with JSON variants but leaves projects pointing at the original OrchestrationProjectShell, whose createdAt and updatedAt fields are Schema.DateTimeUtc. When a cache row is decoded via Schema.fromJsonString, those fields are ISO strings, so any non-empty projects array fails decoding and the snapshot is discarded. This silently breaks fast-start hydration. Add a OrchestrationProjectShellJson variant that uses Schema.DateTimeUtcFromString for its datetime fields and override projects in OrchestrationV2ShellSnapshotJson.

🤖 Copy this AI Prompt to have your agent fix this:

In file @packages/contracts/src/orchestrationV2.ts around line 1601:

`OrchestrationV2ShellSnapshotJson` overrides `threads` and `archivedThreads` with JSON variants but leaves `projects` pointing at the original `OrchestrationProjectShell`, whose `createdAt` and `updatedAt` fields are `Schema.DateTimeUtc`. When a cache row is decoded via `Schema.fromJsonString`, those fields are ISO strings, so any non-empty `projects` array fails decoding and the snapshot is discarded. This silently breaks fast-start hydration. Add a `OrchestrationProjectShellJson` variant that uses `Schema.DateTimeUtcFromString` for its datetime fields and override `projects` in `OrchestrationV2ShellSnapshotJson`.

[macroscopeapp]

🟡 Medium state/shellReducer.ts:16

retainRepositoryIdentity treats every incoming null repositoryIdentity as a transient enrichment miss and restores the previous value, so a legitimate removal of a repository remote can never clear the identity while the project id and workspace root stay the same. After git remote -v finds no fetch remote, the resolver returns null, but both mergeShellSnapshotProjects and applyShellStreamEvent route that through this helper, so the stale identity stays displayed and persisted indefinitely. Consider distinguishing a real null resolution from a missing/in-flight value, or document why all null values should be suppressed here.

🤖 Copy this AI Prompt to have your agent fix this:

In file @packages/client-runtime/src/state/shellReducer.ts around line 16:

`retainRepositoryIdentity` treats every incoming `null` `repositoryIdentity` as a transient enrichment miss and restores the previous value, so a legitimate removal of a repository remote can never clear the identity while the project id and workspace root stay the same. After `git remote -v` finds no fetch remote, the resolver returns `null`, but both `mergeShellSnapshotProjects` and `applyShellStreamEvent` route that through this helper, so the stale identity stays displayed and persisted indefinitely. Consider distinguishing a real `null` resolution from a missing/in-flight value, or document why all `null` values should be suppressed here.

[macroscopeapp]

🟡 Medium state/shell.ts:96

The persist loop never terminates while shell events keep arriving: after each cache.saveShell completes, it reads latestLiveSnapshot and compares by reference equality. Every new snapshot is a fresh object allocation, so the comparison always fails and the loop starts another save immediately, bypassing the 500 ms debounce and producing continuous storage writes for the entire session. Consider limiting the re-save to only the same-sequence race (stop once the sequence differs from what was just saved) or draining via the queue rather than chasing an indefinitely advancing live reference.

      const latestSnapshot = yield* Ref.get(latestLiveSnapshot);
      // Same sequence can still mean newer content (e.g. repository enrichment).
-      if (Option.isNone(latestSnapshot) || latestSnapshot.value === nextSnapshot) {
+      if (
+        Option.isNone(latestSnapshot) ||
+        latestSnapshot.value === nextSnapshot ||
+        latestSnapshot.value.snapshotSequence !== nextSnapshot.snapshotSequence
+      ) {
        return;
      }

🤖 Copy this AI Prompt to have your agent fix this:

In file @packages/client-runtime/src/state/shell.ts around lines 96-100:

The `persist` loop never terminates while shell events keep arriving: after each `cache.saveShell` completes, it reads `latestLiveSnapshot` and compares by reference equality. Every new snapshot is a fresh object allocation, so the comparison always fails and the loop starts another save immediately, bypassing the 500 ms debounce and producing continuous storage writes for the entire session. Consider limiting the re-save to only the same-sequence race (stop once the sequence differs from what was just saved) or draining via the queue rather than chasing an indefinitely advancing live reference.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6764a41. Configure here.}

[cursor]

Stale shell snapshot after sequence reset

Medium Severity

subscribeShell now loads the initial shell snapshot before reading highWater. If the application sequence moves backward between those reads (server DB rebuild/reset), the already-captured snapshot can stay ahead of the new high water. That stale frame is still emitted as the authoritative prefix, so the client can adopt ghost projects/threads and ignore the post-reset replay and live deltas.

 

^{Reviewed by Cursor Bugbot for commit 6764a41. Configure here.}