feat(orchestrator): move a running thread between devices - #5544
feat(orchestrator): move a running thread between devices#5544Bil0000 wants to merge 262 commits into
Conversation
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
One finding on the new handoff error contract; the migration, its tests, and the bundle schemas otherwise follow the Effect conventions (subpath namespace imports, no service-instance injection, no ManagedRuntime/runPromise, no Layer.succeed shortcuts).
Posted via Macroscope — Effect Service Conventions
There was a problem hiding this comment.
Effect service conventions review of the two new service modules. Three findings, all in apps/server/src/orchestration-v2/. Earlier feedback on OrchestrationV2HandoffError.message still stands and is not repeated here.
Posted via Macroscope — Effect Service Conventions
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model - Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types - Add replay runtime, fixtures, and integration coverage - Update shared contracts and probe transcripts Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring - Introduce in-memory orchestration projections and provider registry - Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections - Support cheap fork creation and Codex native fork rollback - Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work - Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions - Add shell snapshot projection support plus coverage tests - Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server` - Add Claude Agent SDK replay fixtures and test harness - Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module - Share the SDK query runner between live and replay paths - Add replay driver error wrapping for unexpected failures
- persist the selected model on run records - surface run model selection in the debug UI - update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes - Add approval and tool-call replay coverage for new orchestration fixtures - Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy - Add read-only replay fixture and policy mapping tests - Reuse shared approval-policy fixtures across orchestrator tests Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures - update Claude adapter/orchestrator turn handling for steering - refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex - log Claude Agent SDK protocol frames to native event traces - project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles - Update MCP capability, tool, and testing guidance for SDK-based injection
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Record provider fixtures for continued forks and sibling merge-backs, exercise context survival end to end, and stop completed run listeners from consuming later provider events. Co-authored-by: codex <codex@users.noreply.github.com>
Allow a pending merge-back to be consumed while changing providers and deliver both the provider-switch history and fork delta to the selected provider. Co-authored-by: codex <codex@users.noreply.github.com>
Send full canonical target-thread context when a provider switch consumes a merge-back, and cover returning to an existing provider thread after a cross-provider fork. Co-authored-by: codex <codex@users.noreply.github.com>
… chevron Archiving a departed copy on completion pushed the duplication into the archive list instead of removing it. The archive step is gone: the sidebar's one-row rule already hides a residual copy while its live peer is visible, hop revival reuses the pair instead of minting new threads, and the copy surfaces only when the owning device is offline — the safety net working as intended, with no hidden state and nothing to lose. Arrivals still clear archivedAt, which heals copies archived by the previous build. The workspace row's kind hint now absorbs the panel row's slack, parking the chevron and divider at the right edge like every other row.
The follow-through after a push lived in the thread view that started it — which does not survive its own departure: the thread drops out of the sidebar, the selection fallback swaps the route, and the remount destroyed the parked destination before it could fire. Pulls worked only because the destination was the thread already on screen. The pending navigation now lives in a module store and the sidebar — which survives the churn — carries the follow-through. The dialog and banners stop calling the residual copy read-only: the user can work with the thread from any device; what moves is where it runs.
Clearing the handoff link when a thread came home erased its memory: after push, pull, push the pair showed no 'moved from' and offered send verbs where pull-back belonged. Every arrival now keeps a 'here' link — it is provenance, not a lock; only 'away' restricts anything — so the moved-from label and the pull-back framing survive any number of hops.
Deleting the visible half of a pair un-hid the residual copy — the offline-peer safety rule saw its peer disappear — so the user deleted the same thread twice. Delete, archive and unarchive now mirror to the handoff peer, best effort: an unreachable peer keeps its copy until that device is next seen, and a failure on the peer never blocks the local action. Organization state (settle, pin, snooze) stays local by design: the hidden half is not in any list, so those never needed doing twice. Renames propagate structurally instead — every arrival takes the carried title, so a name given on either side reaches the pair at the next hop.
- Constrain ThreadHandoffId to url- and ref-safe characters - Reject part writes that would exceed the payload ceiling - Guard depart while a turn is active; refuse mutations on away threads - Recover handoffs interrupted mid-apply on startup - Surface terminal-history rename failures as warnings - Harden transfer state machine: release lock on every pre-receive failure, never release after the destination applied - Fix move-target filtering and move: prefix collision in the env selector - Assorted correctness fixes from review on git, service and client layers
There was a problem hiding this comment.
Two service-shape conventions are still outstanding after the latest fixes. Previously reported findings that remain unchanged (the handoffError construction helper in ThreadHandoffService.ts and the required free-form message field on OrchestrationV2HandoffError in packages/contracts/src/orchestrationV2.ts) are not re-commented here.
Posted via Macroscope — Effect Service Conventions
- Refuse a handoff whose untracked files would overwrite tracked files in the receiving worktree, mirroring the patch-conflict refusal - Isolate the caller's onProgress callback so a throwing callback can never be misread as a transfer failure after the destination is live - Remove the now-unused ThreadHandoffGitShape export
There was a problem hiding this comment.
Effect service conventions review of the thread-handoff services. One new finding on the git service's error modeling; four findings from earlier runs are still present in this head commit and are listed in the check summary rather than re-commented.
Posted via Macroscope — Effect Service Conventions
- Persist pre-tag, stash and apply directory on the hop row before the patch lands, so startup recovery can roll back a first arrival that died before its thread projection existed (migration 048) - Extend the rollback handler over the whole apply section and make undo clean up extracted untracked files, attachments and terminal logs, not just git state - Refuse a late abort once complete recorded the peer thread - Give the untracked-file collision refusal its own tagged error - Inline the service shapes per Effect service conventions
- Persist the root stash and directory in their own columns (migration 049) so a crash with both trees dirty restores each from its own stash - Clear rollback metadata when a retried handoff reuses its row, so recovery never acts on a previous attempt's tag or stash - Remove staged parts when prepare fails after staging - Vet extracted untracked files in a staging directory against tracked paths, immune to newline-embedded names, before moving them in - Refuse a late complete that would repoint an already-recorded peer
…rees - A retried first arrival reuses the thread id its lineage row already recorded, so deterministic event ids fill the same thread in instead of dedup-dropping against a fresh one - Pop the stashes a successful apply took (root and worktree, each against its own directory); a conflicted pop keeps the stash, keeps its persisted ref and logs which ref to pop by hand - A worktree created by a failed hop is removed in undo, not left holding the branch - Include symlinks in the untracked-collision check
git stash pop only accepts a reflog entry, but every hop persists the stash commit's sha — the reflog slot goes stale as soon as another stash lands. Every restore path silently failed: a hop into a dirty checkout stranded the receiver's uncommitted work in the stash forever. popStash now applies the sha directly and drops whichever reflog slot holds it. New coverage (real temp repositories, live git): - extract refusal on tracked-file and symlink collisions, staging cleanup - stash round trip by sha, bundle creation with unresolvable exclusions, worktree add/remove lifecycle - receive integration: advance into a dirty checkout returns the receiver's edits and clears the stash columns; failed applies roll back the branch and remove the worktree they created; a retried first arrival converges on the recorded thread id; startup recovery pops root and worktree stashes each in their own checkout; a failed prepare leaves nothing staged
- undo and startup recovery reset-to-pre-tag only the apply checkout; when the thread lives in a worktree the repository root gets its stash popped but is never hard-reset to a branch tip it does not own - a branch attached in a worktree this hop created is restored to its old tip when the worktree is removed on rollback - a clone-from-bundle failure removes the partial clone (never a pre-existing directory) so a retry finds the target free - extraction staging is cleared before reuse so a retried archive never sweeps a previous attempt's leftovers into the worktree Both rollback tests were confirmed red against the previous code.
- OrchestrationV2HandoffError carries structured reason + detail and derives message from them, matching the other tagged errors - Drop the handoffError factory; each failure boundary constructs the error itself, keeping asHandoffError as the one pass-through mapper
The created worktree lived only in memory: a crash after addWorktree left it holding the branch, and every retry derived the same path and failed on the occupied directory. Migration 050 persists the flag on the hop row; recovery now removes the worktree, restores the branch to its pre-hop tip from the recorded tag, and a retry finds the path free. Covered by an integration test against a real repository.
CI enforces preferSchemaOverJson; the created-worktree recovery path and its test read the stored manifest with JSON.parse. Both now go through Schema.fromJsonString(OrchestrationV2HandoffBundleV1), the same codec that wrote the column.
There was a problem hiding this comment.
Reviewed the new Effect service modules (ThreadHandoffGit, ThreadHandoffService) and their call sites against the service conventions. The service definitions, make/layer exports, dependency acquisition and error modelling all look consistent with the conventions; the remaining findings are service-boundary imports in the two new test files that alias layer and erase the module namespace.
Posted via Macroscope — Effect Service Conventions
- An unreadable attachments directory now fails the prepare instead of silently shipping a conversation whose attachments never travelled; a missing directory still just means no attachments - recordHop keeps rollback metadata when the same attempt records its arrival; startup recovery finishes the stash pops for an arrived hop that stopped before handing the receiver's changes back - Import the handoff modules as namespaces in the new tests, matching every production consumer
- Keep success feedback visible in the Git action control for 10 seconds - Move the running elapsed timer into the panel menu slot
22bd872 to
a27c1cc
Compare
Closes #5363.
Stacked on #2829 (
t3code/codex-turn-mapping).mainand the orchestrator branch share no common ancestor; every primitive this builds on exists only there.What this does
Run onpicks the machine a thread starts on and locks once it has a message — a thread is born on one device and dies there. This makes that reversible: pick another connected environment from the same control, and the conversation, the provider continuation, and the git working state move there. Pull it back later from the same control, which shows where the thread originally ran.Verified live: real threads moved repeatedly between a MacBook (desktop app) and a Linux server and continued on each side, through the actual relay-free client-brokered transfer — including round trips, pull-backs, and the failure modes listed under Verification.
The safety model
refs/handoff/<environment>/<handoff>/<branch>and stops with both machines untouched.thread.handoff.departlocks the giving side before any bundle is applied anywhere; mutations on an away thread are refused in the orchestrator (a small allowlist of local-copy lifecycle verbs excepted). An interrupted transfer releases the lock; a stranded one has aContinue hereescape hatch, offered only while the transfer never landed. A completed handoff can no longer be aborted or repointed.ThreadHandoffIdis schema-restricted to a URL- and git-ref-safe alphabet.What travels
Branch + unpushed commits (git bundle, checkpoint refs ride along so revert keeps working), tracked changes (binary patch), untracked files (null-delimited tar), the conversation (turn items + covered run ordinals), attachments (original names, which the items reference), terminal scrollback (restored on first open like after a restart), workspace shape (worktree threads land in worktrees — reuse the branch's worktree or provision one, detached when git forbids attaching). A target that lacks the repository entirely clones it from the bundle — no remote, credentials, or network needed on the receiving machine.
How it moves
Manifest inline, large parts content-addressed and fetched by digest in resumable 4 MB chunks; the receiver rejects a chunk that does not continue the staged bytes and treats an exact re-send of already-staged bytes as a no-op, so interrupted uploads resume. The client brokers the copy — the only component authenticated to both environments — and only offers targets that are connected and advertise the capability. Arrivals go through the event sink, so the thread appears in every connected client's sidebar the moment it lands.
Lineage, not direction
Each hop records the previous hop in a lineage table; a return trip is an ordinary hop toward an environment already in the chain, and the receiver revives the prior thread pair even when the client no longer remembers it — one conversation, one sidebar row, wherever it currently runs. A caller-named return target is validated against that lineage before anything is applied. Provenance survives round trips: every arrival keeps a link recording where the thread came from. Lifecycle actions (delete, archive, rename) apply to the pair once.
Version skew
threadHandoffcapability follows thethreadSettlementcontract: absent means the environment is not offered as a destination.Verification
Related
Migrations
047 (
orchestration_v2_thread_handoffslineage table), 048 (apply_cwdcolumn for projection-independent crash recovery) and 049 (root_stash_ref/root_cwdso root and worktree stashes recover independently). 047 was renumbered from 046 after the base branch inserted a migration.Note
Add thread handoff to move a running thread between devices
OrchestrationV2HandoffBundleV1manifest contract, typed part kinds (git-bundle,tracked-patch,untracked-tar, etc.), and versioned DB migrations (047–050) for theorchestration_v2_thread_handoffstable.readHandoffPart/writeHandoffPartHTTP endpoints on the environment API with offset-validated, idempotent chunked transfer and a 4 MB chunk size.prepareThreadHandoffandreceiveThreadHandoffWebSocket RPCs (requiringAuthOrchestrationOperateScope) handled in the server WS layer and wired into production startup with a handoff recovery phase.ThreadHandoffDialogfor progress-tracked moves, a banner for threads running elsewhere (with pull-back and continue-here actions), and sidebar deduplication hiding the 'away' copy when the 'here' peer is visible.ThreadHandoffBannerto surface away threads and resume them locally.Macroscope summarized 0250f02.