Skip to content

feat(orchestrator): move a running thread between devices - #5544

Open
Bil0000 wants to merge 262 commits into
pingdotgg:t3code/codex-turn-mappingfrom
Bil0000:feat/thread-handoff
Open

feat(orchestrator): move a running thread between devices#5544
Bil0000 wants to merge 262 commits into
pingdotgg:t3code/codex-turn-mappingfrom
Bil0000:feat/thread-handoff

Conversation

@Bil0000

@Bil0000 Bil0000 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Closes #5363.

Stacked on #2829 (t3code/codex-turn-mapping). main and the orchestrator branch share no common ancestor; every primitive this builds on exists only there.

What this does

Run on picks the machine a thread starts on and locks once it has a message — a thread is born on one device and dies there. This makes that reversible: pick another connected environment from the same control, and the conversation, the provider continuation, and the git working state move there. Pull it back later from the same control, which shows where the thread originally ran.

Verified live: real threads moved repeatedly between a MacBook (desktop app) and a Linux server and continued on each side, through the actual relay-free client-brokered transfer — including round trips, pull-backs, and the failure modes listed under Verification.

The safety model

  • A branch tip only ever moves to a descendant of itself, on either machine. Diverged or unrelated history parks the sender's commits at refs/handoff/<environment>/<handoff>/<branch> and stops with both machines untouched.
  • Exactly one side is live. thread.handoff.depart locks the giving side before any bundle is applied anywhere; mutations on an away thread are refused in the orchestrator (a small allowlist of local-copy lifecycle verbs excepted). An interrupted transfer releases the lock; a stranded one has a Continue here escape hatch, offered only while the transfer never landed. A completed handoff can no longer be aborted or repointed.
  • Old tip tagged and dirty worktrees stashed (root and destination worktree independently) before any pointer moves; patches dry-run first; a failed apply rolls the repository back and cleans up everything it extracted. Rollback metadata (pre-tag, stashes, apply directory) is persisted before the patch lands, so startup recovery can restore the tree from any crash point — including a first arrival that died before its thread projection existed.
  • A busy thread is interrupted first and the transfer starts when it goes idle; the depart command re-checks for active runs under the dispatch lock, so a racing message cannot slip a live run past the snapshot.
  • Payload ceilings (200 MB warn / 1 GB refuse) enforced during prepare and again per chunk on the receiving side. First hops cut the bundle against the repo's remote-tracking refs; a fully pushed branch ships no bundle at all.
  • Untracked files that would overwrite tracked files on the receiver refuse the hop (checked on real extracted paths in a staging directory, immune to path tricks). ThreadHandoffId is schema-restricted to a URL- and git-ref-safe alphabet.

What travels

Branch + unpushed commits (git bundle, checkpoint refs ride along so revert keeps working), tracked changes (binary patch), untracked files (null-delimited tar), the conversation (turn items + covered run ordinals), attachments (original names, which the items reference), terminal scrollback (restored on first open like after a restart), workspace shape (worktree threads land in worktrees — reuse the branch's worktree or provision one, detached when git forbids attaching). A target that lacks the repository entirely clones it from the bundle — no remote, credentials, or network needed on the receiving machine.

How it moves

Manifest inline, large parts content-addressed and fetched by digest in resumable 4 MB chunks; the receiver rejects a chunk that does not continue the staged bytes and treats an exact re-send of already-staged bytes as a no-op, so interrupted uploads resume. The client brokers the copy — the only component authenticated to both environments — and only offers targets that are connected and advertise the capability. Arrivals go through the event sink, so the thread appears in every connected client's sidebar the moment it lands.

Lineage, not direction

Each hop records the previous hop in a lineage table; a return trip is an ordinary hop toward an environment already in the chain, and the receiver revives the prior thread pair even when the client no longer remembers it — one conversation, one sidebar row, wherever it currently runs. A caller-named return target is validated against that lineage before anything is applied. Provenance survives round trips: every arrival keeps a link recording where the thread came from. Lifecycle actions (delete, archive, rename) apply to the pair once.

Version skew

threadHandoff capability follows the threadSettlement contract: absent means the environment is not offered as a destination.

Verification

  • contracts 266 · client-runtime 621 · web 118 · server suites green (orchestration-v2, persistence, environment, ws, startup, terminal) · monorepo typecheck, lint, fmt clean.
  • New coverage: tip-classification table, ref-name sanitization, full dispatch-path depart/refuse/complete/abort with reason surfacing, payload ceilings, chunk windows, sidebar pair-visibility rules, migration idempotency and mid-apply recovery.
  • Live smoke: repeated MacBook ↔ Linux server transfers of real threads, including the failure modes found and fixed along the way (full-history bundle timeout, empty bundle on a fully pushed branch, missing dispatch routing, JSON codec gap on the handoff timestamp, arrivals invisible to connected clients, post-push navigation, pair dedup after round trips).
  • Code review: all Macroscope findings across the review rounds addressed and resolved, including the crash-recovery, payload-abuse, and single-live-side race classes.

Related

Migrations

047 (orchestration_v2_thread_handoffs lineage table), 048 (apply_cwd column for projection-independent crash recovery) and 049 (root_stash_ref/root_cwd so root and worktree stashes recover independently). 047 was renumbered from 046 after the base branch inserted a migration.

Note

Add thread handoff to move a running thread between devices

  • Introduces an end-to-end thread handoff system allowing a running thread to be transferred between execution environments (devices), preserving git state, conversation history, terminals, and attachments.
  • Adds OrchestrationV2HandoffBundleV1 manifest contract, typed part kinds (git-bundle, tracked-patch, untracked-tar, etc.), and versioned DB migrations (047–050) for the orchestration_v2_thread_handoffs table.
  • Exposes readHandoffPart/writeHandoffPart HTTP endpoints on the environment API with offset-validated, idempotent chunked transfer and a 4 MB chunk size.
  • Adds prepareThreadHandoff and receiveThreadHandoff WebSocket RPCs (requiring AuthOrchestrationOperateScope) handled in the server WS layer and wired into production startup with a handoff recovery phase.
  • The web UI gains a ThreadHandoffDialog for progress-tracked moves, a banner for threads running elsewhere (with pull-back and continue-here actions), and sidebar deduplication hiding the 'away' copy when the 'here' peer is visible.
  • The mobile UI gains a ThreadHandoffBanner to surface away threads and resume them locally.
  • Archiving, unarchiving, and deleting a thread now best-effort mirrors the operation to the handoff peer thread.
  • Risk: handoff recovery runs at server startup; interrupted handoffs from a prior session are automatically resumed, which may replay partial git or file operations.

Macroscope summarized 0250f02.

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f38d45d4-abf4-498e-bf1f-f44233f84eeb

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One finding on the new handoff error contract; the migration, its tests, and the bundle schemas otherwise follow the Effect conventions (subpath namespace imports, no service-instance injection, no ManagedRuntime/runPromise, no Layer.succeed shortcuts).

Posted via Macroscope — Effect Service Conventions

Comment thread packages/contracts/src/orchestrationV2.ts Outdated

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Effect service conventions review of the two new service modules. Three findings, all in apps/server/src/orchestration-v2/. Earlier feedback on OrchestrationV2HandoffError.message still stands and is not repeated here.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread packages/contracts/src/orchestrationV2.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
@Bil0000 Bil0000 changed the title feat(contracts): thread handoff bundle contract feat(orchestrator): move a running thread between devices Aug 6, 2026
Comment thread packages/client-runtime/src/state/threadHandoffTransfer.ts
Comment thread apps/web/src/components/ThreadHandoffDialog.tsx
Comment thread apps/server/src/orchestration-v2/http.ts Outdated
Comment thread apps/server/src/orchestration-v2/http.ts Outdated
Comment thread packages/contracts/src/environmentHttp.ts
Comment thread apps/web/src/components/BranchToolbar.tsx
juliusmarminge and others added 23 commits August 7, 2026 13:17
Co-authored-by: codex <codex@users.noreply.github.com>
- Document the target orchestration graph, IDs, lifecycles, and capability model
- Add Codex app-server probe fixtures and update the probe test harness
- Introduce orchestration v2 service interfaces and error types
- Add replay runtime, fixtures, and integration coverage
- Update shared contracts and probe transcripts

Co-authored-by: codex <codex@users.noreply.github.com>
- Add Codex adapter and replay harness wiring
- Introduce in-memory orchestration projections and provider registry
- Expand orchestration contracts for turn and runtime events
Co-authored-by: codex <codex@users.noreply.github.com>
- Add context transfer IDs, schemas, and projections
- Support cheap fork creation and Codex native fork rollback
- Cover fork idempotency and replay behavior in tests
- Track remaining projection, context transfer, rollback, capability, and subagent work
- Clarify current V2 baseline and debugger-only follow-ups
- Map fork and merge-back turns into stored handoffs and transfer resolutions
- Add shell snapshot projection support plus coverage tests
- Update replay fixtures and web contracts for the new turn flow
Co-authored-by: codex <codex@users.noreply.github.com>
- Move Codex replay recording into `apps/server`
- Add Claude Agent SDK replay fixtures and test harness
- Update orchestration-v2 fixture scenarios and docs
- Move Claude provider runtime logic into its own module
- Share the SDK query runner between live and replay paths
- Add replay driver error wrapping for unexpected failures
- persist the selected model on run records
- surface run model selection in the debug UI
- update replay fixtures and contracts for the new field
- Record Claude SDK transcripts across multiple prompts and restart/query modes
- Add approval and tool-call replay coverage for new orchestration fixtures
- Update Claude adapter testkit to model open/prompt/permission frames
- Derive Claude SDK query options from runtime policy
- Add read-only replay fixture and policy mapping tests
- Reuse shared approval-policy fixtures across orchestrator tests

Co-authored-by: codex <codex@users.noreply.github.com>
- add active steering and interrupt-restart replay fixtures
- update Claude adapter/orchestrator turn handling for steering
- refresh replay and integration test coverage
- add interrupt and mid-tool replay fixtures for Claude and Codex
- log Claude Agent SDK protocol frames to native event traces
- project Codex commandExecution start events into orchestration updates
- Map Cursor SDK agents and runs to V2 thread and turn lifecycles
- Update MCP capability, tool, and testing guidance for SDK-based injection
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Record provider fixtures for continued forks and sibling merge-backs, exercise context survival end to end, and stop completed run listeners from consuming later provider events.

Co-authored-by: codex <codex@users.noreply.github.com>
Allow a pending merge-back to be consumed while changing providers and deliver both the provider-switch history and fork delta to the selected provider.

Co-authored-by: codex <codex@users.noreply.github.com>
Send full canonical target-thread context when a provider switch consumes a merge-back, and cover returning to an existing provider thread after a cross-provider fork.

Co-authored-by: codex <codex@users.noreply.github.com>
… chevron

Archiving a departed copy on completion pushed the duplication into the
archive list instead of removing it. The archive step is gone: the sidebar's
one-row rule already hides a residual copy while its live peer is visible,
hop revival reuses the pair instead of minting new threads, and the copy
surfaces only when the owning device is offline — the safety net working as
intended, with no hidden state and nothing to lose. Arrivals still clear
archivedAt, which heals copies archived by the previous build.

The workspace row's kind hint now absorbs the panel row's slack, parking the
chevron and divider at the right edge like every other row.
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/Orchestrator.ts
Bil0000 added 3 commits August 7, 2026 21:23
The follow-through after a push lived in the thread view that started it —
which does not survive its own departure: the thread drops out of the
sidebar, the selection fallback swaps the route, and the remount destroyed
the parked destination before it could fire. Pulls worked only because the
destination was the thread already on screen. The pending navigation now
lives in a module store and the sidebar — which survives the churn —
carries the follow-through.

The dialog and banners stop calling the residual copy read-only: the user
can work with the thread from any device; what moves is where it runs.
Clearing the handoff link when a thread came home erased its memory: after
push, pull, push the pair showed no 'moved from' and offered send verbs
where pull-back belonged. Every arrival now keeps a 'here' link — it is
provenance, not a lock; only 'away' restricts anything — so the moved-from
label and the pull-back framing survive any number of hops.
Deleting the visible half of a pair un-hid the residual copy — the
offline-peer safety rule saw its peer disappear — so the user deleted the
same thread twice. Delete, archive and unarchive now mirror to the handoff
peer, best effort: an unreachable peer keeps its copy until that device is
next seen, and a failure on the peer never blocks the local action.

Organization state (settle, pin, snooze) stays local by design: the hidden
half is not in any list, so those never needed doing twice. Renames
propagate structurally instead — every arrival takes the carried title, so
a name given on either side reaches the pair at the next hop.
Comment thread apps/web/src/hooks/useThreadActions.ts
- Constrain ThreadHandoffId to url- and ref-safe characters
- Reject part writes that would exceed the payload ceiling
- Guard depart while a turn is active; refuse mutations on away threads
- Recover handoffs interrupted mid-apply on startup
- Surface terminal-history rename failures as warnings
- Harden transfer state machine: release lock on every pre-receive failure,
  never release after the destination applied
- Fix move-target filtering and move: prefix collision in the env selector
- Assorted correctness fixes from review on git, service and client layers

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two service-shape conventions are still outstanding after the latest fixes. Previously reported findings that remain unchanged (the handoffError construction helper in ThreadHandoffService.ts and the required free-form message field on OrchestrationV2HandoffError in packages/contracts/src/orchestrationV2.ts) are not re-commented here.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/Orchestrator.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
- Refuse a handoff whose untracked files would overwrite tracked files
  in the receiving worktree, mirroring the patch-conflict refusal
- Isolate the caller's onProgress callback so a throwing callback can
  never be misread as a transfer failure after the destination is live
- Remove the now-unused ThreadHandoffGitShape export

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Effect service conventions review of the thread-handoff services. One new finding on the git service's error modeling; four findings from earlier runs are still present in this head commit and are listed in the check summary rather than re-commented.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts Outdated
- Persist pre-tag, stash and apply directory on the hop row before the
  patch lands, so startup recovery can roll back a first arrival that
  died before its thread projection existed (migration 048)
- Extend the rollback handler over the whole apply section and make
  undo clean up extracted untracked files, attachments and terminal
  logs, not just git state
- Refuse a late abort once complete recorded the peer thread
- Give the untracked-file collision refusal its own tagged error
- Inline the service shapes per Effect service conventions
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/Orchestrator.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts Outdated
- Persist the root stash and directory in their own columns (migration
  049) so a crash with both trees dirty restores each from its own stash
- Clear rollback metadata when a retried handoff reuses its row, so
  recovery never acts on a previous attempt's tag or stash
- Remove staged parts when prepare fails after staging
- Vet extracted untracked files in a staging directory against tracked
  paths, immune to newline-embedded names, before moving them in
- Refuse a late complete that would repoint an already-recorded peer
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts Outdated
…rees

- A retried first arrival reuses the thread id its lineage row already
  recorded, so deterministic event ids fill the same thread in instead
  of dedup-dropping against a fresh one
- Pop the stashes a successful apply took (root and worktree, each
  against its own directory); a conflicted pop keeps the stash, keeps
  its persisted ref and logs which ref to pop by hand
- A worktree created by a failed hop is removed in undo, not left
  holding the branch
- Include symlinks in the untracked-collision check
Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts Outdated
Bil0000 added 3 commits August 8, 2026 15:51
git stash pop only accepts a reflog entry, but every hop persists the
stash commit's sha — the reflog slot goes stale as soon as another stash
lands. Every restore path silently failed: a hop into a dirty checkout
stranded the receiver's uncommitted work in the stash forever. popStash
now applies the sha directly and drops whichever reflog slot holds it.

New coverage (real temp repositories, live git):
- extract refusal on tracked-file and symlink collisions, staging cleanup
- stash round trip by sha, bundle creation with unresolvable exclusions,
  worktree add/remove lifecycle
- receive integration: advance into a dirty checkout returns the
  receiver's edits and clears the stash columns; failed applies roll
  back the branch and remove the worktree they created; a retried first
  arrival converges on the recorded thread id; startup recovery pops
  root and worktree stashes each in their own checkout; a failed prepare
  leaves nothing staged
- undo and startup recovery reset-to-pre-tag only the apply checkout;
  when the thread lives in a worktree the repository root gets its stash
  popped but is never hard-reset to a branch tip it does not own
- a branch attached in a worktree this hop created is restored to its
  old tip when the worktree is removed on rollback
- a clone-from-bundle failure removes the partial clone (never a
  pre-existing directory) so a retry finds the target free
- extraction staging is cleared before reuse so a retried archive never
  sweeps a previous attempt's leftovers into the worktree

Both rollback tests were confirmed red against the previous code.
- OrchestrationV2HandoffError carries structured reason + detail and
  derives message from them, matching the other tagged errors
- Drop the handoffError factory; each failure boundary constructs the
  error itself, keeping asHandoffError as the one pass-through mapper
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Bil0000 added 2 commits August 8, 2026 17:31
The created worktree lived only in memory: a crash after addWorktree
left it holding the branch, and every retry derived the same path and
failed on the occupied directory. Migration 050 persists the flag on
the hop row; recovery now removes the worktree, restores the branch to
its pre-hop tip from the recorded tag, and a retry finds the path free.
Covered by an integration test against a real repository.
CI enforces preferSchemaOverJson; the created-worktree recovery path and
its test read the stored manifest with JSON.parse. Both now go through
Schema.fromJsonString(OrchestrationV2HandoffBundleV1), the same codec
that wrote the column.
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.ts

@macroscopeapp macroscopeapp Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the new Effect service modules (ThreadHandoffGit, ThreadHandoffService) and their call sites against the service conventions. The service definitions, make/layer exports, dependency acquisition and error modelling all look consistent with the conventions; the remaining findings are service-boundary imports in the two new test files that alias layer and erase the module namespace.

Posted via Macroscope — Effect Service Conventions

Comment thread apps/server/src/orchestration-v2/ThreadHandoffGit.test.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.receive.integration.test.ts Outdated
Comment thread apps/server/src/orchestration-v2/ThreadHandoffService.receive.integration.test.ts Outdated
- An unreadable attachments directory now fails the prepare instead of
  silently shipping a conversation whose attachments never travelled;
  a missing directory still just means no attachments
- recordHop keeps rollback metadata when the same attempt records its
  arrival; startup recovery finishes the stash pops for an arrived hop
  that stopped before handing the receiver's changes back
- Import the handoff modules as namespaces in the new tests, matching
  every production consumer
@Bil0000
Bil0000 marked this pull request as ready for review August 8, 2026 18:12
juliusmarminge and others added 2 commits August 10, 2026 03:14
- Keep success feedback visible in the Git action control for 10 seconds
- Move the running elapsed timer into the panel menu slot
@juliusmarminge
juliusmarminge force-pushed the t3code/codex-turn-mapping branch from 22bd872 to a27c1cc Compare August 10, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants