fix(server): make remote updates rollback-safe

[t3dotgg]

Remote server updates had accumulated multiple version owners and handoff paths, so a failed candidate could leave the service pinned to a broken version.

This replaces that implementation with one stable systemd launcher that owns version selection and durable update state. Candidates are staged and checked against the exact migration manifest, complete all fallible startup before reporting prepared, then activate only after the launcher commits. Clients correlate reconnects with the launcher-generated update ID, while foreground servers keep the explicit manual-update path.

Verification

• 281 focused tests across 16 files
• Server, client-runtime, and contracts typechecks
• Targeted lint and formatting
• Production server bundle
• Standalone launcher dependency check
• Missing-database preflight check

Built with GPT-5.6-sol in the Codex harness via T3 Code.

---

Note

High Risk
Changes core boot/update infrastructure: systemd units, version pinning, DB migration gating, and process handoff—failures can brick startup or leave the service on a bad runtime until rollback.

Overview
Replaces remote self-update paths that pointed systemd at a versioned bin.mjs (and foreground respawn) with a stable service-launcher under ~/.t3/runtime, durable service-state.json, and IPC between the running server and the launcher.

Boot service now installs the launcher artifact, writes initial state, and sets ExecStart to node …/service-launcher.mjs (with KillMode=control-group). Pinned runtimes are npm-installed into a staging tree, validated (including __service-preflight against the live DB migration manifest), then atomically published; blocked preflights surface the local npx t3@… service update path instead of mutating the DB remotely.

Remote update RPC stages the target version, runs preflight on the staged entry, then requestUpdate on the launcher and returns a launcher-generated updateId for client correlation—no direct unit rewrite or detached respawn from the server process.

Startup ordering adds ServerActivation / forkParked so background reactors and side effects (runtime state, Tailscale, cloud link reconcile) wait until activation; HTTP ingress is parked behind awaitCommandReady. Launcher trial children skip WAL and migrations so preflight stays read-only.

Servers without an installed launcher-managed service can no longer self-update remotely; the error directs users to t3 service install.

^{Reviewed by Cursor Bugbot for commit ff26b5a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Make remote server updates rollback-safe via a stable systemd launcher and staged installs

• Introduces a new serviceLauncher.ts process that owns update state and child process lifecycle. The systemd unit now points to a stable launcher file instead of a versioned CLI entry, so rollbacks don't require rewriting the unit.
• pinnedRuntime.ts now stages npm installs into a temp directory, validates via a preflight check, writes a sentinel, then atomically publishes via rename. Staging is cleaned up on failure or interruption.
• A new __service-preflight CLI command checks that DB migrations and launcher protocol match before an update is committed; if migrations are required, the update is blocked with a message to run npx t3@<version> service update locally.
• The client now correlates the post-restart ready lifecycle event using a launcher-provided updateId and immediately surfaces a ServerUpdateTerminalError on rollback/failure instead of waiting for a generic reconnect timeout (extended to 4 minutes).
• Introduces a forkParked/ServerActivation pattern so background fibers (tailscale, cloud link reconcile, runtime state persistence) are held until the activation gate is signaled, preventing races during startup.
• Risk: removePinnedRuntimeInstallation is removed as a public API; callers that relied on it will break at compile time.

^{Macroscope summarized ff26b5a.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b7f98895-ff4a-4cf2-abb4-dadad2713869

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Reviewed the new service modules and their call sites against the Effect service conventions. Six findings, mostly around dependency acquisition (module globals / global process instead of the Effect environment) and error modelling (a step string used to drive caller-visible messages). Details inline.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Approvability

Verdict: Needs human review

Diff is too large for automated approval analysis. A human reviewer should evaluate this PR.

^{You can customize Macroscope's approvability policy. Learn more.}

[macroscopeapp]

Reviewed the Effect service changes in this PR (launcher client, shutdown service, self-update, pinned runtime, startup/activation wiring). The prior findings on the module-global shutdown channel, the free-form launcher error, duplicated launcher-context reads in the SQLite layer, and the double ServiceLauncherClient layer are all resolved. Three smaller convention issues remain in the new code.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Three findings, all in the rewritten test files for the new service-launcher update path. The production service modules (serviceLauncherClient.ts, serverShutdown.ts, selfUpdate.ts, Sqlite.ts) now follow the tag/make/layer shape, acquire their dependencies from the environment, and the earlier findings about the module-global shutdown channel, the free-form launcher error, the duplicated launcher client, and the catchTags recovery are resolved.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Two findings in apps/server/src/cloud/bootService.ts. Previously reported issues that were addressed (namespace import for serviceLauncherClient, Effect.catchTags in pinnedRuntime.ts, structured ServiceLauncherClientError/PinnedRuntimePreflightBlockedError, launcher client acquired from the environment and composed once) are not repeated.

^{Posted via Macroscope — Effect Service Conventions}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d3b4919. Configure here.}

[macroscopeapp]

Two remaining convention issues in the newly touched service code; earlier findings on selfUpdate.ts, pinnedRuntime.ts, and serviceLauncherClient.ts look addressed.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Reviewed the changed Effect service code. The service/layer/error structure of the new launcher modules looks consistent with the conventions (namespace imports at service boundaries, Context.Service tags with inline interfaces, Schema.TaggedErrorClass failures with structural attributes, catchTags for known tags).

Findings in this revision are limited to documentation/invariant loss during the move: several comments describing behavior that still exists were dropped from bootService.ts, pinnedRuntime.ts, and the version rule moved into serviceProtocol.ts.

Earlier findings from previous revisions of this PR are still open and are intentionally not re-posted: the named ServiceLauncherClient import in persistence/Layers/Sqlite.ts, the node:fs/promises-based writeDurably in bootService.ts that bypasses the acquired FileSystem/Path services, and the collapse of every PinnedRuntimeInstallError into the generic BootServiceInstallError.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

One convention finding on the new launcher client service module. Everything else in the changed Effect service scope (namespace imports at service boundaries, make/layer shape for bootService/selfUpdate/serverRuntimeStartup, dependency acquisition through FileSystem/Path/ProcessRunner/HostProcess*, structured tagged errors) looks consistent with the conventions.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Reviewed the Effect service changes in this PR (launcher client, boot service, pinned runtime, activation/startup wiring, persistence layer). Prior findings on namespace imports, Path/FileSystem acquisition in bootService, the blocked-preflight error tag, and structured error context all look addressed. Three remaining items below.

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Two error-modeling findings in the new serviceLauncherClient.ts. Everything else in the reviewed scope (the BootService/ServerSelfUpdate/ServiceLauncherClient definitions, make/layer exports, namespace imports, and dependency acquisition through the Effect environment) matches the conventions.

^{Posted via Macroscope — Effect Service Conventions}