feat(orchestrator): Surface waiting background work

[mwolson]

Summary

• Show a Waiting state after a root turn settles while finite provider
  background work remains.
• Reuse projected command, dynamic-tool, and subagent lifecycle for Codex,
  Grok, and Claude subagents.
• Project Claude's SDK background-task roster for opaque work such as
  background Bash.

Relationship to #4255

This is the orchestrator-v2 counterpart to Theo's
#4255. That PR implements the
current main orchestration session path. This PR carries the equivalent
provider-thread and projection contract for orchestrator v2, plus the complete
v2 web UI: a sidebar Waiting pill and a waiting timeline row.

No follow-up frontend plumbing is expected. Once this PR lands on
t3code/codex-turn-mapping, Waiting should work end to end on orchestrator v2
without depending on #4255. If #4255 is later back-merged from main, its
shared-component edits may need conflict resolution, but not another product
feature PR.

Problem and Fix

Problem and Why it Happened	Fix
A completed root turn can still own finite background work, but the v2 UI had no normalized post-settlement signal and could look idle.	Derive normalized pending background work from existing projected turn items and Claude's authoritative SDK task roster.
Closing the provider event subscription as soon as a root run settled could miss the final task-roster update.	Keep the subscription open while that provider thread still has finite background work, then persist the empty roster that clears Waiting.
A session-wide pending-work probe could couple unrelated provider threads on a shared runtime.	Use a thread-scoped probe for run ingestion while retaining the session-wide probe for idle release.
Existing stored thread shells do not contain the new field.	Decode pendingBackgroundTasks with an empty-array default for backward compatibility.

Defensive Fixes

Problem and Why it Happened	Fix
Claude SDK roster levels and completion edges can arrive in either order.	Track authoritative roster state separately from wake eligibility and replay classification so Waiting clears without losing exactly-once continuation.
Replacing a Claude query kills the old CLI process before it can clear its roster.	Clear the dead thread's process-scoped roster and wake state, preserve only explicitly classified buffered Bash notifications for same-thread replacement, and keep buffered subagent notifications on the subagent lifecycle.
A Grok subagent can outlive the root turn through adapter carryover.	Include live carryover subagents in the ACP pending-work probe so idle release cannot kill work that is still projected as Waiting.
Persisted background state can survive a process restart without a provider capable of completing it.	Clear persisted rosters, cancel stale background-capable items, and attribute recovery events to the run, subagent, or matching provider thread.

Validation

• vp test run on the focused server, contract, shared, client-runtime, and
  web logic suites: 270 tests passed across ten files.
• Typechecks passed for server, web, shared, contracts, and client-runtime.
• Focused formatting passed for all 22 changed files, and git diff --check
  passed.
• Real-provider experiment packs passed for Claude background Bash and
  subagents, Codex background exec and subagents, and Grok continuations and
  subagents.
• Isolated integrated web verification with a real Claude background Bash task:
  the timeline row and sidebar pill appeared after root settlement, then both
  cleared after task completion.
• Patch application check passed on fix(orchestrator/grok): Prevent spurious wake run after in-turn monitors #4218. fix(orchestrator): Preserve claude/codex post-interrupt recovery state #4229 has an expected
  RunExecutionService overlap that will need conflict resolution if the
  branches are combined.

Depends on #4193. This change does not semantically depend on #4218 or #4229.

Note

Surface pending background work in the chat timeline and sidebar status

• Adds a pendingBackgroundTasks field to provider thread and thread shell contracts, derived from provider-thread rosters and nonterminal turn items after a run settles.
• Introduces derivePendingBackgroundWork and formatPendingBackgroundWorkLabel utilities in packages/shared to compute and format the task list.
• Adds a waiting-background row type to the chat timeline that displays an animated indicator with a label when background tasks are pending but the run is not actively working.
• Adds a pulsing "Waiting" status pill in the sidebar when a thread has pending background tasks.
• Extends ClaudeAdapterV2 to track background task rosters per native thread using snapshot (background_tasks_changed) and incremental (task_started/task_notification) messages, and emit provider_thread.updated events with pendingBackgroundTasks on turn settlement.
• Extends ProviderRuntimeRecoveryService to cancel stale background-capable turn items and clear provider-thread rosters on startup recovery.
• Risk: the root subscription now stays open after run completion while any provider thread reports pending background work, which changes stream lifecycle behavior.

📊 Macroscope summarized e46af0a. 11 files reviewed, 0 issues evaluated, 0 issues filtered, 0 comments posted

🗂️ Filtered Issues

No issues evaluated.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: d4f64e5c-d2ae-4081-805b-f2e6cb35b41a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

🟡 Medium

t3code/apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts

Line 304 in e46af0a

const recoveredNonterminalRunIds = new Set(runs.map((run) => run.id));

isBackgroundCapableTurnItemType causes replayable waiting runs to have their background-capable turn items (command_execution, dynamic_tool, subagent) cancelled, even though those runs were intentionally skipped from terminalization because a pending/running checkpoint effect still exists. When a waiting run has a replayable checkpoint, reconcileProjection omits it from runs, so its ID is absent from recoveredNonterminalRunIds. The later stale-item loop then matches that run's items via isBackgroundCapableTurnItemType + isNonterminalTurnItemStatus and emits turn-item.updated cancellation events, destroying items that belong to a still-recoverable run and checkpoint. The stale-item loop needs to also exclude turn items whose runId belongs to a replayable waiting run, not just those already in recoveredNonterminalRunIds.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/ProviderRuntimeRecoveryService.ts around line 304:

`isBackgroundCapableTurnItemType` causes replayable `waiting` runs to have their background-capable turn items (command_execution, dynamic_tool, subagent) cancelled, even though those runs were intentionally skipped from terminalization because a pending/running checkpoint effect still exists. When a `waiting` run has a replayable checkpoint, `reconcileProjection` omits it from `runs`, so its ID is absent from `recoveredNonterminalRunIds`. The later stale-item loop then matches that run's items via `isBackgroundCapableTurnItemType` + `isNonterminalTurnItemStatus` and emits `turn-item.updated` cancellation events, destroying items that belong to a still-recoverable run and checkpoint. The stale-item loop needs to also exclude turn items whose `runId` belongs to a replayable `waiting` run, not just those already in `recoveredNonterminalRunIds`.

[macroscopeapp]

🟡 Medium orchestration-v2/RunExecutionService.ts:738

After the root terminal is observed, writeIfRunCurrent is stripped from all provider_thread.updated events for the root thread — including events from a superseded attempt whose subscription stays open for background work. When that stale subscription receives a late provider_thread.updated and persists the old providerThread snapshot, it overwrites the new attempt's thread state (for example reverting status or roster cleared by the new attempt). The gate should use a settled-run-safe conditional write or field-specific merge rather than bypassing attempt ownership entirely based on rootTerminalAlreadySeen.

🤖 Copy this AI Prompt to have your agent fix this:

In file @apps/server/src/orchestration-v2/RunExecutionService.ts around line 738:

After the root terminal is observed, `writeIfRunCurrent` is stripped from all `provider_thread.updated` events for the root thread — including events from a superseded attempt whose subscription stays open for background work. When that stale subscription receives a late `provider_thread.updated` and persists the old `providerThread` snapshot, it overwrites the new attempt's thread state (for example reverting status or roster cleared by the new attempt). The gate should use a settled-run-safe conditional write or field-specific merge rather than bypassing attempt ownership entirely based on `rootTerminalAlreadySeen`.