🤖 feat: add workflow agent graceful timeouts

[ThomasK33]

Summary

Adds explicit graceful timeouts for workflow-owned agent(...) steps: authors can provide timeout.softMs and timeout.graceMs, the runner records durable timeout metadata/events, TaskService soft-prompts children for final reports, and hard timeouts fail the workflow step while preserving diagnostic artifacts.

Background

Long-running workflow sub-agents could previously hit only a waiter timeout, leaving workflow orchestration without a durable soft-finalization path. This change lets workflows recover partial-but-useful agent findings before escalating to a hard timeout.

Implementation

• Added timeout parsing and validation to workflow agent specs with explicit softMs/graceMs requirements.
• Persisted timeout metadata and emitted timeout phase events for soft timeout, prompt sent, recovered, and hard timeout states.
• Added TaskService adapter APIs for idempotent final-report prompting and hard-timeout failure.
• Wired the timeout state machine through workflow agent wait paths, including execution-start deadline persistence, retry-safe timeout metadata reset, late-report hard-timeout rechecks, missing-task restart behavior, terminal task events for non-timeout wait failures, pipeline hard-timeout event de-duplication, and pipeline sibling interruption on hard timeouts.
• Hardened TaskService timeout finalization so accepted finalization turns are not interrupted on retry, prompts start in the background, the workflow waits for prompt acceptance before starting grace, prompt-sent metadata waits for durable acceptance, tokens persist only after accepted prompts, hard timeouts clear queued finalization prompts before aborting streams, and plan agents finalize with propose_plan instead of agent_report.
• Added UI timeline rendering, disabled Chromatic-only timeout stories, and workflow authoring docs.

Validation

• MUX_ESLINT_CONCURRENCY=1 make static-check
• bun test ./src/node/services/workflows/WorkflowRunner.test.ts ./src/node/services/taskService.test.ts (311 pass)
• Earlier validation: bun test ./src/node/services/workflows/WorkflowRunner.test.ts ./src/node/services/workflows/WorkflowRunStore.test.ts ./src/node/services/taskService.test.ts ./src/browser/hooks/useCoderWorkspace.test.ts ./tests/ui/storybook/budget.test.ts (335 pass, before final hardening rounds)

Risks

This touches workflow orchestration, task lifecycle handling, and workflow run UI. The highest-risk areas are races between late reports and hard timeout cleanup, execution-start deadline persistence for queued tasks, queue cleanup during hard timeout, finalization token idempotency, accepted-finalization retry behavior, prompt-acceptance/grace timing, plan-agent completion-tool selection, pipeline sibling cleanup, and resume/retry behavior while a child is finalizing; targeted tests cover those paths.

---

📋 Implementation Plan

Implementation Plan: Graceful Timeouts for Workflow-Owned Sub-Agents

1. Goal

Add a workflow authoring capability for workflow-owned agent(...) steps to enforce a time budget without immediately throwing away useful child-agent work.

The desired behavior is:

1. A workflow starts a sub-agent step normally.
2. If the step exceeds a soft timeout, Mux asks the child to stop starting new work and produce an agent_report with whatever it has.
3. Mux waits a short grace period.
4. If the child reports during grace, the workflow consumes the report normally and the UI marks the step as “completed after timeout finalization”.
5. If the child still does not report, Mux performs a hard timeout: interrupt/terminate the child task subtree and fail the workflow step with a durable timeout error.

This plan intentionally does not add raw workflow setTimeout / setInterval. Time remains a durable runtime concern, not a process-local JavaScript callback.

2. Evidence and repo context

Investigation used dedicated Explore agents plus narrow verification of the stream interruption surface.

Verified repo facts used by this plan:

• Workflow author calls to agent(prompt, options) are compiled into __muxAgent(...), which builds a host spec and calls __workflowAgent(spec) (src/node/services/workflows/WorkflowRunner.ts, especially compileWorkflowSource, __muxAgent, and the __workflowAgent registration).
• WorkflowAgentSpec is parsed/validated in parseWorkflowAgentSpec(...); agent steps require stable IDs and are replay-keyed via hashWorkflowStepInput(spec.id, spec).
• Completed workflow steps are durable-cached in WorkflowRunStore; resume paths reuse existing started task IDs instead of spawning duplicates.
• WorkflowTaskServiceAdapter.waitForAgentTask(...) delegates to TaskService.waitForAgentReport(...) and already passes a timeoutMs, but today that timeout is only a waiter timeout: it rejects the waiter with “Timed out waiting for agent_report” and does not stop the child task.
• TaskService.waitForAgentReport(...) starts its timeout only once the child task reaches running; queued/starting time does not consume the execution timeout budget.
• TaskService already sends synthetic recovery prompts, notably promptTaskForRequiredCompletionTool(...), with tool policy requiring agent_report/propose_plan and recovery-attempt limits.
• A prompt cannot bypass an active stream/turn; while a child session is busy, workspaceService.sendMessage(...) queues/rejects based on options. Soft timeout therefore needs to stop/settle the active child turn before the finalization prompt can run.
• Stream interruption supports stopStream(workspaceId, { soft, abandonPartial, abortReason }); soft: true cancels at a block boundary, and existing task interruption paths use abandonPartial: false to preserve partial output or abandonPartial: true for hard termination.
• Workflow UI renders workflow run events in src/browser/features/Tools/WorkflowRunToolCall.tsx; task events are coalesced by step/task and generic task status is a string, so workflow-specific statuses such as finalizing can be displayed without necessarily adding a global workspace task status.
• Workflow docs live in src/node/builtinSkills/workflow-authoring.md; current docs explicitly say workflow conductors cannot use timers, Date, or Math.random.

3. Recommended approach and LoC estimates

Recommended v1: per-agent graceful timeout object

Expose only on workflow-owned agent(...) steps:

const result = agent("Investigate the issue and report findings.", {
  id: "investigate",
  schema: investigationSchema(),
  timeout: {
    softMs: 20 * 60_000,
    graceMs: 2 * 60_000,
    finalInstructions:
      "Stop starting new work. Summarize completed work, remaining uncertainty, files touched, and validation evidence, then call agent_report.",
  },
});

Net product LoC estimate: ~900–1,400 LoC.

Breakdown, product code only:

• Workflow spec parsing/runtime orchestration: ~300–450 LoC.
• TaskService soft-finalization and hard-timeout APIs: ~300–500 LoC.
• Workflow event/type/schema/UI/doc strings: ~150–250 LoC.
• CLI/tool status polish if needed: ~50–100 LoC.

Tests, stories, and docs are excluded from the product LoC estimate.

Simpler fallback: hard timeout only

Add agent(..., { timeoutMs }) that fails and terminates after the deadline.

Net product LoC estimate: ~250–450 LoC.

This is simpler but not recommended because it discards recoverable findings, code changes, and test evidence from long-running sub-agents.

Deferred larger design: general durable timers/cancellation scopes

Expose sleep, sleepUntil, and general withTimeout(...)/race-like cancellation scopes.

Net product LoC estimate: ~1,800–3,000+ LoC.

This should remain separate. General cancellation scopes are valuable but require durable timers, persisted cancellation tokens, branch/race semantics, and more UI surface area. Per-agent graceful timeouts are a narrower vertical slice.

4. Public API design

4.1 Author-facing API

Add a timeout object to agent(prompt, options).

type WorkflowAgentTimeoutOptions = {
  /** Time budget after the child task starts running; queued/starting time does not count. */
  softMs: number;

  /** Time allowed for finalization after the soft timeout fires. Required; no runtime default. */
  graceMs: number;

  /** Optional extra instructions appended to the standard finalization prompt. */
  finalInstructions?: string;
};

Recommended explicitness and validation:

• Mux must not define semantic default timeout values. The workflow author owns all time-budget decisions.
• Validation min/max bounds are safety guardrails only; they must not be described or implemented as recommended/default durations.
• softMs is required when timeout is present.
• graceMs is required when timeout is present; omitting it is a parse/runtime error, not an implicit default.
• softMs and graceMs must be finite positive integers.
• Enforce conservative minimums to avoid accidental zero-timeout flakiness:
  • softMs >= 1_000
  • graceMs >= 1_000
• Enforce conservative maximums for v1 to prevent accidental effectively-unbounded tasks:
  • softMs <= 24h
  • graceMs <= 1h
• finalInstructions, if present, must be a non-empty string under a bounded length such as 4,000 characters.
• Normalize the explicitly provided timeout object before hashing the step input so replay identity is explicit and stable; do not inject default timeout durations.

4.2 Behavior guarantees

For a timeout-configured agent step:

• The soft budget starts when the child task reaches running, matching current waitForAgentReport(...) timeout behavior; queued/starting time must not count.
• The runtime must persist the child execution start/deadlines (executionStartedAt, softDeadlineAt, hardDeadlineAt) once the task reaches running, instead of deriving deadlines solely from the workflow step startedAt.
• If the child reports before soft timeout, no timeout events are emitted.
• If soft timeout fires, the child receives one durable/synthetic finalization attempt.
• Finalization prompt delivery must be idempotent across crashes by using a stable timeout token derived from { runId, stepId, inputHash, taskId, softTimedOutAt }.
• If the child reports valid output during grace, the workflow step completes successfully and downstream workflow code receives the same shape it would receive from a normal agent step.
• If hard timeout fires, the step fails with a clear typed timeout error.
• A late report must never overwrite an already-recorded terminal hard-timeout failure; conversely, an already-persisted report wins over a racing hard timeout (“report monotonicity”).
• Resume/retry must not duplicate the finalization prompt if it was already sent.
• Deadline enforcement must survive app/backend restart, not only an in-memory active runner wait.

4.3 Non-goals for v1

• No raw setTimeout, setInterval, or Date inside workflow scripts.
• No general Promise.race/withTimeout cancellation scopes.
• No setInterval-style periodic workflow behavior.
• No partial structured-output mode by default. Schema-backed agents still must produce schema-valid agent_report output for the step to complete.
• No implicit timeout defaults at any API layer; absence of timeout means no semantic workflow-agent timeout, and presence of timeout requires all durations explicitly.
• Existing internal waiter/watchdog limits may remain as infrastructure safeguards, but they must not trigger graceful-timeout behavior or be documented as workflow API defaults when agent.timeout is absent.
• No new global workspace task status unless implementation proves it is necessary; prefer workflow-run timeout events plus existing task statuses (running, awaiting_report, interrupted, reported).

5. Data model and durable state

5.1 Extend WorkflowAgentSpec

In src/node/services/workflows/WorkflowRunner.ts:

export interface WorkflowAgentTimeoutSpec {
  softMs: number;
  graceMs: number;
  finalInstructions?: string;
}

export interface WorkflowAgentSpec {
  id: string;
  prompt: string;
  // existing fields...
  timeout?: WorkflowAgentTimeoutSpec;
}

timeout must be part of the replay input hash. Changing timeout values should create a new step identity because it changes orchestration behavior.

5.2 Extend workflow step records

Add optional timeout metadata to WorkflowStepRecordSchema in src/common/orpc/schemas/workflow.ts and corresponding inferred types:

timeout?: {
  /** First instant the child task was observed running; queued/starting time is excluded. */
  executionStartedAt?: string;
  /** Persisted when execution starts so resume/restart does not recompute differently. */
  softDeadlineAt?: string;
  /** Persisted once soft timeout fires; normally softTimedOutAt + graceMs. */
  hardDeadlineAt?: string;
  softTimedOutAt?: string;
  /** Stable idempotency token for the synthetic finalization prompt. */
  finalizationToken?: string;
  finalizationPromptSentAt?: string;
  hardTimedOutAt?: string;
};

Rationale:

• Existing status: "started" | "completed" | "failed" | "interrupted" can remain unchanged.
• A step can be started while it is in the grace period.
• Persisting execution/deadline timestamps on the step record makes resume idempotent and prevents queued/starting time from accidentally counting against the child.
• recordStepCompleted(...) and recordStepFailed(...) must preserve existing timeout metadata instead of clobbering it; timeout metadata updates must be atomic/lease-guarded and safe against completion races.

5.3 Add workflow timeout events

Add a discriminated event type to WorkflowRunEventSchema:

{
  type: "timeout";
  sequence: number;
  at: string;
  stepId: string;
  taskId: string;
  phase: "soft" | "finalization_prompt_sent" | "recovered" | "hard";
  details?: JsonValue;
}

Use cases:

• UI can render an explicit “timeout finalization” timeline row.
• Logs/debugging can distinguish a recovered soft timeout from a terminal hard timeout.
• Tests can assert durable transition order without inspecting private TaskService details.

5.4 Optional workflow task events for coalesced UI rows

Append existing generic task events with workflow-specific status strings:

• finalizing when soft timeout fires and finalization is requested.
• timed_out when hard timeout fires.
• Existing schema accepts task status as z.string().min(1), so this does not require a new global task-status enum.

Do not initially add finalizing to persisted workspace taskStatus unless required. The child workspace can stay running, awaiting_report, or interrupted; the workflow run event log carries the workflow-specific timeout state.

6. Runtime implementation plan

Phase 1 — Types, parsing, and author API

Files:

• src/node/services/workflows/WorkflowRunner.ts
• src/node/workflowRuntime/workflowRuntimeStdlib.js only if helper docs/utilities need updates
• src/node/builtinSkills/workflow-authoring.md
• Relevant tests in src/node/services/workflows/WorkflowRunner.test.ts

Steps:

1. Update __muxAgent(prompt, options) inside compileWorkflowSource(...) to accept options.timeout and place it on the host spec.
2. Add parseWorkflowAgentTimeoutSpec(raw) near parseWorkflowAgentSpec(...).
3. Validate and normalize explicitly supplied softMs, graceMs, and finalInstructions; reject missing durations instead of applying defaults.
4. Include normalized timeout in WorkflowAgentSpec before hashing.
5. Add unit tests that invalid timeout objects fail fast during workflow execution with clear errors.
6. Cover explicit-duration validation cases:
   • timeout: { softMs } fails because graceMs is missing;
   • timeout: { graceMs } fails because softMs is missing;
   • absence of timeout preserves existing no-graceful-timeout behavior.

Quality gate:

• Run targeted workflow runner tests for parser failures and a normal no-timeout agent step to ensure existing behavior is unchanged.

Phase 2 — Durable step timeout metadata and events

Files:

• src/common/orpc/schemas/workflow.ts
• src/common/types/workflow.ts if explicit helper types are needed
• src/node/services/workflows/WorkflowRunStore.ts
• src/node/services/workflows/WorkflowRunStore.test.ts
• src/node/services/workflows/WorkflowRunner.ts

Steps:

1. Add optional timeout metadata to WorkflowStepRecordSchema.
2. Add the new timeout event variant to WorkflowRunEventSchema.
3. Add store APIs such as:
   • recordStepExecutionStarted(runId, { stepId, inputHash, taskId, executionStartedAt, softDeadlineAt })
   • recordStepSoftTimedOut(runId, { stepId, inputHash, taskId, softTimedOutAt, hardDeadlineAt, finalizationToken })
   • recordStepFinalizationPromptSent(runId, { stepId, inputHash, taskId, finalizationToken, finalizationPromptSentAt })
   • recordStepHardTimedOut(runId, { stepId, inputHash, taskId, hardTimedOutAt })
4. Keep updates lease-guarded through WorkflowRunner.appendEvent(...) / store calls with expected lease owner where applicable.
5. Ensure metadata merge semantics: completion/failure writes must preserve prior timeout fields, and timeout metadata writes must not overwrite a concurrently completed step.
6. Ensure old workflow run records without timeout metadata still parse.

Quality gate:

• Run WorkflowRunStore.test.ts targeted tests.
• Add schema round-trip tests for timeout events and timeout metadata.

Phase 3 — TaskService graceful-finalization API

Files:

• src/node/services/workflows/WorkflowTaskServiceAdapter.ts
• src/node/services/taskService.ts
• src/node/services/taskService.test.ts
• Possibly src/node/services/streamManager.ts / src/node/services/aiService.ts only if a thin wrapper or abort reason typing is missing

Add WorkflowTaskAdapter methods:

requestAgentFinalReportForTimeout(
  taskId: string,
  options: {
    workflowRunId: string;
    stepId: string;
    inputHash: string;
    finalizationToken: string;
    finalInstructions?: string;
  }
): Promise<"prompted" | "already_reported" | "not_active">;

failAgentTaskForHardTimeout(
  taskId: string,
  options: {
    workflowRunId: string;
    stepId: string;
    inputHash: string;
    reason: string;
  }
): Promise<void>;

TaskService behavior for requestAgentFinalReportForTimeout(...):

1. Acquire the per-workspace/task event lock before inspecting or mutating child state.
2. Check for an already-persisted report first; return already_reported if present.
3. Persist or check the finalizationToken in child task metadata before sending so a crash/retry can deduplicate the same timeout prompt.
4. If the task no longer exists or is already terminal, return not_active or surface the existing terminal failure.
5. If the child is actively streaming, call the safe interruption path with soft: true, abandonPartial: false, and timeout abort reason if available.
6. Transition or nudge the child into the same recovery lane used by existing completion-tool recovery (awaiting_report where appropriate).
7. Send a synthetic prompt requiring agent_report via tool policy only if the finalizationToken has not already been sent.
8. Reuse or mirror promptTaskForRequiredCompletionTool(...) safeguards:
   • synthetic + agentInitiated metadata,
   • required completion tool policy,
   • recovery attempt limit,
   • logging on send failure.
9. The prompt must make the intended behavior explicit:
   • time expired,
   • do not start new tool work,
   • summarize completed work and uncertainty,
   • include files/tests/evidence,
   • call agent_report now.

TaskService behavior for failAgentTaskForHardTimeout(...):

1. Acquire the task lock.
2. Check report/failure monotonicity before killing.
3. Stop active streams with hard behavior (abandonPartial: true) where appropriate.
4. Use a dedicated timeout-failure path that preserves diagnostics: do not use the deletion-style descendant termination path if it removes child workspaces/artifacts before the parent can inspect them.
5. Interrupt/fail descendant tasks so nested sub-agents do not remain orphaned, but preserve report/failure artifacts, session history, and workspace files useful for debugging.
6. Persist a terminal failure artifact with an explicit timeout error type/message if no report exists.
7. Reject pending waiters and free scheduling slots via existing cleanup paths.

Quality gate:

• Add TaskService tests for:
  • soft timeout sends exactly one finalization prompt;
  • finalization prompt uses required agent_report tool policy;
  • active stream is soft-interrupted before prompt delivery;
  • already-reported child is not prompted;
  • duplicate soft-timeout calls are idempotent;
  • hard timeout does not override an already-persisted report;
  • hard timeout fails/interrupts descendants while preserving diagnostics and releases queued siblings;
  • finalization token deduplicates prompts across repeated calls and simulated restart.

Phase 4 — WorkflowRunner timeout state machine

Files:

• src/node/services/workflows/WorkflowRunner.ts
• src/node/services/workflows/WorkflowRunner.test.ts
• src/node/services/workflows/WorkflowTaskServiceAdapter.test.ts

Implement timeout orchestration as a shared agent-wait helper used by all workflow agent wait paths: single agent(...), parallel(...), and pipeline(...) (startAgentStepWithoutWaiting / waitForAnyStartedAgentStep). Do not implement only the single-agent path unless timeout usage inside pipeline(...) is explicitly rejected for v1 with a clear parser/runtime error.

Recommended helper shape:

private async waitForAgentStepWithGracefulTimeout(
  runId: string,
  sequence: WorkflowEventSequence,
  step: {
    spec: WorkflowAgentSpec;
    inputHash: string;
    taskId: string;
    startedAt: string;
    existingStep?: WorkflowStepRecord;
    waitOptions?: WorkflowAgentWaitOptions;
    leaseGuard: WorkflowRunnerLeaseGuard;
  }
): Promise<WorkflowAgentResult>

State-machine behavior:

1. If spec.timeout is absent, use the existing wait path unchanged.
2. If a completed result already exists, return it unchanged.
3. Ensure timeout metadata contains executionStartedAt and softDeadlineAt derived from the child task's first observed running transition; do not use workflow step startedAt if it may include queued/starting time.
4. If the step has no softTimedOutAt:
   • wait for the report with timeoutMs = max(1, softDeadlineAt - now);
   • if report arrives, return it;
   • if a typed wait-timeout error/result occurs, re-check persisted report/failure before acting;
   • record softTimedOutAt, hardDeadlineAt, and finalizationToken, then append timeout + task finalizing events;
   • call requestAgentFinalReportForTimeout(...) with the finalizationToken.
5. If finalizationPromptSentAt is absent, record it once only after the adapter confirms the prompt was sent or already deduplicated by token. If the adapter says already_reported, immediately re-await/read the report.
6. Use persisted hardDeadlineAt rather than recomputing on every resume.
7. Wait for report with remaining grace.
8. If a schema-valid report arrives during grace:
   • append timeout phase: recovered event;
   • complete the step through the existing recordAgentResult(...) path;
   • preserve normal downstream return shape.
9. If grace expires:
   • re-check persisted report/failure;
   • record hardTimedOutAt and append timeout phase: hard + task timed_out events;
   • call failAgentTaskForHardTimeout(...);
   • call recordStepFailed(...) with a clear timeout message while preserving timeout metadata;
   • throw a typed timeout error so workflow failure/retry behavior remains explicit.

Validation retry interaction:

• If the child reports during grace but schema validation fails, do not reset the hard deadline.
• After a soft timeout, do not use the normal validation retry path if it would spawn a fresh child task; that defeats timeout semantics.
• Allow at most one same-child schema-correction prompt inside the remaining grace window only if it can reuse the finalizing child and the required agent_report policy.
• Otherwise fail the step with validation error and include the timeout context in the failure event/details.
• Keep the default strict: no markdown fallback for schema-backed steps in v1.

Parallel/pipeline interaction:

• Per-step timeout failures should behave like other agent failures: fail-fast should interrupt remaining siblings unless the existing parallel(...) backgrounding/abort semantics say otherwise.
• Timeout metadata and finalization should remain per step ID/task ID.
• Bulk-created parallel tasks must still record task IDs before timeout tracking starts.

Resume/restart interaction:

• On resume, inspect the existing step record:
  • no execution metadata yet: wait for or observe the child running transition, then persist executionStartedAt/softDeadlineAt;
  • execution metadata present and no soft timeout recorded: wait only until persisted softDeadlineAt;
  • soft timeout recorded, prompt not recorded: send/record finalization prompt exactly once using finalizationToken;
  • prompt recorded: wait only until persisted hardDeadlineAt;
  • hard timeout recorded/failed: replay the failure, do not re-terminate unless cleanup is explicitly idempotent and needed.
• Use WorkflowRunnerClock (nowIso, nowMs) for all calculations to make tests deterministic.
• Always re-check persisted reports before sending prompts or hard-terminating.
• Ensure deadline enforcement survives process restart: either WorkflowService must rediscover/resume active runs with pending timeout deadlines, or TaskService must persist child timeout intent/deadlines and run startup recovery. Do not rely solely on in-memory setTimeout waiters.
• WorkflowRunner should learn/persist executionStartedAt from a TaskService lifecycle callback or TaskService-owned persisted deadline metadata, not from polling unstable status snapshots.

Quality gate:

• Add WorkflowRunner.test.ts cases for:
  • normal completion before soft timeout;
  • soft timeout then successful report during grace;
  • soft timeout then hard timeout failure;
  • resume before soft deadline using persisted softDeadlineAt;
  • resume during grace using persisted hardDeadlineAt;
  • restart/recovery after grace already expired still enforces hard timeout;
  • no duplicate finalization prompt on replay because finalizationToken is deduped;
  • late report race wins if persisted before hard termination;
  • parallel sibling behavior when one lane times out;
  • pipeline timeout behavior is covered or explicitly rejected for v1.

Phase 5 — UI, CLI, and observability

Files:

• src/browser/features/Tools/WorkflowRunToolCall.tsx
• src/browser/features/Tools/WorkflowRunToolCall.test.tsx
• src/browser/features/Tools/WorkflowRunToolCall.stories.tsx
• src/browser/utils/workflowRunMessages.ts and/or common workflow run message utilities if summaries need timeout wording
• CLI/tool formatting only if child task status output changes

UI requirements:

1. Render timeout workflow events with concise labels:
   • “Soft timeout reached; requesting final report”
   • “Final report requested”
   • “Recovered during grace period”
   • “Hard timeout; child terminated”
2. Tone mapping:
   • soft/finalization events: warning tone;
   • recovered: success/normal tone;
   • hard timeout: warning/error tone consistent with existing failure styles.
3. Task rows should show finalizing or timed_out when those workflow task events are the latest event.
4. Details panel should include soft/grace durations and deadline timestamps where present.
5. Avoid adding emoji UI indicators; use existing shared icons/components if new visual indicators are necessary.
6. Verify mobile/narrow width around 375px so long task IDs/status labels do not overflow.

CLI/tool requirements:

• If no new persisted workspace task status is added, task_list/task_await do not need a finalizing status.
• If implementation introduces a real finalizing workspace task status, update active-status lists in task_await, task_list, CLI formatters, and tests so finalizing tasks remain awaitable and visible.

Quality gate:

• Add Storybook story with a run that is currently finalizing.
• Add story/test for a recovered timeout and a hard timeout.
• Validate desktop and narrow viewport story rendering.

Phase 6 — Documentation and workflow-authoring skill

Files:

• src/node/builtinSkills/workflow-authoring.md
• Any generated skill/runtime source snapshot if the repo requires regeneration for built-in skills

Update docs under agent(prompt, options):

• Explain timeout.softMs, timeout.graceMs, and timeout.finalInstructions, emphasizing that Mux provides no default durations.
• State that queued/starting time does not count.
• State that soft timeout sends a synthetic finalization prompt requiring agent_report.
• State that hard timeout fails the step if no valid report arrives during the author-specified grace period.
• Warn schema-backed agents to design schemas that can represent “partial but useful” results if authors want graceful timeout reports to be consumable.
• Keep the existing warning that workflows cannot use raw timers/Date/Math.random.

Quality gate:

• Run docs/skill tests if present, plus static checks that cover generated workflow runtime sources.

6.5 Typed timeout results/errors

Do not branch on the literal string "Timed out waiting for agent_report" in workflow timeout orchestration.

Add a typed error/result boundary, for example:

class AgentReportWaitTimeoutError extends Error {
  readonly name = "AgentReportWaitTimeoutError";
}

or a result union from TaskService/adapter wait APIs. The WorkflowRunner timeout state machine should distinguish:

• report arrived;
• typed wait timeout reached;
• task interrupted/terminated;
• terminal child failure artifact exists;
• caller/workflow abort signal fired.

This prevents graceful-timeout logic from accidentally catching unrelated errors whose message happens to include “timeout”.

7. Error messages and prompt text

Standard finalization prompt should be deterministic and concise. Suggested base prompt:

Your workflow step time budget has expired. Stop starting new work and prepare a final report now.

In your report:
- summarize work completed;
- list files changed or inspected;
- include validation/test results already obtained;
- call out uncertainty and remaining work;
- do not run additional long-running tools unless absolutely necessary to write the report.

Call agent_report now with your best available result.

If workflow author provides finalInstructions, append:

Additional workflow-specific finalization instructions:
<finalInstructions>

Hard-timeout failure message:

Workflow agent step <stepId> exceeded its soft timeout (<softMs>ms) and did not produce a valid agent_report within the grace period (<graceMs>ms).

8. Edge cases and decisions

8.1 Report arrives exactly as soft timeout fires

Decision: report wins.

Implementation: after catching a soft-timeout wait error, re-check persisted report/failure before recording soft timeout or prompting.

8.2 Report arrives exactly as hard timeout fires

Decision: persisted report wins over termination.

Implementation: acquire the task/workflow lock and re-check report before terminateAgentTaskForTimeout(...).

8.3 Soft finalization prompt send fails

Decision: treat as timeout failure only after recording enough context.

Implementation:

• record soft timeout event;
• attempt prompt;
• if send fails because task already reported, recover;
• otherwise record failure with “failed to request timeout finalization” and terminate/cleanup according to safety rules.

8.4 Child is in a long-running tool call

Decision: soft timeout should interrupt the active turn at a safe boundary before finalization prompt delivery.

Implementation:

• use stopStream(..., { soft: true, abandonPartial: false }) where possible;
• if a tool process cannot stop cooperatively, the grace period still bounds the total delay;
• hard timeout uses existing descendant termination cleanup.

8.5 Schema validation fails after grace report

Decision: strict schema remains default.

Implementation:

• do not silently return markdown for schema-backed steps;
• either use existing validation retry only if remaining grace permits, or fail with validation error plus timeout context;
• document that schemas should include partial-result fields when partial recovery matters.

8.6 Workflow run is interrupted while child is finalizing

Decision: normal workflow interruption wins.

Implementation:

• abort signal should interrupt runtime waits;
• existing descendant interruption should preserve child workspaces where possible;
• resume later uses durable timeout metadata to continue or hard-timeout based on elapsed deadlines.

8.7 Backgrounded foreground wait

Decision: timeout state is independent of foreground/background UI wait.

Implementation:

• current foreground backgrounding due to queued parent messages should not reset soft/grace clocks;
• workflow resume/background continuation must read persisted step timeout metadata.

9. Testing plan

9.1 Unit/integration tests

Run targeted tests during implementation:

• bun test src/node/services/workflows/WorkflowRunner.test.ts --filter timeout
• bun test src/node/services/workflows/WorkflowRunStore.test.ts --filter timeout
• bun test src/node/services/workflows/WorkflowTaskServiceAdapter.test.ts --filter timeout
• bun test src/node/services/taskService.test.ts --filter timeout
• bun test src/browser/features/Tools/WorkflowRunToolCall.test.tsx --filter timeout

Exact Bun filter syntax may need adjustment to match repo conventions; if filters are awkward, run the specific files.

9.2 Static validation

After implementation:

• make typecheck
• MUX_ESLINT_CONCURRENCY=1 make lint
• make test or targeted test suite expansion if full test is too slow/flaky for the workspace
• make fmt-check

If workflow runtime sources are generated from stdlib/docs, run the appropriate generator or static check that fails when generated output is stale.

9.3 Regression tests to preserve existing behavior

• Existing workflows without timeout behave identically.
• Existing completed step replay ignores timeout code paths.
• Existing parallel(...) and pipeline(...) behavior remains stable for steps without timeout.
• Existing task waiter timeout semantics for non-workflow callers are unchanged unless explicitly extended.

10. Dogfooding plan

Dogfooding must produce screenshots and video recordings for reviewer verification.

10.1 Local workflow dogfood setup

1. Create a temporary workflow under ./workflows/graceful-timeout-dogfood.js during implementation.
2. Use a child prompt that intentionally runs longer than the soft timeout but can produce useful partial work.
3. Example workflow shape:

export default function workflow({ agent }) {
  const report = agent(
    "Start a small investigation, deliberately take long enough to exceed the soft timeout, then comply with any timeout finalization prompt by calling agent_report.",
    {
      id: "slow-investigation",
      timeout: { softMs: 10_000, graceMs: 20_000 },
      schema: mux.schema.object(
        {
          summary: mux.schema.string(),
          completedDuringGrace: mux.schema.boolean(),
          remainingWork: mux.schema.array(mux.schema.string()),
        },
        { additionalProperties: false }
      ),
    }
  );

  return {
    reportMarkdown: report.summary,
    structuredOutput: report,
  };
}

4. Run the workflow through the normal app UI or CLI using the current dev-server sandbox conventions.

10.2 Functional dogfood scenarios

Capture evidence for each scenario:

1. Recovered during grace

   • Child exceeds soft timeout.
   • UI shows finalization/soft timeout event.
   • Child receives finalization prompt and reports.
   • Workflow completes successfully.

2. Hard timeout

   • Child ignores or cannot complete finalization.
   • UI shows hard-timeout event.
   • Workflow step fails with clear timeout error.
   • Child descendants are cleaned up or interrupted per design.

3. Resume during grace

   • Start workflow.
   • Let soft timeout fire and finalization begin.
   • Restart the app/backend or interrupt/resume the workflow.
   • Verify the finalization prompt is not duplicated and remaining grace is enforced.

4. Parallel lane timeout

   • Run a workflow with at least two parallel lanes, one slow and one fast.
   • Verify timeout handling for the slow lane does not corrupt the fast lane’s recorded result.

10.3 UI dogfood with screenshots/video

Use agent-browser for frontend verification:

• Start the dev server.
• Open the app.
• Trigger the dogfood workflow.
• Capture screenshots for:
  • task running before timeout;
  • soft timeout/finalizing state;
  • recovered state or hard-timeout failure;
  • narrow/mobile width around 375px.
• Capture a short video covering the transition from running to finalizing to recovered/hard-timeout.
• Attach screenshots/video artifacts to the PR or final implementation report using the repository’s screenshot attachment workflow.

11. Rollout and compatibility

• This is additive for workflow authors.
• Existing workflow runs without timeout metadata remain parseable because all new fields are optional.
• Existing completed steps remain cache hits because their old specs do not include timeout.
• New timeout-configured steps should hash differently from no-timeout steps.
• Avoid migrations unless implementation discovers persisted state cannot tolerate optional fields.
• If adding new event types requires UI compatibility, ensure old UI either ignores unknown event types or update shared schema/UI in the same PR.

12. Implementation order

1. Add types/parser support and tests for agent(..., { timeout }).
2. Add workflow step timeout metadata and timeout event schema/store tests.
3. Add TaskService soft-finalization/hard-timeout APIs with focused tests.
4. Wire WorkflowRunner state machine for single-agent steps.
5. Extend/revalidate parallel and pipeline paths.
6. Add UI rendering and Storybook stories.
7. Update workflow authoring docs.
8. Run targeted tests.
9. Dogfood recovered, hard-timeout, resume, and parallel scenarios with screenshots/video.
10. Run static validation and final broader tests.

13. Open implementation risks

• The exact safest TaskService transition for “stop current turn, then require agent_report” should reuse existing awaiting_report recovery behavior as much as possible; avoid creating a parallel recovery system.
• Hard termination must not delete useful artifacts before the parent can include timeout diagnostics.
• Validation retry behavior during grace could become complex; keep v1 strict and minimal.
• If a true persisted workspace finalizing status becomes necessary, the blast radius expands to active-status lists, CLI formatting, task listing, and cleanup logic. Avoid that unless required.
• Clock/deadline tests must use injected clocks/fake timers; do not use real sleeps in unit tests.

---

Generated with mux • Model: openai:gpt-5.5 • Thinking: xhigh • Cost: $247.27

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 89bcc0aa43

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the Codex findings:

• WorkflowRunner.ts: timeout deadlines are no longer persisted before a child task begins running. The runner now records executionStartedAt/softDeadlineAt from an onExecutionStarted callback when waitForAgentReport starts the execution timer, with coverage for queued tasks.
• WorkflowRunStore.ts: timeout metadata is now reset when a validation retry starts a fresh task ID, while same-task completion/failure records still preserve timeout metadata. Added coverage for validation retry after grace recovery.

[ThomasK33]

@codex review

Please take another look after the timeout deadline and retry-metadata fixes.

[ThomasK33]

@codex review

Please review the refreshed head after the resolved feedback and CI check refresh.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b70096025c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the second Codex findings:

• failAgentTaskForHardTimeout now clears the timed-out task queue before aborting the stream, matching the descendant interrupt behavior and preventing queued finalization prompts from auto-flushing after a hard timeout.
• requestAgentFinalReportForTimeout now records the finalization token only after sendMessage succeeds, so a failed send does not suppress the sole finalization prompt on resume. Added TaskService coverage for both behaviors.

[ThomasK33]

@codex review

Please take another look after the TaskService queue clearing and finalization-token fixes.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9fb71a0b2d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the third Codex findings:

• Timeout finalization sends now use startStreamInBackground with token persistence in onAccepted, so the workflow grace timer is not blocked by the finalization turn while tokens are still written only after prompt acceptance.
• Hard-timeout handling now re-checks TaskService-accepted reports before and after hard-timeout cleanup, so a just-accepted agent_report can win instead of being overwritten by a workflow hard-timeout failure.
• Timeout-configured resumed steps now restart like normal agent steps when the checkpointed task workspace is missing/interrupted, with coverage for the missing-task restart path.

[ThomasK33]

@codex review

Please take another look after the background finalization, late-report hard-timeout recheck, and missing-task restart fixes.

[ThomasK33]

@codex review

Please review the refreshed head after all third-round threads were resolved.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e93578bb3a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the fourth Codex findings:

• WorkflowRunner now only records finalizationPromptSentAt and the finalization_prompt_sent event when TaskService reports that the finalization prompt has been durably accepted; queued prompts keep the step retryable on resume.
• Timeout wait failures now record terminal task events before rethrowing non-timeout/non-restartable errors, while hard-timeout errors keep their timed_out event without adding an extra failed task row.

[ThomasK33]

@codex review

Please review the refreshed head after all fourth-round threads were resolved.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 05d4ba484a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the fifth Codex finding:

• Timeout finalization now chooses the required completion tool based on the timed-out task type. Plan-like workflow tasks receive a propose_plan-required prompt, while other tasks continue to require agent_report. Added TaskService coverage for timed-out plan agents.

[ThomasK33]

@codex review

Please review the refreshed head after the timed-out plan-agent completion-tool fix.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1790a74af

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the sixth Codex findings:

• requestAgentFinalReportForTimeout now returns immediately when the finalization token is already accepted, without soft-stopping a possibly-active finalization/reporting turn. Added coverage that an accepted retry does not call stopStream or send a replacement prompt.
• Pipeline timeout failures now skip adding an extra failed task event when the timeout path already recorded timed_out. Added pipeline coverage to assert the task row remains timed_out without a duplicate failed event.

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 65300b7f2b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the seventh Codex finding:

• Pipeline hard-timeout failures still suppress the duplicate failed task row, but now continue through the fail-fast sibling interruption path so other active pipeline agents are stopped. Added coverage that a pipeline hard timeout records timed_out, avoids failed, and calls interruptRun for siblings.

[ThomasK33]

@codex review

Please review the refreshed head after the pipeline sibling-interrupt hard-timeout fix.

[ThomasK33]

@codex review

Please review the rebased head after conflict resolution. Local validation after rebase:

• MUX_ESLINT_CONCURRENCY=1 make static-check
• bun test ./src/node/services/workflows/WorkflowRunner.test.ts ./src/node/services/taskService.test.ts (311 pass)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e74e57915b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

Addressed the eighth Codex finding:

• TaskService now waits until the timeout finalization prompt is accepted (the onAccepted token persistence has completed) before returning prompted. This prevents WorkflowRunner from starting the grace wait while the prompt is merely queued behind the soft-interrupted turn.

[ThomasK33]

@codex review

Please review the refreshed head after the deferred-grace prompt-acceptance fix.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3aae103701

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed the hard-deadline persistence finding in 5c2a431. The finalization acceptance path now records a refreshed hardDeadlineAt with finalizationPromptSentAt, and a regression test covers prompt acceptance after elapsed time.

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c2a431737

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed the queued-finalization wait finding in 18bf635. TaskService now returns queued instead of awaiting onAccepted forever when the finalization prompt is queued; WorkflowRunner can therefore continue to its bounded grace/hard-timeout path. Added coverage that the token is only persisted once the queued prompt is later accepted.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 18bf635347

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".