[core] Exclude inline step execution from replay timeout#2013
Open
TooTallNate wants to merge 5 commits into
Open
[core] Exclude inline step execution from replay timeout#2013TooTallNate wants to merge 5 commits into
TooTallNate wants to merge 5 commits into
Conversation
The v5 combined workflow+step handler wraps inline step bodies in the same setTimeout(..., REPLAY_TIMEOUT_MS) guard that previously only bounded the v4 'workflows' function's fast deterministic replay. As a result, any workflow with a single step exceeding 240s hard-fails with FatalError: Workflow replay exceeded maximum duration (240s) after 4 attempts — even though the step could legitimately run for the full function maxDuration (up to 800s on Pro Fluid). Replace the setTimeout guard with a per-invocation budget that only accumulates non-step time. pauseReplayBudget() / resumeReplayBudget() bracket each executeStep() call, and the loop checks the budget at iteration boundaries. The retry-then-fail semantics from #1567 are preserved verbatim for the pure-replay case. Also adds a WORKFLOW_REPLAY_TIMEOUT_MS env var override (clamped to 30s..780s) so operators can adjust the bare-replay ceiling without patching @workflow/core. Fixes #2009.
🦋 Changeset detectedLatest commit: ae34ff9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests▲ Vercel Production (4 failed)fastify (1 failed):
nextjs-webpack (1 failed):
nitro (1 failed):
sveltekit (1 failed):
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
❌ Some E2E test jobs failed:
Check the workflow run for details. |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro workflow with 10 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 25 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 50 sequential data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 10 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) workflow with 25 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro workflow with 50 concurrent data payload steps (10KB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro stream pipeline with 5 transform steps (1MB)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro 10 parallel streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro fan-out fan-in 10 streams (1MB each)💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
❌ Some benchmark jobs failed:
Check the workflow run for details. |
Contributor
There was a problem hiding this comment.
Pull request overview
Fixes a v5 regression where inline step bodies were incorrectly counted toward the 240s replay timeout, causing legitimately long-running steps to fail with REPLAY_TIMEOUT. The PR switches from a whole-handler timeout to a replay “budget” that only measures non-step time between step boundaries, and adds an env var override for the budget.
Changes:
- Replace the global
setTimeout(..., REPLAY_TIMEOUT_MS)guard with replay-budget tracking that pauses duringexecuteStep(...). - Add
getReplayTimeoutMs()withWORKFLOW_REPLAY_TIMEOUT_MSoverride (clamped to 30s–780s) plus unit tests. - Update
REPLAY_TIMEOUThint messaging and corresponding tests; add a changeset.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/core/src/runtime/constants.ts | Adds replay-timeout env override + bounds constants; keeps default at 240s. |
| packages/core/src/runtime/constants.test.ts | Unit tests for parsing/clamping/fallback behavior of getReplayTimeoutMs(). |
| packages/core/src/runtime.ts | Implements replay-budget bookkeeping and excludes inline step execution time from the budget. |
| packages/core/src/describe-error.ts | Updates the REPLAY_TIMEOUT hint to reflect new semantics and env override. |
| packages/core/src/describe-error.test.ts | Updates assertions/snapshots for the revised replay-timeout hint. |
| .changeset/replay-timeout-excludes-step-bodies.md | Patch changeset describing the regression fix and new env var. |
Comments suppressed due to low confidence (2)
packages/core/src/runtime.ts:274
- The warning message says "attempt < maxRetries" but the condition is
metadata.attempt <= REPLAY_TIMEOUT_MAX_RETRIES. Either adjust the message or the condition so logs accurately reflect the retry semantics.
const handleReplayBudgetExhausted = async (): Promise<void> => {
if (metadata.attempt <= REPLAY_TIMEOUT_MAX_RETRIES) {
runLogger.warn(
'Workflow replay exceeded timeout but will be re-attempted (attempt < maxRetries)',
{
timeoutMs: replayTimeoutMs,
attempt: metadata.attempt,
maxRetries: REPLAY_TIMEOUT_MAX_RETRIES,
}
packages/core/src/runtime.ts:246
- The comment above
pauseReplayBudgetsays it "reset[s] the start marker", but the function currently only accumulates elapsed time and does not updatenonStepStart. Either update the comment or setnonStepStartthere to keep the bookkeeping self-consistent and avoid accidentally counting step time if a future call toisReplayBudgetExhausted()occurs beforeresumeReplayBudget().
// Accumulate the elapsed delta since the last checkpoint (handler
// entry or the most recent `resumeReplayBudget`) and reset the
// start marker. Call this immediately before any inline step body.
const pauseReplayBudget = (): void => {
replayElapsedMs += Date.now() - nonStepStart;
};
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
pranaygp
reviewed
May 18, 2026
pranaygp
reviewed
May 18, 2026
pranaygp
reviewed
May 18, 2026
Contributor
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Thanks for the thorough fix and the detailed PR description — the pause/resume bracketing pattern is clean and the rationale (v4 parity + Fluid headroom) is well laid out. A few inline comments below; the main one I'd want resolved before merge is the loss of the VERCEL_URL gating around process.exit(1).
- Extract budget bookkeeping into ReplayBudget class (replay-budget.ts) with sentinel-protected idempotent pause()/resume() to avoid double-counting in future refactors that nest step execution - Restore VERCEL_URL gate around process.exit(1) so a long pure-replay in local dev/non-Vercel runtimes can't hard-kill the host process - Warn (once per distinct raw value) when WORKFLOW_REPLAY_TIMEOUT_MS is clamped or rejected, so misconfiguration is observable - Correct Hobby maxDuration comment (60s standard / 300s Fluid) - Document budget-check responsiveness trade-off vs. old setTimeout - Tighten describe-error test assertions to match the full new hint - Shorten changeset description - Add ReplayBudget unit tests (9) including 8-minute step regression - Add warn-once tests for getReplayTimeoutMs (extended)
VaguelySerious
approved these changes
May 19, 2026
Member
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM, something I missed in testing that would be useful to do on this branch: run a workflow with multiple sequential ~4 minute steps, and multiple parallel 4 minute steps, so we can check whether the replay timeout and the max-inline times interact correctly
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2009.
The v5 combined workflow+step handler (
workflowEntrypoint) was wrapping inline step bodies in the samesetTimeout(..., REPLAY_TIMEOUT_MS)(240s) guard that, in v4, only bounded the dedicatedworkflowsfunction's fast deterministic replay. Thestepsfunction previously had its ownmaxDuration: 'max'ceiling (800s on Pro Fluid) and was not subject to the replay timeout.Net effect on v5: any workflow with a single step exceeding 240s hard-fails after 4 attempts with:
…even though the step could legitimately run for the full function
maxDuration. This is a regression in apparent behavior from v4.Changes
packages/core/src/runtime.ts— replace thesetTimeoutguard with a per-invocation budget that only accumulates non-step time. Two small closures (pauseReplayBudget/resumeReplayBudget) bracket everyawait executeStep(...)call so step duration is excluded. The loop checksisReplayBudgetExhausted()at iteration boundaries; if exhausted,handleReplayBudgetExhausted()runs the same retry-then-fail flow as before (preserving [core] Extend flow route duration to "max" and fail runs where replay takes too long #1567's defense for pathological pure replays).packages/core/src/runtime/constants.ts— keepREPLAY_TIMEOUT_MS = 240_000as the default. AddgetReplayTimeoutMs()which readsWORKFLOW_REPLAY_TIMEOUT_MS(clamped to[30s, 780s]). 780s leaves ≥20s of headroom under Pro Fluid's 800s function ceiling so the handler can still writerun_failedbefore SIGTERM.packages/core/src/describe-error.ts— tightenREPLAY_TIMEOUT_HINTto reflect new semantics ("between step boundaries", "step bodies excluded") and mention the env var override.packages/core/src/runtime/constants.test.tscovers env var parsing, clamping, and fallback behavior. Existingdescribe-error.test.tsassertions updated for the new hint wording.Interaction with existing limits
NO_INLINE_REPLAY_AFTER_MS(120s default,WORKFLOW_V2_TIMEOUT_MSenv override) is unchanged and continues to re-queue long invocations so they don't exhaust the function's actualmaxDuration. For an 8-minute step: the step runs inline to completion (now possible because the replay budget is paused), then the next loop iteration immediately re-queues since 8min > 120s elapsed. Same correctness as today, just no longer false-failing.Backport
This bug is v5-only — in v4 step bodies ran in the separate
stepsfunction which had no replay-timeout wrapper. No backport tostableneeded.Test plan
pnpm testpasses (956 tests, including 12 new tests inconstants.test.ts).pnpm typecheckpasses.await new Promise(resolve => setTimeout(resolve, 8 * 60 * 1000))inside a"use step"function) — to be validated by reviewer against a deployed Pro Fluid project.