Retry 5xx errors from workflow-server in step handler#1011
Merged
Conversation
🦋 Changeset detectedLatest commit: 87a0c91 The changes in this PR will be included in the next version bump. This PR includes changesets to release 14 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (43 failed)mongodb (1 failed):
turso (42 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends workflow-server transient error handling into the step handler by introducing a 5xx retry helper and applying it to step lifecycle event writes, aiming to avoid consuming step attempts on transient infrastructure failures.
Changes:
- Add
withServerErrorRetryhelper to retry workflow-server 5xx errors with exponential backoff. - Wrap step handler
world.events.createcalls forstep_started,step_completed,step_failed, andstep_retryingwith the retry helper. - Add a 5xx “bubble to queue retry” path in the step execution error handling block.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
packages/core/src/runtime/step-handler.ts |
Wrap step lifecycle event creation with 5xx retry; add logic to rethrow persistent 5xx to defer to queue retry. |
packages/core/src/runtime/helpers.ts |
Introduce withServerErrorRetry with 3 retries and exponential backoff for 5xx WorkflowAPIErrors. |
.changeset/retry-5xx-step-handler.md |
Patch changeset describing the new retry behavior in the step handler. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+120
to
+126
| const startResult = await withServerErrorRetry(() => | ||
| world.events.create(workflowRunId, { | ||
| eventType: 'step_started', | ||
| specVersion: SPEC_VERSION_CURRENT, | ||
| correlationId: stepId, | ||
| }) | ||
| ); |
There was a problem hiding this comment.
This adds new retry behavior that changes how the step handler reacts to workflow-server 5xx responses, but there are no unit tests covering the new helper’s retry/backoff semantics or the step-handler’s 5xx fast-path (throwing to queue vs. emitting step_retrying). Adding vitest coverage (similar to runtime/start.test.ts) would help prevent regressions in retry counts/delays and in when attempts are consumed.
Contributor
Author
There was a problem hiding this comment.
Added unit tests in helpers.test.ts covering: success passthrough, retry on 5xx with recovery, exponential backoff across 3 retries, retry exhaustion, and non-retryable error passthrough (non-5xx, non-WorkflowAPIError, 429).
pranaygp
force-pushed
the
pranaygp/retry-5xx
branch
from
fffe061 to
e3ef703
Compare
Contributor
VaguelySerious
approved these changes
Feb 12, 2026
Member
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM with some concerns about what "transient" means for 500s and forever-retrying workflows
| export async function withServerErrorRetry<T>( | ||
| fn: () => Promise<T> | ||
| ): Promise<T> { | ||
| const delays = [500, 1000, 2000]; |
Member
There was a problem hiding this comment.
I'd like more backoff here. 500s might be transient, but in our history, most 500s were fixed after 5-60 minutes, not a few seconds. The only transient 500s I remember is dynamodb throttling, which we should be returning 429s for, but I guess this is safe since we're only doing three re-tries.
ctgowrie
reviewed
Feb 12, 2026
| }; | ||
| } | ||
| // Wrap VQS server errors as WorkflowAPIError so withServerErrorRetry can catch them | ||
| if (error instanceof InternalServerError) { |
Contributor
There was a problem hiding this comment.
Would prefer we let queue client handle so we can control with server
Contributor
Author
|
moving to draft till I remove the queue retrying and rebase this on main for merge conflicts |
Add `withServerErrorRetry` helper that retries world calls on 5xx errors with exponential backoff (500ms, 1s, 2s ≈ 3.5s total). Applied to all `world.events.create` calls in the step handler so transient workflow-server errors don't consume step attempts. If retries are exhausted, the error is thrown to the queue for higher-level retry without burning a step attempt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix misleading `maxAttempts` log field to `maxRetries` in withServerErrorRetry - Update step-handler comment to accurately note that queue retries may still consume step attempts since step_started has already incremented - Add unit tests for withServerErrorRetry (7 tests covering success, retry/backoff, exhaustion, and non-5xx passthrough) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Unit tests for withThrottleRetry and withServerErrorRetry helpers, plus an e2e test that exercises the 5xx retry codepath during step execution via run-scoped fault injection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
VQS throws its own error types (InternalServerError, ConsumerDiscoveryError, ConsumerRegistryNotConfiguredError) that don't match WorkflowAPIError.is(). Wrapping them at the world-vercel boundary enables withServerErrorRetry in queueMessage() to automatically retry transient queue failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Queue retrying will be handled natively by the @vercel/queue client instead. Reverts VQS error wrapping and withServerErrorRetry in queueMessage(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
pranaygp
changed the title
pranaygp
force-pushed
the
pranaygp/retry-5xx
branch
from
4dd3ee5 to
87a0c91
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human (hey it's me Pranay)
In addition to retrying 5xx errors, I added an e2e test here for steps. For AI and human reviewers: pleae pay careful attention to the e2e test to validate that it actually works and tests the right thing and isn't a possible false positive.
read on:
AI
Summary
withServerErrorRetryhelper that retries world calls on 5xx errors with exponential backoff (500ms, 1s, 2s ≈ 3.5s total)world.events.createcalls in the step handler (step_started,step_completed,step_failed,step_retrying) with the retry helperstep_retrying, so no step attempt is consumedExtends the 5xx retry behavior from #966 to the step handler. The key insight: transient workflow-server errors should be retried silently without consuming step attempts, while persistent errors bubble to the queue for higher-level retry.
Tests added
helpers.test.ts): 26 tests coveringgetWorkflowQueueName(10 tests),withServerErrorRetry(7 tests) andwithThrottleRetry(9 tests) — success passthrough, retry on 5xx with exponential backoff, retry exhaustion, non-retryable error passthrough, short/long retryAfter handling, double-throttle deferral, and default retryAfter behaviore2e.test.ts):serverError5xxRetryWorkflowuses run-scoped fault injection to makestep_completedcalls throw 500 errors, then verifies the workflow completes correctly, retries actually fired, and no step attempt was consumedTest plan
pnpm buildpassespnpm testinpackages/corepassespnpm vitest run packages/core/src/runtime/helpers.test.ts— 26 tests passserverError5xxRetryWorkflowpasses locally against nextjs-turbopack dev server🤖 Generated with Claude Code