Change user input/output to be binary data at the World interface#853
Merged
Conversation
🦋 Changeset detectedLatest commit: ce52cc0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (161 failed)mongodb (40 failed):
redis (40 failed):
starter (41 failed):
turso (40 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
Contributor
Member
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
TooTallNate
requested review from
Schniz,
VaguelySerious,
Copilot,
ijjk,
karthikscale3 and
pranaygp
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the World interface to treat workflow/step input and output as opaque binary payloads (Uint8Array) instead of JSON-compatible “revived” devalue arrays, enabling transport-level enhancements (e.g., compression/encryption) without coupling World implementations to the underlying serialization.
Changes:
- Switch core serialization to emit/consume
Uint8Array(TextEncoder/TextDecoder overdevalue.stringify/parse). - Update
@workflow/worldschemas/types and World implementations (local/postgres/vercel) to store/transport binary data and supportresolveData='none'via*WithoutDatatypes. - Update workbench triggers and tests to send/receive
application/octet-streamrequest bodies and handle binary values in JSON where needed.
Reviewed changes
Copilot reviewed 54 out of 54 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| workbench/vite/routes/api/trigger.post.ts | Reads request body as binary and hydrates args from Uint8Array. |
| workbench/sveltekit/src/routes/api/trigger/+server.ts | Reads request body as binary and hydrates args from Uint8Array. |
| workbench/nuxt/server/api/trigger.post.ts | Uses web Request + arrayBuffer() to support binary args in Nuxt/h3. |
| workbench/nitro-v3/routes/api/trigger.post.ts | Reads request body as binary and hydrates args from Uint8Array. |
| workbench/nitro-v2/server/api/trigger.post.ts | Uses web Request + arrayBuffer() to support binary args in Nitro v2. |
| workbench/nextjs-webpack/pages/api/trigger-pages.ts | Disables body parser and reads raw binary body for args (pages router). |
| workbench/nextjs-webpack/app/api/trigger/route.ts | Reads request body as binary for args (app router). |
| workbench/nextjs-turbopack/pages/api/trigger-pages.ts | Disables body parser and reads raw binary body for args (pages router). |
| workbench/nextjs-turbopack/app/api/trigger/route.ts | Reads request body as binary for args (app router). |
| workbench/hono/src/index.ts | Reads request body as binary for args in Hono handler. |
| workbench/fastify/src/index.ts | Adds octet-stream parser and hydrates args from binary body. |
| workbench/express/src/index.ts | Adds raw octet-stream middleware and hydrates args from binary body. |
| workbench/example/api/trigger.ts | Reads request body as binary for args in example route. |
| workbench/astro/src/pages/api/trigger.ts | Reads request body as binary for args in Astro route. |
| packages/world/src/steps.ts | Updates Step schema to use SerializedData and adds StepWithoutData. |
| packages/world/src/serialization.ts | Defines SerializedData = Uint8Array and a Zod schema for it. |
| packages/world/src/runs.ts | Updates run schemas to use SerializedData and adds WorkflowRunWithoutData. |
| packages/world/src/interfaces.ts | Adds overloads for resolveData='none' returning *WithoutData types. |
| packages/world/src/index.ts | Re-exports SerializedData and SerializedDataSchema. |
| packages/world/src/hooks.ts | Changes hook metadata to use SerializedData. |
| packages/world/src/events.ts | Updates event payload fields (input/output/result/payload) to SerializedData. |
| packages/world-vercel/src/utils.ts | Updates hard-coded workflow-server URL override to a new test deployment. |
| packages/world-vercel/src/storage.ts | Casts storage methods to new overloaded Storage signatures. |
| packages/world-vercel/src/steps.ts | Updates step wire schema to accept Uint8Array and adjusts resolveData filtering. |
| packages/world-vercel/src/runs.ts | Updates run wire schema/types and adds overloads for resolveData-none responses. |
| packages/world-testing/src/util.mts | Adds JSON reviver to decode base64-wrapped Uint8Array from server responses. |
| packages/world-testing/src/server.mts | Adds JSON serialization for Uint8Array as base64-wrapped objects. |
| packages/world-testing/src/null-byte.mts | Updates tests to hydrate workflow output from Uint8Array. |
| packages/world-testing/src/idempotency.mts | Updates tests to hydrate workflow output from Uint8Array. |
| packages/world-testing/src/hooks.mts | Updates tests to hydrate workflow output from Uint8Array. |
| packages/world-testing/src/errors.mts | Updates tests to hydrate workflow output from Uint8Array. |
| packages/world-testing/src/addition.mts | Updates tests to hydrate workflow output from Uint8Array. |
| packages/world-postgres/test/storage.test.ts | Updates postgres tests and legacy inserts for CBOR/binary input columns. |
| packages/world-postgres/src/storage.ts | Updates storage typing and filtering for resolveData='none' and binary IO. |
| packages/world-local/src/test-helpers.ts | Updates helper types to use SerializedData for input/metadata. |
| packages/world-local/src/storage/steps-storage.ts | Updates resolveData-none behavior and typing for step listing/getting. |
| packages/world-local/src/storage/runs-storage.ts | Updates resolveData-none behavior and typing for run listing/getting. |
| packages/world-local/src/storage/legacy.ts | Adjusts legacy event handling with updated run filtering types. |
| packages/world-local/src/storage/filters.ts | Adds overloads returning *WithoutData when resolveData='none'. |
| packages/world-local/src/storage/events-storage.ts | Updates run creation to store binary input values. |
| packages/world-local/src/storage.test.ts | Updates local storage tests and uses writeJSON helper. |
| packages/world-local/src/fs.ts | Adds JSON replacer/reviver to persist Uint8Array as base64-wrapped objects. |
| packages/web-shared/src/api/workflow-server-actions.ts | Adjusts UI server actions to handle resolveData='none' data shapes. |
| packages/core/src/workflow.ts | Changes workflow execution to return binary (Uint8Array) results. |
| packages/core/src/step.test.ts | Updates step tests to use binary dehydrated step results. |
| packages/core/src/serialization.ts | Switches serialization to binary (Uint8Array) via TextEncoder/TextDecoder. |
| packages/core/src/serialization.test.ts | Updates serialization tests for binary output representation. |
| packages/core/src/runtime/suspension-handler.ts | Updates hook/step event payload typing to SerializedData. |
| packages/core/src/runtime/step-handler.ts | Updates step_completed event payload typing to Uint8Array. |
| packages/core/src/runtime.ts | Updates run_completed event payload to store binary output. |
| packages/core/src/observability.ts | Updates observability hydration to detect Uint8Array input/output. |
| packages/core/e2e/e2e.test.ts | Sends trigger args as application/octet-stream with binary body. |
| packages/cli/src/lib/inspect/output.ts | Updates CLI pagination typing for *WithoutData result shapes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
88
to
92
| deploymentId: string; | ||
| workflowName: string; | ||
| input: SerializedData[]; | ||
| input: SerializedData; | ||
| executionContext?: SerializedData; | ||
| } |
There was a problem hiding this comment.
CreateWorkflowRunRequest.executionContext is typed as SerializedData (now Uint8Array), but the run schema defines executionContext as z.record(z.string(), z.any()).optional(). This makes the request type inconsistent with what WorkflowRunBaseSchema accepts and with UpdateWorkflowRunRequest.executionContext (record). Consider changing executionContext?: Record<string, any> (or a dedicated type) here to match the schema (and keep only input/output as SerializedData).
Comment on lines
108
to
112
| return { | ||
| ...deserialized, | ||
| input: [], | ||
| input: new Uint8Array(), | ||
| output: undefined, | ||
| }; |
There was a problem hiding this comment.
When resolveData === 'none', this returns a Step with input: new Uint8Array().
But the updated World/Storage typings introduce StepWithoutData where input/output are undefined for resolveData='none'. Returning an empty Uint8Array makes the behavior inconsistent across World implementations and can confuse callers that use undefined as the signal that data was excluded. Consider returning StepWithoutData here (and updating the list/get return types accordingly) with input: undefined and output: undefined.
Comment on lines
640
to
643
| return createResponse({ | ||
| data: (result.data as Step[]).map(hydrate), | ||
| // StepWithoutData has undefined input/output, but after hydration the structure is compatible | ||
| data: (result.data as unknown as Step[]).map(hydrate), | ||
| cursor: result.cursor ?? undefined, |
There was a problem hiding this comment.
This code path calls hydrate(...), but hydrate() currently JSON-stringifies/parses the resource before calling hydrateResourceIO. With the new binary I/O, that round-trip converts Uint8Array into a plain object, so hydrateResourceIO will no longer detect input/output as Uint8Array and hydration will silently stop working.
Consider updating hydrate() to preserve Uint8Array (e.g., hydrate first, then make JSON-compatible; or use a custom replacer/reviver that encodes Uint8Array as base64).
Comment on lines
+840
to
+845
| const deploymentId = run.deploymentId; | ||
| // hydrateResourceIO deserializes the binary input back to the original array | ||
| const newRun = await start( | ||
| { workflowId: run.workflowName }, | ||
| hydratedRun.input, | ||
| hydratedRun.input as unknown as unknown[], |
There was a problem hiding this comment.
hydratedRun comes from hydrate(...), but hydrate() currently JSON-stringifies/parses the run before calling hydrateResourceIO, which strips Uint8Array instances. That means hydratedRun.input may not actually be hydrated back into an arguments array, and the double-cast to unknown[] can mask this at compile time.
After fixing hydrate() to preserve binary data, consider removing the unsafe cast here and ensuring hydratedRun.input is validated/typed as an array before passing it to start(...).
| ...getProtectionBypassHeaders(), | ||
| 'Content-Type': 'application/octet-stream', | ||
| }, | ||
| body: dehydratedArgs.buffer as BodyInit, |
There was a problem hiding this comment.
body: dehydratedArgs.buffer ignores byteOffset/byteLength and will send the entire underlying ArrayBuffer if dehydratedArgs is ever a view into a larger buffer. Using the Uint8Array directly (or slicing the buffer to the view range) avoids accidentally sending extra bytes.
Suggested change
| body: dehydratedArgs.buffer as BodyInit, | |
| body: dehydratedArgs as BodyInit, |
Comment on lines
+1
to
+6
| import { | ||
| defineEventHandler, | ||
| getRequestURL, | ||
| readRawBody, | ||
| toWebRequest, | ||
| } from 'h3'; |
There was a problem hiding this comment.
Unused import readRawBody.
Comment on lines
+1
to
+6
| import { | ||
| defineEventHandler, | ||
| getRequestURL, | ||
| readRawBody, | ||
| toWebRequest, | ||
| } from 'h3'; |
There was a problem hiding this comment.
Unused import readRawBody.
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
pranaygp
reviewed
Jan 28, 2026
Contributor
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Code Review Summary
This is a well-executed architectural change that properly abstracts serialization concerns from the World interface. The implementation is thorough and consistent across all packages.
Key Strengths
- Consistent implementation across all World backends (local, postgres, vercel)
- Good use of function overloads for type safety with
resolveDataoptions - Thorough test coverage updates
- Clean separation allowing future encryption/compression without World changes
Issues Requiring Attention
1. Critical: Backwards Compatibility for Observability (see inline comments)
The observability code in packages/core/src/observability.ts now only checks for Uint8Array, which will break rendering of specVersion 1 (legacy) runs. Legacy runs store data as JSON arrays, not Uint8Array.
Recommendation: Use specVersion to branch between legacy (unflatten on arrays) and new (TextDecoder + parse on Uint8Array) formats. This can be shipped in a follow-up PR but must be done before releasing a new version to avoid shipping broken observability.
2. Changeset should be marked as breaking change
Per project conventions, breaking changes should include **BREAKING CHANGE** in the changeset description.
3. Minor: Inconsistency in world-vercel/steps.ts
When resolveData='none', this file returns input: new Uint8Array() while other implementations return input: undefined. This should be consistent.
Verdict
Approve with the understanding that the observability backwards compatibility fix will be addressed before release.
pranaygp
reviewed
Jan 28, 2026
Contributor
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Code Review: Binary Serialization at World Interface
Overall Assessment
This is a well-executed architectural change that properly abstracts serialization concerns from the World interface. The implementation is thorough and consistent across all packages.
Key Strengths
- Consistent implementation across all World backends (local, postgres, vercel)
- Good use of function overloads for type safety with
resolveDataoptions - Thorough test coverage updates
- Clean separation allowing future encryption/compression without World changes
🚨 Critical: Backwards Compatibility Issue in Observability
The observability code in packages/core/src/observability.ts now only checks for Uint8Array, which will break rendering of specVersion 1 (legacy) runs.
The Problem
Legacy runs (specVersion 1) store data as JSON arrays, not Uint8Array. The new code:
// packages/core/src/observability.ts - hydrateStepIO
input:
step.input instanceof Uint8Array && step.input.byteLength > 0
? hydrateStepArguments(step.input, ...)
: step.input, // ← Legacy data falls through UNHYDRATED!| specVersion | Storage Format | Observability Check | Result |
|---|---|---|---|
| 1 (legacy) | JSON arrays | instanceof Uint8Array → false |
❌ Raw arrays shown |
| 2 (new) | Binary | instanceof Uint8Array → true |
✅ Properly hydrated |
Recommended Fix (use specVersion to branch)
const hydrateStepIO = <T extends { stepId?: string; input?: any; output?: any; runId?: string; specVersion?: number }>(
step: T
): T => {
const isLegacy = (step.specVersion ?? 1) < 2;
if (isLegacy) {
// Legacy path: use unflatten directly on JSON arrays
return {
...step,
input: step.input && Array.isArray(step.input) && step.input.length
? unflatten(step.input, { ...getStepRevivers(...), ...streamPrintRevivers })
: step.input,
output: step.output
? unflatten(step.output, { ...getWorkflowRevivers(...), ...streamPrintRevivers })
: step.output,
};
}
// New path: decode Uint8Array (current implementation)
return { ... };
};Functions that need this fix in observability.ts:
hydrateStepIO- step input/outputhydrateWorkflowIO- workflow input/outputhydrateEventData-eventData.resulthydrateHookMetadata- hook metadata
This can be shipped in a follow-up PR, but must be done before releasing a new version of @workflow/web-shared to avoid shipping broken observability for historical runs.
Other Issues
Changeset should be marked as breaking
Per project conventions, the changeset should include **BREAKING CHANGE** in the description.
Minor: Inconsistency in world-vercel/steps.ts
When resolveData='none', packages/world-vercel/src/steps.ts:110 returns input: new Uint8Array() while other implementations return input: undefined. Should be consistent.
Verdict
👍 Approve with the understanding that the observability backwards compatibility fix will be addressed before release.
pranaygp
approved these changes
Jan 28, 2026
Contributor
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Approving with the understanding that the observability backwards compatibility fix (for specVersion 1 legacy runs) will be addressed before releasing a new version of @workflow/web-shared.
VaguelySerious
approved these changes
Jan 28, 2026
Member
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
LGTM with the latest fixes - let's 🚢 and then do a bug bash
Merged
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a breaking change to the
Worldinterface, where we are changing serialization of workflow and step data to use binary format (Uint8Array) instead of JSON arrays ("revived" devalue serialization). This will allow the workflow client to be fully responsible for the data serialization format and introduce further enhancements such as encryption, compression, etc. without theWorldimplementation needing to care what the underlying data represents.