Add format prefix to stream chunks and use .bin extension for chunk files (in world-local)#1020
Merged
Merged
Conversation
…k files
Stream chunks were serialized as raw devalue text without the 'devl'
format prefix, unlike step inputs/outputs. This meant stream data
couldn't go through the same hydration/deserialization pipeline.
Core changes (serialization.ts):
- getSerializeStream: each chunk is now individually wrapped with
encodeWithFormatPrefix('devl', payload) — same as step data
- getDeserializeStream: handles both format-prefixed chunks (current)
and legacy newline-delimited text (backwards compatible)
World-local changes (streamer.ts):
- Stream chunk files now use .bin extension instead of .json, since
the content is binary (1-byte eof flag + serialized chunk data)
- Add listFilesByExtension() utility to fs.ts for generic file listing
Contributor
🦋 Changeset detectedLatest commit: aaaf076 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Nitro | Express workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Next.js (Turbopack) | Express Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (41 failed)turso (41 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
When reading stream chunks, list both .bin (current) and .json (legacy) files so streams written before the extension change continue to work.
Stream chunks were serialized as raw devalue text without the 'devl'
format prefix or any framing, unlike step inputs/outputs.
Core changes (serialization.ts):
- getSerializeStream: each chunk is now wrapped with
encodeWithFormatPrefix('devl', payload) inside a length-prefixed
frame: [4-byte big-endian length][format-prefixed payload]
- getDeserializeStream: handles both length-prefixed framed chunks
(current) and legacy newline-delimited text (backwards compatible)
- Added 6 unit tests covering framing, round-trip, concatenation,
splitting, and legacy fallback
World-local changes (streamer.ts):
- Stream chunk files now use .bin extension instead of .json
- readFromStream lists both .bin and .json files for backwards compat
- Updated streamer.test.ts to expect .bin extension
Split changesets: core (format + framing) and world-local (.bin extension)
TooTallNate
force-pushed
the
nate/stream-chunk-format-prefix
branch
from
3ee8b66 to
aaaf076
Compare
TooTallNate
changed the title
.bin extension for chunk files (in world-local)
VaguelySerious
approved these changes
Feb 12, 2026
This was referenced Feb 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stream chunks were serialized as raw devalue text without the
devlformat prefix or any framing, unlike step inputs/outputs. This meant stream data couldn't go through the same hydration/deserialization pipeline used for other serialized data, and future encryption support (encrprefix) couldn't be applied to stream chunks.Changes
@workflow/core(serialization.ts)getSerializeStream: Each chunk is now individually wrapped withencodeWithFormatPrefix('devl', payload)inside a length-prefixed frame:getDeserializeStream: Buffers incoming data and extracts complete frames. Auto-detects framed vs legacy format:@workflow/world-local(streamer.ts,fs.ts).binextension instead of.json(the content is binary, not JSON)readFromStreamlists both.binand.jsonfiles for backwards compatibility with existing stored streamslistFilesByExtension()utility tofs.tsstreamer.test.tsto expect.binextensionBackwards Compatibility
getDeserializeStreamfalls back to legacy newline-delimited text parsing when the data is not length-prefixedreadFromStreamreads both.bin(current) and.json(legacy) chunk files