Add end-to-end encryption for workflow user data#950
Closed
TooTallNate wants to merge 7 commits into
Closed
Conversation
🦋 Changeset detectedLatest commit: f95f584 The changes in this PR will be included in the next version bump. This PR includes changesets to release 19 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 1 step💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 25 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) workflow with 50 sequential steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.all with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.all with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Promise.race with 10 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Nitro | Express | Next.js (Turbopack) Promise.race with 25 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Next.js (Turbopack) | Nitro Promise.race with 50 concurrent steps💻 Local Development
▲ Production (Vercel)
🔍 Observability: Express | Nitro | Next.js (Turbopack) Stream Benchmarks (includes TTFB metrics)workflow with stream💻 Local Development
▲ Production (Vercel)
🔍 Observability: Next.js (Turbopack) | Express | Nitro SummaryFastest Framework by WorldWinner determined by most benchmark wins
Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
Contributor
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (169 failed)mongodb (42 failed):
redis (42 failed):
starter (43 failed):
turso (42 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
Contributor
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements end-to-end encryption for workflow user data using AES-256-GCM with per-run key derivation via HKDF-SHA256. The implementation requires client-side runId generation to enable encryption before data serialization.
Changes:
- Added encryption module with AES-256-GCM + HKDF-SHA256 key derivation
- Converted all (de)hydration serialization functions to async with encryption support
- Implemented client-side runId generation for encryption context
- Updated workflow execution, steps, hooks, and observability for async serialization
- Added comprehensive encryption test coverage (18 unit tests)
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/interfaces.ts | Added Encryptor, EncryptionContext, and KeyMaterial interfaces; extended World interface |
| packages/world/src/events.ts | Added optional client-provided runId field to run_created event |
| packages/world-vercel/src/encryption.ts | New encryption implementation with HKDF-based per-run key derivation |
| packages/world-vercel/src/encryption.test.ts | Comprehensive test suite for encryption functionality |
| packages/world-vercel/src/index.ts | Integrated encryptor into Vercel World implementation |
| packages/core/src/serialization.ts | Made all serialization functions async; added encryption/decryption helpers |
| packages/core/src/serialization.test.ts | Updated test wrappers for async serialization |
| packages/core/src/workflow.ts | Added world parameter to runWorkflow; updated hydration calls |
| packages/core/src/workflow.test.ts | Updated test helpers with mock world and async wrappers |
| packages/core/src/step.ts | Updated step hydration for async operation |
| packages/core/src/step.test.ts | Updated test helpers with mock world |
| packages/core/src/workflow/hook.ts | Updated hook payload hydration for async operation |
| packages/core/src/workflow/hook.test.ts | Updated test helpers with mock world |
| packages/core/src/runtime/start.ts | Implemented client-side runId generation for encryption |
| packages/core/src/runtime/start.test.ts | Updated mocks to return client-provided runId |
| packages/core/src/runtime/run.ts | Updated result hydration for async operation |
| packages/core/src/runtime/resume-hook.ts | Updated hook metadata hydration for async operation |
| packages/core/src/runtime/step-handler.ts | Updated step I/O serialization for async operation |
| packages/core/src/runtime/suspension-handler.ts | Updated event creation with async serialization |
| packages/core/src/runtime.ts | Pass world instance to runWorkflow |
| packages/core/src/private.ts | Added runId and world to WorkflowOrchestratorContext |
| packages/core/src/observability.ts | Made hydrateResourceIO async with world parameter |
| packages/core/src/observability.test.ts | Updated test helpers with mock world |
| packages/core/src/writable-stream.test.ts | Removed Promise support tests |
| packages/web-shared/src/api/workflow-server-actions.ts | Updated all hydration calls to async with world parameter |
| packages/cli/src/lib/inspect/output.ts | Updated all hydration calls to async with world parameter |
Comments suppressed due to low confidence (2)
packages/world/src/interfaces.ts:109
- The Streamer interface still allows
runId: string | Promise<string>for writeToStream, writeToStreamMulti, and closeStream, but WorkflowServerWritableStream now only acceptsstring(line 393). This could cause confusion for World implementations.
Since runId is now always generated client-side before serialization (as required for encryption), the Streamer interface should be updated to only accept string for consistency. World implementations (world-local, world-postgres, world-vercel) may need to be updated to match this stricter type.
writeToStream(
name: string,
runId: string | Promise<string>,
chunk: string | Uint8Array
): Promise<void>;
/**
* Write multiple chunks to a stream in a single operation.
* This is an optional optimization for world implementations that can
* batch multiple writes efficiently (e.g., single HTTP request for world-vercel).
*
* If not implemented, the caller should fall back to sequential writeToStream() calls.
*
* @param name - The stream name
* @param runId - The run ID (can be a promise)
* @param chunks - Array of chunks to write, in order
*/
writeToStreamMulti?(
name: string,
runId: string | Promise<string>,
chunks: (string | Uint8Array)[]
): Promise<void>;
closeStream(name: string, runId: string | Promise<string>): Promise<void>;
packages/core/src/runtime/start.ts:143
- The client-generated runId is passed in the event data (line 134) but the server implementations (world-local and world-postgres) check the first parameter of events.create() to decide whether to use a client-provided runId or generate one. Since the client passes
nullas the first parameter (line 130), the server will ignore the client-provided runId in the event data and generate its own runId.
This means encryption will fail because the client encrypted data with one runId but the server will use a different runId when attempting to decrypt.
The fix should be to pass the client-generated runId as the first argument to events.create() instead of null, or update all server implementations to check data.runId from the event data for run_created events.
const result = await world.events.create(
null,
{
eventType: 'run_created',
specVersion,
runId, // Pass client-generated runId to server
eventData: {
deploymentId: deploymentId,
workflowName: workflowName,
input: workflowArguments,
executionContext: { traceCarrier, workflowCoreVersion },
},
},
{ v1Compat }
);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
TooTallNate
force-pushed
the
e2e-encryption
branch
from
ddfa99f to
2b43f36
Compare
This implements AES-256-GCM encryption with per-run key derivation via HKDF-SHA256 for workflow user data. Key changes: - Add encryption module (packages/world-vercel/src/encryption.ts) with createEncryptor() and createEncryptorFromEnv() functions - Add Encryptor, EncryptionContext, KeyMaterial interfaces to @workflow/world - Make all (de)hydrate* serialization functions async and accept encryptor - Add maybeEncrypt/maybeDecrypt helpers with 'encr' format prefix - Add getEncryptStream/getDecryptStream transform streams - Update runWorkflow() to take world as 4th parameter - Update WorkflowOrchestratorContext to include runId and world - Integrate encryption into WorkflowServerWritableStream/ReadableStream - Update hydrateResourceIO and observability helpers for async + world param - Update all test files with mock world and async serialization wrappers - Add 18 encryption unit tests covering round-trip, key isolation, tampering Format: [encr (4 bytes)][nonce (12 bytes)][ciphertext + auth tag] Signed-off-by: Nathan Rajlich <n@n8.io>
- Add .catch() error handlers to async hydrateStepReturnValue calls in hook.ts and step.ts to address PR review feedback - Update all (de)hydrate* function calls to use new async signatures: - hydrateWorkflowArguments(value, runId, encryptor, global?) - hydrateWorkflowReturnValue(value, runId, encryptor, ops?, global?) - hydrateStepReturnValue(value, runId, encryptor, global?) - Fix TypeScript errors in world-testing and workbench trigger endpoints - Add changeset for E2E encryption feature Signed-off-by: Nathan Rajlich <n@n8.io>
The E2E encryption feature requires the client to generate the runId before serializing workflow arguments (so it can use the runId for encryption context). Update world-local and world-postgres to use the client-provided runId from the run_created event data when present, instead of always generating a new runId server-side. Signed-off-by: Nathan Rajlich <n@n8.io>
For E2E encryption, the client generates the runId before serializing workflow arguments. Instead of passing runId inside the event data, pass it as the first parameter to events.create(). Changes: - Update Storage.events.create() interface to accept string | null for run_created events (null = server generates, string = client provided) - Update start() to pass runId as first argument instead of null - Remove runId field from RunCreatedEventSchema (no longer needed) - Simplify world implementations - they use the runId parameter directly - Update tests to expect runId as first argument Signed-off-by: Nathan Rajlich <n@n8.io>
Update e2e.test.ts and bench.bench.ts to use the new async dehydrateWorkflowArguments signature that takes (value, runId, encryptor, ops) instead of the old (value, ops, runId) order. Since the tests don't use encryption, we pass an empty string for runId and an empty object for encryptor. Signed-off-by: Nathan Rajlich <n@n8.io>
pranaygp
reviewed
Feb 9, 2026
Contributor
pranaygp
left a comment
pranaygp
left a comment
There was a problem hiding this comment.
Review: PR #950 - Add end-to-end encryption for workflow user data
This appears to be the original monolithic PR that has since been split into a cleaner stack: #978 -> #979 -> #956 -> #957. The split PRs are much easier to review incrementally. I'd recommend closing this in favor of the split stack unless there's a reason to keep it open.
Relationship to other PRs:
- #955 appears to be an intermediate iteration combining what became #978 + #979
- The active review stack is: #978 (async serde) -> #979 (Encryptor interface) -> #956 (Vercel AES-256-GCM) -> #957 (wire encryption)
Detailed reviews are on the individual PRs in the stack.
Member
Author
|
Good bot |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This implements AES-256-GCM encryption with per-run key derivation via
HKDF-SHA256 for workflow user data.
Key changes:
createEncryptor() and createEncryptorFromEnv() functions
Format: [encr (4 bytes)][nonce (12 bytes)][ciphertext + auth tag]