feat(evals): typed runner core + automation/demo mode split

[benjaminshafii]

What

Decouples the eval runner from the voice-over/demo machinery and gives it a typed core, per the primitives plan (create the contract, delete the fusion, combine the entrypoints):

• Mode split — one engine, two policies. pnpm evals is now automation mode: no narration requirements anywhere (no drift check, no unnarrated-frame warnings, scriptless scaffold). pnpm fraimz is demo mode: byte-for-byte today's behavior (voice-over drift check enforced, narrated frames, script-first scaffold). Default mode for direct node evals/runner/run.mjs invocations stays demo, so existing automations/CI behave identically.
• Typed core on Node 24 native type-stripping. The runner is now TypeScript (evals/runner/*.ts) with no build step and no new runtime dependencies. run.mjs remains as a thin bootstrap so every existing entrypoint keeps working.
• Typed flow contract. defineFlow() + FlowContext (evals/runner/flow.ts): new flows are *.flow.ts with compile-time checking (pnpm evals:typecheck); a flow that misuses the ctx API fails tsc before it ever drives an app. All 200+ legacy *.flow.mjs flows load unchanged via compat shims (voiceover.mjs, cdp.mjs, pr.mjs, context.mjs, daytona-computer-use.mjs).
• Scaffold emits typed stubs. Narrated stub when an approved script exists; plain typed stub in automation mode when none does; demo mode still requires the approved script.
• report.json/fraimz.html now carry the run mode; nightly workflow pinned to pnpm fraimz --all so nightly keeps narration-drift coverage.

Why

The voiceover policy was fused into runFlow — a passing automation could be failed by narration drift, scaffolding required an approved script, and 150 flows imported narration plumbing just to exist. The engine and the demo policy are now separate primitives; voiceover is a layer on top. Typed ctx makes generated flow code verifiable before runtime — the contract agents compose against.

Tests run

• pnpm evals:typecheck — clean (runner + typed flows).
• pnpm evals:test — 4/4 runner unit tests pass.
• pnpm fraimz --flow voiceover-first-dx — the demo-path canary passes end to end (scaffold, drift-failure fixture, red-run exit codes, PR renderer).
• pnpm fraimz --flow typed-automation-dx — the new internal demo (frame proof posted below), including the voice-over coverage step.
• Real app on a Daytona Linux sandbox (.devcontainer/test-on-daytona.sh on this branch): pnpm fraimz --flow app-smoke --flow core-flow --cdp-url <sandbox> — both pass: app-smoke runs from the migrated .flow.ts, and the canonical core-flow journey (boot → new task → message → real model response via opencode zen → close/reopen → session intact) is unchanged. pnpm evals --flow app-smoke (automation mode) also passes against the live app.
• Decoupling fixture: the same drifted-narration flow exits 1 in demo mode ("Voice-over script coverage") and 0 in automation mode.

Frame proof (fraimz) posted as a PR comment below (typed-automation-dx + app-smoke + core-flow driven against the sandbox).

Notes / follow-ups

• core-flow's "agent responds" assertion is satisfiable by the user's own message text rendering (waitForText("core-flow ok") matches the typed message; the step-3 screenshot can fire before the reply streams). Pre-existing behavior, faithfully preserved by this refactor — worth tightening in a follow-up (assert on an assistant-role row).
• Phase 3 of the plan (a Target primitive: --target local | daytona-linux | daytona-windows + macOS CI lane) is designed but intentionally not in this PR.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
openwork-app	Error		Jul 22, 2026 7:35am
openwork-den	Ready	Preview, Comment	Jul 22, 2026 7:35am
openwork-den-worker-proxy	Ready	Preview, Comment	Jul 22, 2026 7:35am
openwork-diagnostics	Ready	Preview, Comment	Jul 22, 2026 7:35am
openwork-landing	Ready	Preview, Comment, Open in v0	Jul 22, 2026 7:35am

[benjaminshafii]

fraimz — ✅ PASSED

3 passed · 0 failed · 0 skipped — run 2026-07-21T20-52-10-807Z

Full frame proof with validated screenshots: evals/results/2026-07-21T20-52-10-807Z/fraimz.html (re-run: pnpm fraimz --flow app-smoke --flow core-flow --flow typed-automation-dx)

✅ app-smoke — App boots and renders a known route

1. The app rendered meaningful visible content after boot.

   • 📸 app-smoke-01-booted.png — 3 validations passed
   

✅ core-flow — Open app, send a message, get a response, reopen with session intact

1. App boots clean to a known route

   • 📸 core-flow-01-booted.png — 4 validations passed
   

2. A new session is created and becomes active

   • 📸 core-flow-02-session-created.png — 3 validations passed
   

3. The composer accepts a message and the agent responds

   • ✅ Visible text does not include "Something went wrong"
   • 📸 core-flow-03-task-response.png — 4 validations passed
   

4. Reopening restores the session and its message history

   • 📸 core-flow-04-reopened-session.png — 4 validations passed
   

✅ typed-automation-dx — Typed automation DX: automation mode, demo mode, scaffold, and typecheck stay distinct

Internal demo

1. pnpm evals runs automation mode while pnpm fraimz runs demo mode

   🎙 The eval runner now has two front doors. Running evals drives automations with no narration requirements, while fraimz stays the demo path. Same engine, two policies.

   • ✅ package.json evals script uses --mode automation
   • ✅ package.json fraimz script uses --mode demo
   • ✅ Plain fixture passes in automation mode
   • ✅ Plain fixture writes fraimz.html in its run dir
2. Narration drift fails demo mode and passes automation mode for the same fixture

   🎙 Here is the decoupling in one frame: a flow whose narration drifted from its approved script fails the demo path — and the exact same flow passes as a plain automation, because automation runs don't police narration.

   • ✅ Drifted fixture exits 1 in demo mode
   • ✅ Demo output includes the voice-over coverage step
   • ✅ The same drifted fixture exits 0 in automation mode
   • ✅ Automation output reports Result: PASSED
3. Automation scaffold writes a typed stub, while demo scaffold requires an approved script

   🎙 Scaffolding no longer requires a script. In automation mode, one command generates a typed flow stub with a compile-checked contract — and the demo path still refuses to scaffold without an approved script.

   • ✅ Automation scaffold without a script exits 0
   • ✅ Automation scaffold generated _plain-demo.flow.ts
   • ✅ Generated stub uses defineFlow
   • ✅ Generated plain stub does not contain voiceover
   • ✅ Generated plain stub does not load voice-over paragraphs
   • ✅ Demo scaffold without a script exits non-zero
   • ✅ Demo scaffold explains the approved-script requirement
4. The eval runner typechecks, and a context API misuse fails tsc

   🎙 Flows are TypeScript now. The whole runner typechecks, and a flow that misuses the context API is caught by the compiler before it ever drives an app.

   • ✅ pnpm exec tsc -p evals exits 0
   • ✅ A flow that misuses ctx.clickText fails typecheck
   • ✅ The compiler reports the wrong argument is not assignable
5. The real flow list still includes legacy flows, and runner tests stay green

   🎙 Nothing breaks behind us: two hundred legacy flows load side by side with the typed ones, and the runner's unit tests still pass.

   • ✅ Automation list exits 0
   • ✅ Flow list includes the typed app-smoke flow
   • ✅ Flow list contains at least 200 flows
   • ✅ Flow list includes legacy .mjs flow core-flow
   • ✅ pnpm evals:test exits 0
   • ✅ Runner unit tests report pass 4
6. Running without --mode still executes the voiceover-first demo with coverage

   🎙 And the demo path is untouched: the voiceover-first demo that guards this whole workflow still passes, drift check and all.

   • ✅ Default-mode voiceover-first-dx exits 0
   • ✅ Default-mode output includes the voice-over coverage step
   • ✅ Default-mode output reports Result: PASSED
   • ✅ Script frame 1 narrated: "The eval runner now has two front doors. Running evals drives automations with no narrat"
   • ✅ Script frame 2 narrated: "Here is the decoupling in one frame: a flow whose narration drifted from its approved sc"
   • ✅ Script frame 3 narrated: "Scaffolding no longer requires a script. In automation mode, one command generates a typ"
   • ✅ Script frame 4 narrated: "Flows are TypeScript now. The whole runner typechecks, and a flow that misuses the conte"
   • ✅ Script frame 5 narrated: "Nothing breaks behind us: two hundred legacy flows load side by side with the typed ones"
   • ✅ Script frame 6 narrated: "And the demo path is untouched: the voiceover-first demo that guards this whole workflow"