Skip to content

test(evals): drop the reopen/persistence requirement from the core flow - #3123

Merged
benjaminshafii merged 1 commit into
devfrom
fix/core-flow-drop-persistence-requirement
Jul 25, 2026
Merged

test(evals): drop the reopen/persistence requirement from the core flow#3123
benjaminshafii merged 1 commit into
devfrom
fix/core-flow-drop-persistence-requirement

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

Problem

evals/flows/core-flow.flow.mjs is the canonical smoke proof referenced by AGENTS.md and the fraimz skill. Its 4th step — "User closes and reopens the app; the session survives"fails on clean dev. I hit it while validating an unrelated series and then reproduced it twice, in two freshly provisioned Linux sandboxes, with a real model:

✓ App boots to a usable session surface
✓ User creates a fresh task in the active workspace
✓ User writes a message and runs it
✗ User closes and reopens the app; the session survives
  — Session ses_… was not listed after reopen (not persisted).

It is a flow defect, not a product bug

The step reloads the renderer and then asserts:

await ctx.waitFor("…listActions().some((a) => a.id === 'session.list_sessions')", )
const sessions = await ctx.control("session.list_sessions");
ctx.assert(sessions.some((s) => s.sessionId === before), `Session ${before} was not listed after reopen (not persisted).`);

session.list_sessions (apps/app/src/react-app/domains/session/control/session-control-actions.ts) just reads the in-memory sessionsByWorkspaceId store. The flow waits for the action to be registered, not for that store to be hydrated after the reload — so it races hydration and can legitimately observe an empty list. The failure message ("not persisted") actively points the reader at the wrong layer.

Change

Removes that step, so the canonical flow is: open the app → create a task → write a message → get a response. Title and header comment updated to match, with a short note recording why reopen/persistence was dropped and what proving it properly would require (wait for store hydration, not action registration). Also corrects the two places that described the old step: .opencode/skills/fraimz/SKILL.md and .opencode/commands/fraimz.md.

New shape:

core-flow — Open app, create a task, write a message, get a response
  1. App boots to a usable session surface
  2. User creates a fresh task in the active workspace
  3. User writes a message and runs it

Verification

pnpm evals:typecheck    passed
pnpm evals:test         passed (4/4)
pnpm evals --list       lists the updated title
git diff --stat origin/dev...HEAD -- apps/ ee/ packages/   (empty — no product code touched)

evals/voiceovers/core-flow.md does not exist, so there is no narration to drift. Checked AGENTS.md and evals/README.md — no other stale step-level prose. No remaining references to core-flow-04 or session intact.

If we want session persistence proven, it belongs in a dedicated flow that waits for hydration — happy to follow up.

@vercel

vercel Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview, Comment Jul 25, 2026 3:54pm
openwork-den Ready Ready Preview, Comment Jul 25, 2026 3:54pm
openwork-den-worker-proxy Ready Ready Preview, Comment Jul 25, 2026 3:54pm
openwork-diagnostics Ready Ready Preview, Comment Jul 25, 2026 3:54pm
openwork-landing Ready Ready Preview, Comment, Open in v0 Jul 25, 2026 3:54pm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant