feat(evals): three proof primitives (screenshot / validate / photoRoll) + first core-journey specs

[benjaminshafii]

What

Replaces the bundled proof helper with three independent primitives for evidence, and starts moving the eval suite onto the core user journeys.

Before, fraimz(annotate) captured a screenshot and recorded it in one call, validated nothing, and coupled proof to vitest. Now:

const shot = await screenshot(app);                     // 1. capture — png + hash + route + visible text
const seen = await validate(shot, [                     // 2. judge — a vision model answers; returns facts
  "the connection is shown as connected to the member's own account",
  "no error or 'something went wrong' message is visible",
]);
expect(seen.ok, seen.why).toBe(true);                    // the assertion stays in the spec
await roll.add(shot, seen);                              // 3. record — evidence, separately

• screenshot(app) → Shot — capture only.
• validate(shot, expectations) → SeenFacts — describe-then-match (the model describes the screen with no knowledge of the expectations, then judges each one with an evidence quote, so it can't rubber-stamp), structured verdict, cached by pixel hash so re-runs are deterministic and free, transport injectable for tests. Throws when no provider key is configured — missing infrastructure is a throw, not a soft pass.
• photoRoll(name) → Roll — writes index.html + roll.json, and refuses duplicate pixels within a roll (the anti-fabrication guard the legacy runner had).

@openwork/fraimz now contains zero vitest imports, so the same three calls work in a spec, in a support script, or in the legacy runner. That deletes the custom-reporter design entirely.

Also in this PR: CI wiring for the spec lanes (pr gates every PR; nightly runs the app specs), four new behaviour modules (onboarding, composer, models, skills) extracted from legacy flows, three core-journey specs (first-run-local, skills-local, models-available), and an idempotent ensureReadyWorkspace so specs are order-independent.

Tests run

Local: pnpm evals:typecheck clean · pnpm --dir evals run test 91/91 (includes new network-free unit tests for photoRoll and validate) · pnpm --dir evals run spec 4/4 (the pr lane — real egress labs + shipped product diagnostics) · rg vitest evals/packages/fraimz/src → empty.

Real app + den stack in a Daytona sandbox (MariaDB + den-api + den-web + seeded org + Electron + OPENAI_API_KEY from the secrets volume), four full runs while iterating.

The primitives work end to end — first-run-local produced a 5-frame roll, 10/10 expectations passed, with genuine model descriptions:

welcome (no cloud)	composer reached

More importantly, on their first real runs the primitives caught two defects that DOM assertions missed:

1. Duplicate screenshot pixels for "The open Models picker visibly explains that a different model must be selected"; the same pixels were already added as "A Model no longer available warning…" — the spec claimed an interaction it never performed. Fixed by actually opening the picker via the real session.model_picker.open action.
2. Failed 1/2 visual expectations: The composer capability menu visibly shows Agents, Commands, Skills, and Extensions — Only centered text reading "Preparing workspace" is visible; no composer capability menu appears. — a premature assertion against a still-loading app.

Status of the app-driving specs — honest

The three new core-journey specs and the two retrofitted ones are not green yet against a real app: remaining failures are app-state readiness/restoration (onboarding reset semantics, restoring a deliberately-seeded unavailable model, waiting out "Preparing workspace"). They are nightly-only, skipIf(!OPENWORK_EVAL_CDP_URL), and the nightly step is continue-on-error with a comment, so nothing gates on them while they stabilise — they still publish photo rolls as evidence. The pr lane (egress labs) is fully green and now actually gates.

Also, first-run-local documents a product truth rather than papering over it: with no cloud and no in-app provider key, a first-run user reaches the task UI but cannot run a task ("Connect a model provider"), so the "runs a task and sees a response" half of evals/onboarding-welcome-flows.md's review bar is explicitly not proven yet.

Note

An earlier commit in this branch refactored 15 legacy flows away from composer.set_text on my incorrect claim that the action didn't exist. It does exist — it is conditionally registered by the session surface (session-surface.tsx:1381, requiresArgs: true), so my single snapshot was incomplete. That refactor is reverted; the behaviour now prefers the product's own control seam and keeps a DOM fallback.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
openwork-app	Ready	Preview	Jul 30, 2026 8:53am
openwork-den	Ready	Preview	Jul 30, 2026 8:53am
openwork-den-worker-proxy	Ready	Preview	Jul 30, 2026 8:53am
openwork-diagnostics	Ready	Preview	Jul 30, 2026 8:53am
openwork-landing	Ready	Preview, v0	Jul 30, 2026 8:53am

[benjaminshafii]

Update: readiness + isolation moved into the lifecycle layer

Per review feedback, the spec-level state management is gone. Readiness and isolation are now postconditions of acquiring the handle, owned by the lifecycle layer — the same contract startEgressLab() has (it doesn't return until the lab serves TLS):

await using app = await desktop({ name: "app-smoke" });   // fresh profile, ready, disposable

desktop() (in @openwork/hosts) spawns an isolated Electron (its own Vite port, CDP port, userData dir — already supported by spawnElectron), attaches, and returns only when the app is ready: control API present, no transitional text ("Preparing workspace", "Connecting signed-in services", "Loading available resources"), and a stable route (/welcome or /workspace/<id>/session with the task UI mounted).

Deleted, because they only existed to fake isolation on one shared app: resetOnboarding, ensureReadyWorkspace (and its invalid-model / den-reconnect recovery arms), clearDesktopDenSession, the reload-at-teardown, and every onTestFinished hook that mutated app state. first-run-local no longer resets anything — a fresh profile starts at /welcome natively, which is the point of that journey.

Real-run status (Daytona, den stack + per-spec Electron)

app-smoke now passes end to end with the new architecture: isolated spawn → readiness gate → assertions → vision-validated photo roll. Score is 5 passed / 4 failed files (was 4/5), and the remaining failures are journey-level rather than infrastructure:

Spec	Error	Cause
org-connection-lifecycle, models-available (managed)	Timed out waiting for persisted den auth token / active org resolved	the isolated app isn't pointed at the local den — needs bootstrap: { baseUrl, apiBaseUrl } = the den web origin (the /api/den proxy lives there)
first-run-local	CDP call Runtime.evaluate timed out after 20000ms	one heavy onboarding eval exceeds the CDP client's per-call timeout
all	SuppressedError: An error was suppressed during disposal	dispose should log rather than throw when the app is already gone

Three environmental blockers were fixed along the way to get isolated Electron running inside a container at all: Chromium's SUID sandbox (ELECTRON_DISABLE_SANDBOX=1), missing DISPLAY in exec sessions (Xvfb runs on :99), and pnpm prompting non-interactively.

CI is unaffected: the pr lane (egress labs + product diagnostics) is green and gating; app specs are nightly-only, opt-in via OPENWORK_EVAL_APP_SPECS=1, and the nightly step is continue-on-error while they stabilise.