fix(evals): app readiness as interactive UI, shared predicate, den bootstrap, self-diagnosing failures

[benjaminshafii]

What

Follow-up to #3322, fixing the infrastructure-level reasons the app-driving eval specs could not pass, and making every remaining failure self-diagnosing.

1. App readiness is now defined by an interactive UI, not a route allowlist. The previous predicate accepted only /welcome or /workspace/<id>/session, so it rejected a genuinely-ready app: a fresh profile with no workspace legitimately sits on /session offering "Create or connect a workspace". Readiness is now: control API registered · no transitional copy on screen (Preparing workspace, Connecting signed-in services, Loading available resources, Loading tasks, Pulling in the latest messages — taken from apps/app/src/i18n/locales/en.ts, not invented) · one interactive surface observable (welcome | workspace | no-workspace).
2. One shared predicate. It lives in @openwork/cdp (app-state.ts) because both the lifecycle layer (@openwork/hosts) and the behaviours specs call (@openwork/behaviors) already depend on it, so there is no cycle and the two cannot drift. waitUntilInteractive() is exported for use after any action that navigates or creates a workspace/session.
3. Workspace id comes from the product's own state (openwork.react.activeWorkspace, verified against a live app) with the route only as a fallback — a selected workspace does not always appear in the hash.
4. Den-dependent specs point at the local den: bootstrap: { baseUrl, apiBaseUrl } = the den web origin, because the app derives its /api/den proxy base from those and den-api does not serve that prefix (documented in the legacy flow).
5. Per-call CDP timeouts are parameterised (send/evaluate accept timeoutMs; default stays 20s) and the polling helpers derive a budget from the wait's own deadline, so a busy renderer no longer kills a poll with CDP call Runtime.evaluate timed out after 20000ms.
6. Disposal is best-effort and logged, so a dead app no longer masks the real assertion error behind SuppressedError: An error was suppressed during disposal.

Tests run

pnpm evals:typecheck clean · pnpm --dir evals run test 91/91 · pnpm --dir evals run spec 4/4 (the gating pr lane) · pnpm --dir evals run spec:nightly app specs skip with their opt-in reason.

Real Daytona sandbox (den stack + isolated per-spec Electron), several runs: 5 of 9 spec files pass — both egress specs in both projects, plus app-smoke end-to-end (isolated spawn → readiness gate → assertions → vision-validated photo roll).

Remaining failures, now precisely known

Four files still fail, and the errors name themselves:

• Timed out after 120000ms waiting for created workspace selected (models-available, org-connection-lifecycle) and Workspace creation did not produce a workspace ID (skills-local) — workspace.create (real, registered by session-route.tsx:2308, requires an absolute path) reports success but the app does not settle on a workspace in a fresh profile. Needs live observation next, the way the onboarding sequence was pinned down.
• first-run-local: vision refused an expectation and was right — "The selected model reads 'big-pickle,' but the 'Run task' control is gray and visibly disabled." The spec asserts a usable model; the honest state without cloud or an in-app key is selected but not runnable. The expectation needs to state that truth.
• models-available: the duplicate-pixel guard fires on the Models-picker frame — the picker action runs but the frame is captured before it visibly renders, so the capture must wait for the picker.

None of this gates CI: the pr lane is green and gating; app specs are nightly-only, opt-in via OPENWORK_EVAL_APP_SPECS=1, and that step is continue-on-error while they stabilise.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
openwork-app	Ready	Preview	Jul 30, 2026 10:25am
openwork-den	Ready	Preview	Jul 30, 2026 10:25am
openwork-den-worker-proxy	Ready	Preview	Jul 30, 2026 10:25am
openwork-diagnostics	Ready	Preview	Jul 30, 2026 10:25am
openwork-landing	Ready	Preview, v0	Jul 30, 2026 10:25am