fix(evals): make specs placement-portable — host workspace root, sandbox-aware ports, profile pruning - #3388
Conversation
…he driver's A spec passing process.cwd() as a workspace path is only correct when the driver and the app share a filesystem. Driving a sandbox from a laptop, the app was asked to open a directory that exists only on the laptop — observed as onboarding hanging 120s on 'Power your first task' with no error, which reads like a broken app rather than a bad path. Host.workspaceRoot states it (repoRoot locally, /workspace on Daytona) and DesktopHandle re-exposes it, so app-smoke asks the app's own host.
…andbox The local host finds a free port by binding one; the Daytona host cannot, because the port must be free on another machine. It allocated from a local counter instead, so the OpenCode sidecar — which picks its own port at boot — was observed holding 9825, the CDP primary. Electron's debugger never bound and the preview URL timed out after 180s with nothing naming the cause. Ask the sandbox which ports are listening and skip those.
Routing app-smoke through Host.workspaceRoot made it open the whole monorepo (/workspace, node_modules included). The engine scan then blocked the renderer past 240s and the spec failed reading location.hash — a regression versus the old process.cwd() (which happened to be the smaller /workspace/evals). workspaceRoot stays the right primitive for specs that genuinely need the repo (skills-local reads .opencode/skills); a workspace path just needs to be small and valid on the app's host.
Disposal removes a profile on the happy path, but every killed or failed run leaks one at ~50MB. Twenty-seven filled a 10GB sandbox to 99%, and the symptom was not 'disk full' — it was the renderer failing to answer Runtime.evaluate for 240s, indistinguishable from a broken app. Cost four debug cycles to find. clearStaleSurfaces already pkills by rootDir and so already assumes exclusive ownership; deleting the directories it just orphaned is the matching half.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…nnector and call its tools
Two SEPARATE desktops, own profiles, signed in as different members. A single
desktop clearing localStorage only simulates per-member isolation; two make it
real — the load-bearing assertion is that member B still needs their own
sign-in after member A connected.
The connector is the authority on use: mock MCP now records tool ARGUMENTS and
a sha256 fingerprint of the caller's bearer token, so the spec asserts on calls
actually served and on DISTINCT credentials rather than the app's own
'Connected' text.
Placement: both desktops use the ambient host; moving either to its own sandbox
is desktop({ host: daytonaSandbox(id) }) once mode B is green.
…ws is the test ensureMemberSession threw when mark-verified failed, even though the member had just been created successfully and could sign in. On the eval sandbox the helper uses -uroot with no password while den's MariaDB root has one, so bootstrapping a second member aborted the whole spec at 2s. Warn and continue; if the subsequent sign-in fails, report both causes.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…the way the proven spec does
waitForText reads innerText, which is render-aware: a badge styled with CSS text-transform reads NEEDS YOUR SIGN-IN on screen while textContent holds 'Needs your sign-in'. The reveal could never agree with the wait.
Status: 26 of ~29 steps green, including the headline assertionI was wrong about the product regression — retracting itI claimed The real cause was mine. The app canonicalises the route — What now passesEverything through member B connecting their own account:
What is still blocked, and why it argues for the original designThe tool-call phase (steps 27–29) never starts: This is resource exhaustion, not product behaviour. Two Electrons, two OpenCode engines, den and MySQL on one 9GB sandbox. Which is an argument for the two-sandbox topology this branch was groundwork for: one desktop and one engine per sandbox removes the ceiling entirely. Cheapest next step without more hardware: reorder so only one desktop is heavily active — A connects → A's tool call → then create B's desktop → B sees Resilience fixes landed while getting here
The spec stays opt-in ( |
publicUrl was a dead seam: waitForHealth read child?.exitCode on the null child and declared the (never-spawned) server exited, and the request log of a long-lived mock satisfied atLeast with a PREVIOUS run's tool calls, returning before this run's calls arrived. Guard the health check to only fail on a spawned child that exited, carry the mock's timestamp on each tool call, and let callers scope toolCalls with sinceIso.
… sandbox
Two desktops plus two engines starve renderers on one 9GB sandbox — that
ceiling, not any product defect, is what kept the tool-call phase red, so
the header's retracted defect story goes too. OPENWORK_EVAL_DAYTONA_SANDBOX_A/_B
now place each member via desktop({ host: daytonaSandbox(id) }), the mock
publishes through OPENWORK_EVAL_CONNECTOR_MOCK_PUBLIC_URL so Den, both
desktop browsers, and the driver can all reach it, and this run's tool
calls are scoped with sinceIso.
Green twice end to end (133.0s, 135.2s): both members OAuth-connect their
own account, call mock_echo from the composer, and the connector witnesses
two distinct bearer fingerprints.
org-connector-two-members: green on two Daytona sandboxes — twiceThe last opt-in spec now completes its tool-call phase. The fix was placement, as predicted: each member's desktop gets its own sandbox via Runs (driver on a Mac outside all sandboxes, mode B):
Topology: Den server sandbox ( The connector's own witness log (the authority, not the UI): each run served both markers under distinct bearer fingerprints — Two real bugs found by running it:
Vision-validated frames (run 2,
Also ran: The spec header's earlier "blocked on a product defect (#3375)" claim is now retracted in the code itself; the honest reason it stays opt-in is that it needs provisioned placement (two sandboxes + a reachable Den + a published mock), not that anything product-side is broken. Repro: |
CodeQL: js/polynomial-redos — ^Bearer\s+(.+)$ backtracks polynomially on 'Bearer' followed by header spam. One shared linear parse for the auth check and the fingerprint.
|
Addendum: CodeQL flagged a polynomial-ReDoS in the mock's bearer-header regex ( CodeQL, openwork-tests (ubuntu + macos), and both Analyze jobs now pass on the tip. That makes four greens for this spec today (133.2s, 133.0s, 135.2s, 127.9s). |
|
@OmarMcAdam @src-opn — requesting your approval; auto-merge (squash) is armed, so one approval ships it. 30-second review guide:
|
3 participants
Groundwork for the two-member connector spec (two Electrons in two sandboxes, den + connector reachable from both). I set out to write that spec; driving mode B for the first time surfaced three real bugs that had to land first. The spec itself is not in this PR — see "What's left".
Three bugs, each found by running
1. Specs assumed the driver's filesystem is the app's.
app-smokepassedprocess.cwd()as a workspace path. Correct only when driver and app share a disk. Driving a sandbox from a laptop, the app was told to open a directory that exists only on the laptop — the symptom was onboarding hanging 120s on "Power your first task" with no error, reading like a broken app.Host.workspaceRootnow states it (repoRootlocally,/workspaceon Daytona) andDesktopHandlere-exposes it, for specs that genuinely need the repo (skills-localreads.opencode/skills).2. Daytona surface ports were allocated from a local counter. The local host finds a free port by binding one; the Daytona host cannot, since the port must be free on another machine. It guessed from
{ primary: 9825, next: 9830 }— and the OpenCode sidecar, which picks its own port at boot, was observed holding 9825, the CDP primary. Electron's debugger never bound; the preview URL timed out after 180s naming nothing. Now it asks the sandbox which ports are listening and skips them.3. Stale surface profiles filled the disk. Disposal removes a profile on the happy path; every killed or failed run leaks one at ~50MB. 27 of them took a 10GB sandbox to 99% full. The symptom was not "disk full" — it was the renderer failing
Runtime.evaluatefor 240s. That cost four debug cycles.clearStaleSurfacesalready pkills byrootDirand so already assumes exclusive ownership; it now deletes the directories it orphans.A regression I caught and reverted
Routing
app-smokethroughworkspaceRootmade it open the whole monorepo (/workspace,node_modulesincluded) instead of the smaller/workspace/evalsthatprocess.cwd()happened to give. Engine scan blocked the renderer past 240s. It now uses a fresh/tmp/openwork-app-smoke-<stamp>— small, and valid on any host. Only caught because I re-ran mode A after each change; worth keeping that habit.Proof
Mode A (in-sandbox), after the disk was freed:
app-smokeskills-localPruning verified:
profiles-left=1(the current run only) where 27 had accumulated; disk steady at 84%.tsc -p evalsclean.pnpm --dir evals test95 pass / 0 fail. ThreeHostfakes in runner tests neededworkspaceRoot— the type system caught them, which is the seam working.Mode B status, honestly
Mode B (vitest on a laptop, Electron in a sandbox, CDP over a preview URL) went from hanging at 120s to completing workspace creation in 68.5s. It is not yet green end to end. Two things remain, both documented rather than fixed:
validate()runs driver-side, soOPENAI_API_KEY/ANTHROPIC_API_KEYmust be on the driver —daytona-spec.shsources/daytona-secrets/*.envinside the sandbox, which does nothing for mode B. Same family as bugs 1 and 3: something moved sides.What's left for the connector spec
startMockMcp({ host })— labs are local-only.publicUrlalready exists as an attach hatch, so a genericHost.startService()is the clean shape.captureOpenedUrls({ host })— thexdg-openshim lands on the driver, not where Electron runstoolNames+authorized)requires([...])capabilities, model knobNote this spec cannot run in nightly — nightly is
blacksmith-4vcpu-ubuntu-2204with no Daytona access. That is why the capability model matters: without it, it would skip silently and read green.