feat(evals): agent-first test engine — ambient evidence, bounded helpers, @openwork/testkit, fraimz:publish - #3431
Merged
Merged
Conversation
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Member
Author
Photo roll — testkit-boots-a-local-den-and-signed-in-app-with-ambient-evidence — 1/1 frames passed✅ 1/1 frames passed · 1 expectations passed · 0 failed ✅ PASS — 1. The OpenWork workspace shell is visible and ready for a task
Roll created 2026-08-02T15:12:09.948Z · Source: |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
benjaminshafii
enabled auto-merge (squash)
August 2, 2026 15:23
…wned helpers Phase 1 — the body cam: screenshot() auto-records every take into the current test's tape; validate() attaches claim verdicts (pass OR fail) by pixel hash; a vitest auto-fixture opens and closes the tape per test. Failed frames are recorded — a red run now leaves evidence instead of nothing. The duplicate guard moves to claims: retakes are legitimate, double-claiming one screenshot is not. photoRoll survives only as a deprecated shim; on-disk roll format is unchanged so scan/publish keep working. Phase 2 — hygiene by construction: every waiter has a finite default timeout; core waiters append an On-screen dump to their timeout errors; disposal kills the spawned process GROUP then verifies the surface's ports are actually free (the tsx-watch orphan and the respawning-Electron incidents); den cleanup retries once with a fresh session on 403 fresh_auth_required.
…ement flag, and fraimz:publish The resource language: a spec says 'I need 1 den and 2 apps' and the testkit materializes it. server() boots an isolated Den (ephemeral db, derived trusted origins, mocks contribute env AT BOOT, den-web included because desktop handoff resolves through its proxy) or wraps an existing one via OPENWORK_EVAL_DEN_API_URL; app() is a signed-in member desktop; needs() converts missing resources into a visible 'skipped — needs: X' verdict instead of a false green. OPENWORK_EVAL_DAYTONA=1 moves the same spec onto sandboxes; exposeMock() is the one seam where placement differs. Publishing is its own command: pnpm fraimz:publish -- --pr <n> selects a tape (newest or --roll), refuses stale evidence when the tape's git sha is not the PR head (--force annotates instead), renders failed frames with FAIL markers and fact frames without images, and never re-runs anything. Proof: testkit-selftest boots and fully disposes a real local Den (ports freed, database dropped) in ~18s; 134 unit tests.
…nd to end, skills for the new world
The reference spec migrates to the resource language: needs() carries every
gate (and names each missing env in its skip title), server({reuse}) wraps the
env-provided den, app() absorbs the member-desktop dance, and the tape records
witness facts alongside the vision frames. Its private surface helpers are
promoted into behaviors with their race comments intact.
testkit-app-boot proves the whole loop for real on a cold machine: boot an
isolated Den, boot a signed-in app, screenshot into the ambient tape, dispose,
ports freed and database dropped. The first cold run exposed workspace.create
exceeding the new 8s evaluate bound; the call site now allows 60s.
Skills: write-a-spec, run-tests, diagnose-a-red-run, publish-evidence,
build-a-witness encode the decision rules (baseline-first triage, claim
wording, witness negatives, kill-by-port forensics); pr-photo-roll is deleted;
fraimz and run-evals carry a legacy note pointing new work at specs+testkit.
…ps (CodeQL js/polynomial-redos)
The nightly workflow is deleted: there is no batch cadence to be red at. The vitest lane formerly named nightly is now 'stack' — named for what it requires (a real app + den), not for when it runs; app specs run on demand by whoever owes the proof, and post their tape to the PR. The fraimz and run-evals skills are tagged DEPRECATED in their descriptions so agents are routed to write-a-spec / run-tests / publish-evidence; they remain loadable solely to maintain existing evals/flows files. Coverage tradeoff stated plainly: flows no longer run on a schedule anywhere — the sweep, when wanted, is an agent running the stack lane on a capable host.
…, drop the duplicate module
benjaminshafii
force-pushed
the
feat/agent-testkit
branch
from
August 2, 2026 16:02
b4dd7ac to
603f3f7
Compare
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
… be LAN-reachable) den-api honors optional DEN_BIND_HOST and den-web dev:local honors DEN_WEB_HOST; defaults unchanged everywhere. The testkit pins both to 127.0.0.1 so an ephemeral dev-mode Den (whose email outbox is deliberately unauthenticated) is never reachable from the local network. Selftest re-run green bound to loopback.
Member
Author
|
Warden thread addressed with a real fix rather than a dismissal: eval Dens now bind loopback only (den-api DEN_BIND_HOST, den-web DEN_WEB_HOST, defaults unchanged; testkit pins both to 127.0.0.1). Selftest re-run green bound to loopback. Also in this PR since the last review: nightly retired (workflow deleted, lane renamed stack), fraimz/run-evals skills tagged DEPRECATED. |
…ds .github); spec:nightly aliases the stack lane Warden by design never self-clears a PR that edits review machinery, so the workflow deletion moves to its own human-approved PR. The restored workflow keeps working via the alias.
![diff-warden[bot]](https://avatars.githubusercontent.com/u/118544361?s=60&v=4){kind=small}
There was a problem hiding this comment.
Warden security clearance: clear. No new security issues found in this diff (1fb7b0a88636fc1d6f35c276d2a6f422fae386c4). Automated clearance satisfies the required-review gate only — a human still reviews and merges. Analysis run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuilds the eval engine around one principle: make it cheaper for an agent to tell the truth than to guess.
What changed
screenshot()auto-records every take into the current test's tape;validate()attaches claim verdicts — pass or fail — by pixel hash; a vitest auto-fixture opens/closes the tape per test. A red run now leaves evidence instead of nothing; cherry-picking is structurally impossible; the dup guard moves to claims (retakes legal, double-claiming one screenshot is not).photoRollis retired to a deprecated shim; on-disk roll format unchanged, so existing publish/scan tooling is untouched.On screen: {...}dump to timeout errors; disposal kills the spawned process group then verifies the ports are actually free; den cleanup retries once with a fresh session on403 fresh_auth_required. Each rule maps to a documented incident (20-min silent stall; EADDRINUSE zombie answering /health; respawning Electron helpers; teardown 403s).@openwork/testkit— the resource language. A spec saysneeds() → server() → app().server()boots an isolated Den (ephemeral db, derived trusted origins, den-web included, mocks contribute env at boot) or wraps an existing one viaOPENWORK_EVAL_DEN_API_URL;app({den,as})is a signed-in member desktop;needs()renders missing resources asskipped — needs: X, never green.OPENWORK_EVAL_DAYTONA=1moves the same byte-identical spec onto sandboxes;exposeMock()is the one seam where placement differs.pnpm fraimz:publish -- --pr <n>— publishing decoupled from running: selects a tape (newest or--roll), refuses stale evidence when tape sha ≠ PR head (--forceannotates), renders failed frames with FAIL markers and image-less fact frames, updates one sticky comment, never re-runs anything.org-connector-two-members(the reference spec) now speaks testkit with identical assertions and gating; five skills encode the decision rules (write-a-spec, run-tests, diagnose-a-red-run, publish-evidence, build-a-witness);pr-photo-rolldeleted;fraimz/run-evalsmarked legacy for new work.Tests run
pnpm --dir evals run test— 135/135pnpm evals:typecheck— cleanpnpm evals:spec(PR lane) — green, including the new app-less fixture spec actually runningtestkit-selftest.slow(real run): boots + fully disposes an isolated local Den in ~18s — ports freed, database droppedtestkit-app-boot.slow(real run): Den + signed-in Electron app + claimed frame on the ambient tape, cold start, single attempt (42.8s) — after fixing a real flake it caught (workspace.createvs the new 8s bound)skipped — needs: OPENWORK_EVAL_DEN_API_URL, OPENWORK_EVAL_APP_SPECS=1, OPENWORK_EVAL_CONNECTOR_SPEC=1, OPENWORK_EVAL_MODEL, OPENAI_API_KEY or ANTHROPIC_API_KEYEvidence tape posted below with the new command (its sha guard was exercised for real: tapes produced pre-commit are refused without
--force).Known-incomplete, stated plainly: the daytona
Placepath is typed and unit-tested but not live-provisioned in this PR; the migrated two-members spec still needs its provisioned placement to run end-to-end (unchanged from before); nightly-lane retirement is deliberately NOT in this PR — flows keep running until specs cover their ground.