Skip to content

feat(evals): evidence roll S1 — photo-roll collection view + sticky PR publisher + skill - #3417

Merged
benjaminshafii merged 2 commits into
devfrom
feat/evidence-roll
Aug 1, 2026
Merged

feat(evals): evidence roll S1 — photo-roll collection view + sticky PR publisher + skill#3417
benjaminshafii merged 2 commits into
devfrom
feat/evidence-roll

Conversation

@benjaminshafii

Copy link
Copy Markdown
Member

What

Slice S1 of the Evidence Roll program (spec: prds/evidence-roll/evidence-roll-program.md, included). Reuses the existing photoRoll proof primitives (#3322/#3359) and adds the two missing adapters — nothing new is captured, only rendered:

  • @openwork/evidence (zero new deps): tolerant scanRolls reader over evals/results/rolls/*/roll.json (legacy flow-lane dirs readable read-only), renderCollectionHtml (roll-of-rolls: cards with pass/fail badges, thumbnails, links to each roll's own index.html), renderPrMarkdown (claim → screenshot gallery with per-expectation PASS/FAIL), publishPr (Vercel Blob upload — env token → Infisical fallback, lifted from the deprecated runner/reporters/pr.ts, not imported — and a single sticky <!-- photo-roll --> comment, edited in place).
  • CLI: pnpm --dir evals run roll [-- --open] · pnpm --dir evals run publish:pr -- --pr <n> [--roll <dir|name>] [--dry-run].
  • Skill .opencode/skills/pr-photo-roll: judgment only (pick the roll, only roll.json-attributed images), mechanics in the CLI.
  • fraimz: photoRoll additionally records best-effort gitSha/branch (additive; readers tolerate absence).
  • Reserved (schema/docs only, S2): serialized surfaces + handoff entries for held-Daytona human testing; S3: snapshot-per-frame resumable rolls.

Flow lane untouched (evals/runner/** unchanged).

Tests (commands + results)

pnpm --dir evals run test        # 108 pass / 0 fail (incl. real-writer compatibility test)
pnpm --dir evals run typecheck   # clean

Real end-to-end: headless hosts.chrome() rendered the generated collection page → screenshot()validate() (OpenAI vision, both expectations PASS) → photoRoll → collection rebuilt → the sticky gallery comment below was posted by the new publisher itself (Blob-hosted images).

Known env finding (out of scope): hosts.desktop() local spawn on macOS brought up CDP but no page target in 120s (twice); app specs' home remains the Daytona nightly.

Proof format

Spec acceptance suites + the self-published gallery comment below, per program-owner decision (no fraimz flow lane).

@vercel

vercel Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
openwork-app Ready Ready Preview Aug 1, 2026 8:15pm
openwork-den Ready Ready Preview Aug 1, 2026 8:15pm
openwork-den-worker-proxy Ready Ready Preview Aug 1, 2026 8:15pm
openwork-diagnostics Ready Ready Preview Aug 1, 2026 8:15pm
openwork-landing Ready Ready Preview, v0 Aug 1, 2026 8:15pm

@benjaminshafii

Copy link
Copy Markdown
Member Author

Photo roll — evidence-roll-collection-proof

1/1 frames passed · 0 failed · 0 unvalidated · 2 expectations passed · 0 failed

1. A page titled as a photo roll collection lists at least one roll card

  • PASS A page titled as a photo roll collection lists at least one roll card — The page is titled “Photo rolls” and displays one bordered roll card for “evidence roll smoke.”
  • PASS A roll card shows a pass/fail style summary badge — The roll card displays the gold status badge “UNVALIDATED.”

A page titled as a photo roll collection lists at least one roll card


Roll created 2026-08-01T19:35:16.906Z · Source: evals/results/rolls/2026-08-01T19-35-16-906Z-evidence-roll-collection-proof/roll.json · Repro: pnpm --dir evals run publish:pr -- --pr 3417 --roll 2026-08-01T19-35-16-906Z-evidence-roll-collection-proof

Comment thread evals/packages/evidence/src/publish-pr.ts
@benjaminshafii
benjaminshafii enabled auto-merge (squash) August 1, 2026 20:10
Comment thread evals/packages/evidence/bin/publish-pr.mjs

@diff-warden diff-warden Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warden security clearance: clear. No new security issues found in this diff (4b09c908500d9a0ebfd7baf360a42b677e54f88d). Automated clearance satisfies the required-review gate only — a human still reviews and merges. Analysis run

@benjaminshafii
benjaminshafii merged commit 474ae34 into dev Aug 1, 2026
19 checks passed
@benjaminshafii
benjaminshafii deleted the feat/evidence-roll branch August 3, 2026 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant