[B4] ci(visual): nightly/label trigger + deterministic golden-image diff as primary gate

[MichaelFisher1997]

Part of #834 (Production-Readiness Audit) — Batch B4 (make bench/visual actually fail, parallel-safe).
Replaces audit items #3b + #17 (merged because both touch the visual-test workflow + harness). Parallel with: audit-b4 siblings — this issue owns visual-test.yml + the visual harness.

Problem

Two related gaps sharing one area, so they ship as one PR:

1. #3b — visual tests never run. visual-test.yml is workflow_dispatch only. Visual regressions are never caught.
2. Refactor: Split main.zig into modular game components #17 — visual regression is LLM-judged only. visual-test.yml feeds screenshots to MiniMax-M3 for verification. There's no deterministic perceptual/pixel comparison against a golden image, so silent visual regressions the model normalizes away will ship.

Acceptance criteria

• visual-test.yml runs on a nightly schedule and on a run-visual-test label trigger.
• Deterministic golden-image diff added as the primary gate: pixelmatch / odiff / ImageMagick compare, with configurable tolerance.
• LLM verification kept as a secondary diagnostic (comment-only, not gating) until golden-image confidence is established.
• Golden images committed under docs/visual-test/golden/ (or similar) with a regeneration procedure documented.
• Tolerance tuned for Lavapipe non-determinism across versions (flag in the workflow so it's not under-scoped).

Files in scope

• .github/workflows/visual-test.yml
• visual-test harness source (wherever screenshots are produced/compared)
• docs/visual-test/golden/ (new — golden images)
• comparison script (scripts/compare_images.sh or equivalent)

Dependencies

• Coordinate with B4-1 (bench/Bencher) — different workflow file, parallel-safe.

Out of scope

• New test scenes (use existing visual-test scenes as the golden set).
• Replacing the LLM judge entirely (it stays as diagnostic).

Branch / PR

• Branch: ci/visual-golden-diff · Targets: dev

[github-actions]

Analysis

Classification: Feature Request (CI workflow + harness enhancement). Scope is appropriate for a single PR — one workflow file, one new script, one new docs directory — and not over-broad.

Validation against parent audit #834

I confirmed items #3b + #17 in the parent issue. Relevant excerpts:

• Refactor: Split main.zig into modular game components #17 explicitly says: "Add an ImageMagick compare step (or pixelmatch, odiff) with a configurable tolerance." — this issue delivers exactly that.
• #3b says: "Add a label-triggered (run-visual-test) or nightly scheduled run for visual-test.yml so it actually executes." — run-visual-test is intended as a label name (distinct from the existing visual-test label that visual-test.yml itself creates at visual-test.yml:50-54).

No duplicate PRs/branches exist locally for golden-diff work; no docs/visual-test/, scripts/compare_images.sh, or any pixelmatch/odiff usage exists yet — this is greenfield.

Codebase findings

• visual-test.yml:1-135 — confirmed workflow_dispatch only, LLM-only gate, no deterministic diff.
• visual-test.yml:82 — ImageMagick is already in the toolchain via nix run nixpkgs#imagemagick, so adopting compare as the v1 diff backend adds zero new dependencies. pixelmatch/odiff would require Node setup.
• .github/prompts/visual-test-verify.md:26-27 — the LLM is already told "Be lenient with software rendering artifacts (Lavapipe may not render identically to a real GPU)", confirming Lavapipe non-determinism is a real, acknowledged concern that warrants the tolerance-flag acceptance criterion.
• scripts/compare_benchmarks.sh — existing pattern (threshold flags, set -euo pipefail, exit code on regression) is the right template for scripts/compare_images.sh.
• .github/workflows/opencode-audit.yml:9-21 and opencode-test-writer.yml:9-21 — schedule + concurrency-key patterns (cron: '0 6 * * 1', ${{ github.workflow }}-${{ inputs.module || 'scheduled' }}) to mirror.
• .github/actions/setup-lavapipe/action.yml — currently doesn't pin nixpkgs#mesa.drivers; the "tolerance tuned for Lavapipe non-determinism across versions" criterion is best satisfied by pinning and capturing the resolved Mesa/Lavapipe version into $GITHUB_STEP_SUMMARY so golden regeneration is reproducible.
• Screenshot env (visual-test.yml:69-75): Lavapipe + headless Weston at the game's native size with ZIGCRAFT_SMOKE_FRAMES=5, ZIGCRAFT_SAFE_RENDER=1. The golden set must be regenerated under this identical env or the diff will be meaningless.

Clarifications requested before implementation

1. Label trigger scope — confirm whether run-visual-test should trigger on issue labeled (issues: types: [labeled] with names: filter) or PR labeled (pull_request: types: [labeled]). These have different on: syntax and semantics; pinning this down prevents rework.
2. Concurrency on schedule — current key ${{ github.workflow }}-${{ github.ref }} will cancel scheduled runs if a dispatch hits the same ref. Sibling B4 workflows use ${{ github.workflow }}-${{ inputs.module || 'scheduled' }} with cancel-in-progress: false. Decide whether nightly should be cancellable.
3. Diff backend for v1 — recommend ImageMagick compare -metric fuzz (already in nix shell) for v1 to avoid a Node toolchain dependency, then upgrade to odiff once the golden set stabilizes and perceptual diffs are needed. The author should pick.
4. Lavapipe version pinning — extend setup-lavapipe to pin nixpkgs#mesa.drivers and surface the resolved version so future golden regeneration can detect driver drift.

Suggested implementation outline (not coding it)

• Add schedule: - cron: '0 6 * * 1' and a labeled: trigger for run-visual-test to on:.
• Update concurrency key to handle scheduled runs (mirror B4 sibling pattern).
• Author scripts/compare_images.sh modeled on scripts/compare_benchmarks.sh: args = golden PNG, new PNG, --tolerance flag, exit non-zero on threshold breach, write a side-by-side diff PNG on failure for the artifact upload.
• New docs/visual-test/golden/ directory with one PNG per current visual-test scene, plus a README.md describing the regeneration procedure (re-run workflow, copy screenshot.png from artifact, commit).
• Wire comparison step between Convert PPM to PNG and Upload screenshot artifact in visual-test.yml; have it exit 1 on diff failure and upload both the candidate PNG and the diff PNG as artifacts.
• Demote the opencode verify step: continue-on-error: true, never let its failure() fail the job, and gate the issue-filing branch to if: failure() && steps.golden-diff.outcome == 'failure' only — so the LLM becomes purely diagnostic.
• Extend setup-lavapipe to pin Mesa and emit the resolved version, then pass it as a tag in the golden filename or $GITHUB_STEP_SUMMARY for traceability.

Scope verdict

One PR is fine. No need to split. Coordinate with B4-1 (bench/Bencher) per the issue's dependency note — different workflow file, so parallel-safe as stated.

[github-actions]

Comment posted to issue #874: #874 (comment)

Summary of the analysis:

• Classified as a Feature Request with appropriate single-PR scope.
• Validated against parent issue Production-Readiness Audit: missing CI / lint / benchmark / regression guardrails #834 — items #3b and Refactor: Split main.zig into modular game components #17 are confirmed; no duplicate work exists locally.
• Found 4 ambiguities the author should pin down before coding: label trigger target (issue vs PR), concurrency key behavior on schedule, diff backend choice, and Lavapipe version pinning.
• Outlined an implementation path that mirrors existing patterns (scripts/compare_benchmarks.sh, the B4 sibling workflow schedule+concurrency shape) and reuses ImageMagick already in the nix toolchain.
• No code changes made — comment-only as required.

opencode session  |  github run