[B1] ci(bench): populate docs/benchmarks/baseline.json and enable regression gating

[MichaelFisher1997]

Part of #834 (Production-Readiness Audit) — Batch B1 (foundation, parallel-safe).
Replaces audit item #1 (P0). Parallel with: audit-b1 siblings — this issue owns docs/benchmarks/* + scripts/compare_benchmarks.sh.

Why P0: this is the audit's #1 critical finding. Today no benchmark regression can ever fail CI.

Problem

docs/benchmarks/baseline.json is an all-zeros placeholder (line 80 literally says "Populate this file with real benchmark baselines before enabling regression gating."). The benchmark workflow runs on every dev push, "compares" against the zeroed baseline, and publishes a green commit status. No regression can ever fail.

Acceptance criteria

• A real low/medium/high preset pass is captured on fixed reference hardware (Blacksmith runner or a pinned self-hosted box) — pick one and document it.
• Real numbers committed to docs/benchmarks/baseline.json (replace all zeros).
• scripts/compare_benchmarks.sh flipped to exit non-zero on regression beyond tolerance.
• docs/benchmarks/README.md documents: hardware used, driver/Lavapipe version, Zig version, capture procedure, tolerance policy — so future drift is interpretable.
• Tolerance is explicit (e.g. ±10% p1-FPS) and applied per metric.

Files in scope

• docs/benchmarks/baseline.json
• docs/benchmarks/README.md
• scripts/compare_benchmarks.sh

Dependencies

• Unblocks: B4-1 (bench PR trigger + Bencher), B4-3 (SLO assertions). Those need a real baseline to compare against.

Out of scope

• SLO assertions themselves (those are B4-3 / audit feat: worm cave system with connected underground networks #4).
• Bencher trend store (B4-1 / audit 🔍 Conduct Comprehensive Code Audit - Architecture, Performance, SOLID, and Quality Standards #19).

Branch / PR

• Branch: ci/bench-baseline-real · Targets: dev

[github-actions]

Validation

Confirmed — the gate is a no-op. Root cause is two-fold and worth pinning in the PR description so reviewers don't have to re-derive it:

1. docs/benchmarks/baseline.json:80 literally carries the placeholder note ("Populate this file with real benchmark baselines before enabling regression gating.") and "generated": false.
2. scripts/compare_benchmarks.sh:42-47 reads .generated and exit 0s before any threshold logic when the flag is false:

   baseline_generated=$(read_json '.generated' "$baseline")
   if [[ "$baseline_generated" != "true" ]]; then
       printf 'Baseline placeholder detected, skipping comparison.\n'
       exit 0
   fi

3. .github/workflows/benchmark.yml:64-69 loops over low medium high calling compare_benchmarks.sh, and the publish-status block at lines 82-105 maps compareOutcome === 'skipped' to a green "Benchmark completed (baseline placeholder)" status. So every dev push gets a green ✅ from a workflow that compared nothing.

Notes for the implementer

A few things to align before this lands — none of them require scope expansion:

• The "flip" is mostly already wired. compare_benchmarks.sh already exits 1 on FPS regression (scripts/compare_benchmarks.sh:91-93) and exposes --warn / --fail flags (scripts/compare_benchmarks.sh:22-30). The real change is: capture real numbers, set "generated": true, and verify the threshold defaults (warn_threshold=10, fail_threshold=20 on line 4-5) are appropriate for Lavapipe noise. Suggest running 3-5 back-to-back captures on blacksmith-2vcpu-ubuntu-2404 first to size the noise floor before committing the baseline.

• AC says "p1-FPS" but the script reads .fps.avg (scripts/compare_benchmarks.sh:70). p1 (worst 1%) is the right SLO metric for a voxel engine (chunk-load hitches show up there, not in averages), so the script should be updated to use .fps.p1 (or both p1 and avg) before flipping the gate on. This is also what audit item feat: worm cave system with connected underground networks #4 (SLO assertions) will key off of, so getting it right here pays back.

• Per-metric tolerance is incomplete. AC says "Tolerance is explicit and applied per metric," but today gpu_ms.total_avg and draw_calls_avg only emit warnings (scripts/compare_benchmarks.sh:87-89) and never trigger exit 1. Either tighten the script to fail on those too, or explicitly document in README.md why only FPS is gating (e.g. GPU/draw-call noise on Lavapipe is too high for a hard gate at this stage). The vertex_avg and per-preset chunks_rendered_avg fields in the baseline are never even read by the comparator.

• README capture procedure — .github/actions/setup-lavapipe already pins a Mesa version, so capture the resolved mesa-vulkan-drivers / lavapipe version plus the Zig commit SHA and the nix develop .#ci-graphics rev into docs/benchmarks/README.md alongside the hardware SKU (blacksmith-2vcpu-ubuntu-2404). Without those, future drift is uninterpretable, which is the whole point of the AC.

• generated: true flag is a foot-gun once any preset's real numbers exist but others are still placeholders. Consider either gating per-preset (compare script already supports --preset, so the workflow could just skip a preset if its entry has generated: false) or removing the top-level flag entirely and keying off the presence of non-zero values. Not blocking, but worth deciding before the file shape is "real."

Scope

Well-scoped as-is — 3 files, clear AC, no subsystem spillover. No need to split. Correctly identified as parallel-safe with other audit-b1 siblings; the only coupling is that B4-1 (Bencher) and B4-3 (SLO assertions) consume this output, which is correctly noted in the Dependencies section.

Suggested PR checklist (for the implementer)

• 3-5 back-to-back benchmark runs on blacksmith-2vcpu-ubuntu-2404 to characterize Lavapipe noise floor
• Update compare_benchmarks.sh to read .fps.p1 (not .fps.avg) and add a fail exit on gpu_ms.total_avg / draw_calls_avg regressions per the AC's per-metric tolerance requirement
• Capture single run → commit baseline.json with "generated": true
• Add docs/benchmarks/README.md with hardware, lavapipe/Mesa version, Zig SHA, capture procedure, tolerance policy, and refresh cadence
• Verify .github/workflows/benchmark.yml:82-105 commit-status branch handles the new failure path correctly (it already does, but worth a sanity test before merge)

[github-actions]

Comment posted on issue #861. Validated the finding (gate is a no-op via compare_benchmarks.sh:42-47 placeholder skip + benchmark.yml:82-105 green status mapping), noted that the "flip" is mostly already wired but called out three alignment items: AC says p1-FPS but script reads .fps.avg, GPU/draws only warn not fail (per-metric tolerance gap), and generated: true flag handling. Confirmed scope is appropriate — no split needed.

opencode session  |  github run