Production-Readiness Audit: missing CI / lint / benchmark / regression guardrails

[MichaelFisher1997]

Production-Readiness Audit: Missing Infrastructure, Lint, Benchmark & Regression-Prevention Guardrails

This is a tracking issue for a repo-wide audit of infrastructure, tooling, and process (not source code). The audit compared ZigCraft against what a production game engine repo would be expected to have. Someone should break this up into focused sub-issues — each section below is roughly one sub-issue.

The repo already has unusually strong automation for a project this size: Nix flakes, opencode AI workflows, actionlint, dependabot, 4 PR templates, AI test-writer/auditor/triage loops, path-filtered CI, and 75 test files. The gaps below are the difference between "CI exists" and "CI actually prevents regressions."

Scope: lint, benchmarks, regression prevention, security, safety, observability. Out of scope: release engineering (CHANGELOG, tags, GitHub Releases) — covered separately once a release is on the roadmap.

---

P0 — Critical (current gates are effectively no-ops)

1. Benchmark baseline is a placeholder

docs/benchmarks/baseline.json is all zeros, with the literal note:

"Populate this file with real benchmark baselines before enabling regression gating."

The benchmark workflow runs on every dev push, "compares" against this zeroed baseline, and publishes a green commit status. No regression can ever fail.

Fix:

• Run a real low/medium/high pass on known-good reference hardware (Blacksmith runner or a fixed self-hosted box).
• Commit real numbers to baseline.json.
• Flip scripts/compare_benchmarks.sh to exit non-zero on regression beyond tolerance.
• Document the hardware used in docs/benchmarks/README.md so future drift is interpretable.

2. zig fmt --check is not enforced in CI

• The format check only lives in .githooks/pre-push, which is opt-in (./scripts/setup-hooks.sh) and bypassable with git push --no-verify.
• The hook only checks src/, not modules/ or libs/.
• No PR-side enforcement at all.

Fix: Add a fast zig fmt --check src/ modules/ job (or step) to .github/workflows/build.yml that gates every PR.

3. Benchmarks & visual tests don't run on PRs

• benchmark.yml triggers on push: dev + workflow_dispatch only.
• visual-test.yml is workflow_dispatch only.

Performance and visual regressions land on dev and are detected after merge, when bisecting is harder.

Fix:

• Add pull_request: trigger to benchmark.yml (gated by path filter or label to control cost).
• Add a label-triggered (run-visual-test) or nightly scheduled run for visual-test.yml so it actually executes.

4. No frame-time / SLO assertion in benchmarks

Benchmarks report FPS percentiles (p1, p5, p50, p95, p99), CPU ms, GPU ms, draw calls, vertices — but nothing fails. A 50% FPS regression ships green.

Fix: Add absolute thresholds in the benchmark harness:

• p1 FPS floor per preset
• Spike guard: no single frame > N ms (e.g. 50 ms)
• Draw-call ceiling per preset

These are different from regression checks — they're absolute SLOs.

---

P1 — High (no safety/security net)

5. No code coverage measurement or gating

• 75 test files / ~100+ test cases, but no visibility into what's covered.
• No kcov / llvm-cov / codecov upload.
• No coverage threshold gate.
• No trend over time.

Fix: Wire kcov (Zig's best option today) into the unit-test job, upload to Codecov, add a non-blocking threshold comment first, then make it blocking once a baseline is captured.

6. No memory-safety test runs

Zig supports AddressSanitizer; Valgrind is also an option. Neither is wired into CI. For a voxel engine with manual allocators, packed structs shared with the GPU, and a job system passing chunk pointers across threads, this is high-value.

Fix: Add a sanitizer matrix job (zig build -Dsanitize=address test) running the unit suite at least nightly.

7. No Vulkan validation layer enforcement

Lavapipe is set up in the integration test, but there's no fail-on-validation-error gate. Shader misuse, wrong pipeline barriers, and binding bugs slip through silently.

Fix:

• Enable VK_LAYER_KHRONOS_validation with validate_core=true and best-practices enabled in CI.
• Set VK_LAYER_LOGGING=stderr and fail the integration/world-smoke steps on any validation error.

8. No security / supply-chain scanning

None of these exist today:

• SAST: CodeQL or Semgrep
• Dependency vulnerabilities: Trivy / Grype / nix flake show audit
• Secret scanning: gitleaks / trufflehog (PR + history)
• SBOM: CycloneDX / SPDX generation
• Provenance: SLSA / sigstore signing of build artifacts

The repo already grants id-token: write to build jobs, so signing/provenance is the natural next step.

9. No SECURITY.md / CODEOWNERS / CODE_OF_CONDUCT.md

• No documented vulnerability reporting path.
• No code ownership enforcement on PRs (reviewers aren't auto-requested per path).
• No contributor code of conduct.

Fix: Add the three files. Wire CODEOWNERS to trigger mandatory reviews for hot paths (modules/engine-graphics/, modules/world-persistence/, .github/workflows/).

10. Tested in one build mode only

CI runs zig build test in Debug only. Release-only UB (the kind that bites players) is invisible.

Fix: Add a ReleaseSafe matrix job to the unit-test step. ReleaseFast is overkill for correctness but ReleaseSafe catches optimization-sensitive UB.

---

P2 — Medium (lint & process hygiene)

11. No shellcheck

scripts/*.sh only gets bash -n (syntax check) in workflow-validation.yml. Real issues — unquoted expansions, SC2086, SC2046 — pass silently. The repo has 30+ shell scripts driving CI.

Fix: Add shellcheck to workflow-validation.yml with a config that's strict on errors and warnings.

12. No markdown / JSON / YAML linters

README, CONTRIBUTING, AGENTS, the 4 PR templates, issue templates, and docs/**/*.md are all unchecked. baseline.json and any future config files have no schema. YAML is only parse-checked via ruby.

Fix:

• markdownlint-cli2 for prose.
• prettier --check for JSON/YAML.
• Consider JSON schemas for baseline.json, labeler.yml, issue-labeler.json.

13. No conventional-commit enforcement

CONTRIBUTING.md / AGENTS.md mandate feat:/fix:/refactor:/test:/docs: but nothing enforces it on commit messages or PR titles. Commits drift from convention silently.

Fix: Add an amiquick / commitlint / PR-title-check action.

14. No DCO / CLA / commit signing

Contribution provenance is unverified. For an engine that may eventually be licensed or distributed, signed-off-by trails matter.

Fix: Add a DCO check (dco-app) as a required status check; optionally enforce GPG/SSH commit signing.

15. No nix flake check in CI

The flake could be malformed in ways that only show up outside the dev shell. Reproducibility story is undermined if the flake itself isn't validated.

Fix: Add nix flake check --no-build (or scoped variant) to workflow-validation.yml.

16. No stale issue / PR bot

Only stale branch cleanup exists (stale-branch-cleanup.yml). Issues and PRs can rot indefinitely.

Fix: Add actions/stale with sane exemption labels (pinned, roadmap, batch-*).

17. Visual regression is LLM-judged only — no deterministic golden-image diff

visual-test.yml feeds screenshots to MiniMax-M3 for verification. There's no deterministic perceptual/pixel comparison against a golden image.

Fix: Add an ImageMagick compare step (or pixelmatch, odiff) with a configurable tolerance. Use it as the primary gate, with the LLM as a secondary diagnostic. Otherwise silent visual regressions the model normalizes away will ship.

---

P2 — Medium (observability)

18. No crash reporting hook

No Sentry / Backtrace / minidump capture. Integration tests discard cores. In-dev crashes from testers are unattributable.

Fix: Wire a minimal crash handler (Breakpad / minidump) and an opt-in upload sink. Even just persisting minidumps as CI artifacts on integration failures would be a start.

19. No historical benchmark trend store

Each benchmark run overwrites the last. There's no Bencher / Conbench / InfluxDB / Hydra-style history, so month-over-month drift is invisible.

Fix: Push benchmark results to a trend store (Bencher is the obvious OSS pick; works well with GitHub Actions).

20. Single-platform CI

Linux + Lavapipe only. No Windows runner. No macOS runner. No MoltenVK path. Production engines test on every target platform — Lavapipe won't catch driver-specific issues that real users hit.

Fix: Add a Windows runner to build.yml (Vulkan on Windows is straightforward via SDK). macOS is harder (MoltenVK) — at least ensure it builds.

21. No automated profiling capture in CI

Tracy / Perfetto capture isn't automated. Profiling requires manual -D flags and a developer at the keyboard.

Fix: Add a nightly job that captures a Tracy frame for a fixed world and uploads it as an artifact so regressions are inspectable after the fact.

---

P3 — Lower (game-engine-specific guardrails)

22. No asset size guardrails

Textures and models can grow without a CI check. The repo already gitignores *_4k.* to keep 4K sources out, but there's no automated ceiling on what's committed or shipped.

Fix: Add a CI step that fails if any new texture exceeds a budget (e.g. 4 MB) or if total assets/ size grows beyond a threshold.

23. No draw-call / GPU-memory / vertex budget test

Benchmarks log draw_calls_avg, vertices_avg, GPU ms — but never assert a ceiling. A 3× draw-call regression ships green.

Fix: Add per-preset ceilings to the benchmark gate.

24. No shader compile-time / SPIR-V size regression

glslangValidator validates correctness in test, but doesn't fail on a 2× SPIR-V size growth or compile-time spike. Shader bloat creeps in.

Fix: Log SPIR-V size per shader in test, fail on >N% regression vs a baseline.

25. No fuzz / property-based tests

Worldgen has many determinism tests, but no fuzz harness. Edge cases that matter for a voxel engine — chunk coordinate boundaries, save-file corruption, malformed region files, integer overflow in light propagation — are untested.

Fix: Add fuzz harnesses for:

• Worldgen at extreme chunk coordinates
• Region file parsing with corrupted/short reads
• Light packing/unpacking round-trips
• AABB / ray-cast edge cases

26. No save-format backward-compatibility test

world-persistence has no "load a v0.1 save" golden fixture. A schema change can silently brick every existing world.

Fix: Commit a small golden save file per format version, add a test that loads it after every persistence change.

---

Suggested breakdown order

When this is split into sub-issues, the dependency order is roughly:

Wave	Items	Why first
1	#1 baseline.json, #2 fmt-check in CI, #11 shellcheck	Unblock all other CI improvements; trivial wins
2	#3 PR triggers for bench/visual, #4 SLO assertions, #17 golden-image diff	Make existing pipelines actually fail
3	#5 coverage, #6 sanitizers, #7 validation layers, #10 ReleaseSafe matrix	Catch real bugs
4	#8 security scanning, #9 SECURITY/CODEOWNERS/COC, #13 commitlint, #14 DCO	Hardening
5	#19 benchmark trend store, #18 crash reporting, #20 multi-platform, #21 profiling	Observability
6	#22–#26 engine guardrails (assets, draw calls, shaders, fuzz, save compat)	Engine-specific

Items #12 (markdown/json/yaml lint), #15 (nix flake check), #16 (stale bot) are independent quick wins that can land anytime.

---

Audit context

• Audited at: HEAD of dev
• Scope: infrastructure / tooling / process only (not source code)
• Excluded: release engineering (CHANGELOG, tags, releases) — separate scope once shipping
• Strengths already in place: Nix flakes, actionlint, dependabot (actions), 4 PR templates, AI test-writer/auditor/triage loops, path-filtered CI, 75 test files

/cc anyone picking this up — please comment on which wave/item you're starting before opening sub-issues so we don't duplicate.

[github-actions]

I have verified the key P0 claims directly in the repo (baseline.json is all zeros with the noted caveat, .githooks/pre-push only runs zig fmt --check src/, benchmark.yml triggers on push: dev + workflow_dispatch only, visual-test.yml is workflow_dispatch only). The audit is accurate. Posting a structured response comment now.

---

Comment (posted to issue)

Thanks for the thorough audit, Michael. I spot-checked the P0 claims against HEAD of dev and they all hold up:

• docs/benchmarks/baseline.json is the all-zeros placeholder (line 80 has the "Populate this file…" note). Confirmed no regression can ever fire.
• .githooks/pre-push:10 runs zig fmt --check src/ only — modules/ and libs/ are not covered, and the hook itself is opt-in.
• .github/workflows/benchmark.yml:3-9 triggers on push: dev + workflow_dispatch (no pull_request:).
• .github/workflows/visual-test.yml:3-4 is workflow_dispatch only.
• .github/workflows/build.yml runs zig build test in Debug only — no ReleaseSafe matrix.

I also searched recent commits and open branches — there is no in-flight work on any of the 26 items, so this is clean territory.

Classification

Tracking issue / Feature Request — infrastructure/tooling gaps, no source-code bug. Definitely too broad for a single PR. The wave breakdown in the body is sound and the dependency ordering is correct.

Recommended next step: split into sub-issues now

Rather than letting this rot as a 26-item checklist, I'd open one tracking sub-issue per item with a shared parent label (audit-2026-q3 or similar) and link them back here. That gives us:

1. Independently closable items (so people can pick up just one).
2. A real PR-driven workflow per fix instead of mega-PRs.
3. Per-item CI failure attribution when something regresses.

Concrete sub-issue titles (drop-in ready)

#	Suggested sub-issue title	Wave
1	ci(bench): populate docs/benchmarks/baseline.json and enable regression gating	1
2	ci(fmt): enforce zig fmt --check on src/ and modules/ in build.yml	1
3	ci(bench): run benchmarks on pull_request with path-filter / label gate	2
3b	ci(visual): nightly schedule + label-triggered run for visual-test.yml	2
4	bench: add absolute p1-FPS / spike / draw-call SLO assertions per preset	2
5	ci: wire kcov coverage, upload to Codecov, add non-blocking threshold	3
6	ci: add AddressSanitizer nightly test matrix	3
7	ci: enforce VK_LAYER_KHRONOS_validation fail-on-error in integration tests	3
8	ci: add CodeQL + gitleaks + Trivy + SBOM security pipeline	4
9	repo: add SECURITY.md, CODEOWNERS, CODE_OF_CONDUCT.md	4
10	ci: add ReleaseSafe matrix to unit-test job	3
11	ci: add shellcheck to workflow-validation.yml	1
12	ci: add markdownlint-cli2 + prettier --check (JSON/YAML)	any
13	ci: enforce conventional commits via PR-title-check + commitlint	4
14	ci: add DCO check + optional commit signing	4
15	ci: add nix flake check to workflow-validation.yml	any
16	ci: add actions/stale bot for issues and PRs	any
17	ci(visual): add deterministic golden-image diff (pixelmatch/odiff) as primary gate	2
18	engine: wire minidump capture + CI artifact upload on integration failures	5
19	bench: push results to Bencher for historical trend tracking	5
20	ci: add Windows runner (and macOS build-only) to build.yml	5
21	ci: nightly Tracy/Perfetto capture + artifact upload	5
22	ci: enforce per-asset and total assets/ size budgets	6
23	bench: enforce draw-call / vertex / GPU-memory ceilings per preset	6
24	ci: track and gate SPIR-V size regression per shader	6
25	test: add fuzz harnesses (worldgen edges, region parse, light packing, AABB/ray)	6
26	test(persistence): commit golden save-fixtures per format version, regression-test loads	6

Couple of small additions I'd suggest folding in

• Item 11 (shellcheck): the audit correctly notes scripts use bash -n only. A ready-made .shellcheckrc could land in this PR: enable=add-default-case,deprecate-which,quote-safe-variables and severity=warning.
• Item 14 (DCO): the existing repo-automation.yml and opencode-pr.yml already gate on AI review completion — DCO should plug into the same status-check family to avoid a second required-check sprawl.
• Item 17 (golden-image diff): Lavapipe renders are non-deterministic across versions, so the tolerance will need to be tuned. Worth flagging in the sub-issue description so it's not under-scoped.
• Item 20 (multi-platform): Blacksmith Linux runners handle Vulkan via Lavapipe, but Windows Vulkan CI needs the SDK pre-installed; the cheapest path is runs-on: windows-latest + subosaur/Vulkan-Setup-Action (or equivalent). macOS MoltenVK should be build-only, not test-only, in the first iteration.

Quick wins I'd prioritize for a first PR

If someone wants to ship measurable safety today, the lowest-risk/highest-impact trio is #2 (fmt-check in CI) + #11 (shellcheck) + #15 (nix flake check) — all three are config-only, no source changes, and each independently closes a P2 gap without touching anyone's workflow. Once those are green, the harder P0/P1 items (#1, #4, #7, #10) become much easier to land because the base CI is trustworthy.

What I'm NOT doing

Per the audit instructions in this issue (and standard tracking-issue etiquette), I am not opening the 26 sub-issues from here. Whoever picks up this tracking issue should open them — that way each sub-issue gets the right assignee, labels, and milestone up front, and this parent stays a clean index. Happy to draft the body text for any specific sub-issue if you point me at one.

---

Closing remarks

The audit is well-scoped and the wave ordering is correct. Strongly agree it should be split before any code work begins — the dependency graph (especially "make existing CI fail on real regressions" in wave 2 before "catch real bugs" in wave 3) only works if each item ships as its own PR. Marking this issue as accepted from my side.

opencode session  |  github run