test(backend): Phase 1 golden-snapshot regression gate - #96
Conversation
ASA is server-side Python on native Essentia and already computes a superset of the crate, so a same-class cross-check oracle would be decorative (cross-impl agreement != correctness; the crate's R128 is from-scratch, no ebur128 dep). tonnetz, the only gap, has no consumer and is dropped -- Harmonia can derive it from the chroma ASA already exports. Python-native validation is tracked separately, trigger-gated. Supersedes Plan 4's "cheap and high-value" oracle recommendation in incorporations/forking-plans-2026-05-14.md. https://claude.ai/code/session_018hYXgv4LK9Hm1BNGWgMZf8
Snapshot default-mode analyze.py output against a committed golden so any unintended drift in a measured field fails CI -- a safety net for upcoming analyze.py/Essentia changes. - tests/test_phase1_golden.py runs analyze.py on a deterministic procedural fixture and diffs the JSON against the golden with a recursive tolerant comparator (NaN/Inf-aware, bool-before-int, key-set + list-length drift, capped report). Output was byte-identical across 5 runs; the tolerance is cross-environment float-jitter insurance, not run-to-run noise. - Committed gate-bites meta-tests prove the comparator catches drift and does not over-bite (incl. in-memory NaN/Inf/bool and cap cases). - melodyDetail.midiFile (a transient per-run artifact path) is the sole excluded field, stored in the golden as a placeholder. - Regenerate after an intentional change with UPDATE_PHASE1_GOLDEN=1. No analyze.py / schema / requirements.txt change. https://claude.ai/code/session_018hYXgv4LK9Hm1BNGWgMZf8
slittycode
left a comment
slittycode
left a comment
There was a problem hiding this comment.
Verdict: APPROVE
(GitHub won't let you approve your own PR, so this is a COMMENT — verdict is approve.)
Summary
Test-only PR adding a golden-snapshot regression gate over default-mode analyze.py output, plus a decision doc. No production code touched. The comparator implementation is sound — bool-before-int ordering is correct (critical Python gotcha), NaN/Inf are handled, key-set diff catches additions and removals, MAX_REPORTED cap prevents multi-thousand-line noise on large-array drift. Ten meta-tests confirm the comparator bites correctly and doesn't over-bite. Frontend ✅ and Loudness WASM ✅ are green; Backend CI was in progress at time of review.
Findings
Should fix (pre-existing, not introduced here):
The committed golden captures 65 top-level keys from full-mode analyze.py. EXPECTED_TOP_LEVEL_KEYS in test_analyze.py:26 has 59. Six full-mode fields are absent from that set: keyProfile, lufsMomentaryMax, lufsShortTermMax, pitchDetail, tuningCents, tuningFrequency. The fast-mode test at test_analyze.py:342 claims "Fast mode must emit the same top-level key set as full mode" — but full mode now emits 65 keys, not 59. The test passes because fast mode also skips those six fields, so set(payload.keys()) == EXPECTED_TOP_LEVEL_KEYS holds, but the stated invariant is false. The golden makes this discrepancy undeniable for the first time. Either add the six fields to EXPECTED_TOP_LEVEL_KEYS (and decide whether fast mode should null them or omit them), or reword the test comment to "fast mode emits a stable subset of full-mode keys" — whichever matches the intended contract. Not blocking this PR.
Worth considering (tolerance):
ABS_TOL=REL_TOL=1e-3 is reasonable. On LUFS around -20 dB it gives ~0.02 dB slack; on spectral centroid values in the hundreds it gives ~0.1%. The CI float-parity risk is honestly flagged in the PR description with a documented remediation path.
Test results
Local: 703 tests OK, 1 skip (regen helper, by design). CI: Frontend ✅, Loudness WASM ✅, Backend ⏳ at review time.
Phase boundary check
Clean. Three new files only: apps/backend/tests/test_phase1_golden.py, apps/backend/tests/fixtures/golden/phase1_default.json, docs/history/audio-analyzer-rs-decision-2026-05-20.md. No Phase 2/3 code touched, no Phase 1 production code changed, no schema mutations.
Generated by Claude Code
Backend CI failed on the new gate. analyze.py rounds every output float (1-4 decimals); a different BLAS/thread order on the runner can push a raw value across a rounding boundary, flipping the last digit by a whole step (e.g. 135.4 -> 135.5). The 1e-3 tolerance was far tighter than a 0.1 step, so any one of ~1633 boundary-prone leaves was enough to fail in CI while passing locally (same environment, identical rounding). Widen the numeric comparison to also absorb one rounding step at each field's own decimal precision, plus a 1e-9 float-safety epsilon for clean decimal values at the step boundary. Stays portable across environments while still biting any change larger than a field's reported resolution and every string/enum/key/list-length change. Verified by a cross-environment simulation: +/-1 step on all 2154 numeric leaves -> 0 mismatches; a multi-step change still bites. Adds a meta-test pinning the behavior. https://claude.ai/code/session_018hYXgv4LK9Hm1BNGWgMZf8
The fast-mode test's docstring/message said fast emits "the same top-level key set as full mode", but full mode emits 6 more detail-only fields (keyProfile, tuningFrequency, tuningCents, lufsMomentaryMax, lufsShortTermMax, pitchDetail). The assertion itself (== EXPECTED_TOP_LEVEL_KEYS, the shared contract) is correct; only the wording was misleading. Surfaced by the new Phase 1 golden snapshot in this PR's review. No behavior change. https://claude.ai/code/session_018hYXgv4LK9Hm1BNGWgMZf8
The full-output golden snapshot was not reproducible on CI. The fixture WAV
is bit-identical across machines, but the runner's CPU/BLAS/FFT order yields
last-bit-different floats, and many fine-grained fields (per-beat accent
patterns, per-frame spectral series, MFCC/chroma vectors, onset counts) sit
on discrete decision boundaries that flip wholesale between machines. No
numeric tolerance can absorb that, so the gate failed on CI while passing
locally.
Redesign (per maintainer decision) to two portable assertions:
- contract/structure: exact top-level key set + each field's type category
(catches added/removed fields, type changes, a detector returning null);
- curated core values: environment-stable, high-value measurements (BPM,
key, loudness, true-peak, crest, 7 spectral-balance bands, stereo)
compared with the rounding-step tolerance.
The curated set was chosen with a local fragility probe (tiny PCM noise):
only fields that never flip under perturbation are compared by value. The
new gate was validated against noise up to 2 LSB across 9 runs -> 0
mismatches, while real changes still bite (meta-tests). Golden regenerated
to the new {topLevelKeys, topLevelTypes, coreValues} form. Full backend
suite green (702 ok, 1 skip).
https://claude.ai/code/session_018hYXgv4LK9Hm1BNGWgMZf8
2 participants
Summary
Adds a golden-snapshot regression gate over default-mode
analyze.pyoutput — a safety net for upcominganalyze.py/Essentia changes. It runs the analyzer on a fixed, deterministic procedural fixture and fails when the emitted JSON drifts from a committed golden, naming the exact drifted path. Catches drift in any field (not just hand-curated ones), which is what you want when a change's blast radius is unknown.Test-only change plus one committed fixture JSON. No change to
analyze.py, the JSON schema,src/types.ts, orrequirements.txt.What's here
apps/backend/tests/test_phase1_golden.py— runsanalyze.py(default mode) on an inlined deterministic A-minor fixture and diffs the output against the golden with a self-contained recursive comparator:abs(a-g) <= max(ABS_TOL, REL_TOL*|g|)), NaN/Inf-aware, bool checked before int (soTruevs1bites), key-set drift (added/removed), list-length drift, type mismatch, and aMAX_REPORTEDcap so a large-array drift can't dump thousands of lines.UPDATE_PHASE1_GOLDEN=1 ./venv/bin/python -m unittest tests.test_phase1_golden.apps/backend/tests/fixtures/golden/phase1_default.json— the committed golden (~50 KB, 65 top-level keys), writtensort_keys=True, indent=2.Determinism
Output was confirmed byte-identical across 5 repeated local runs, so the tolerance is purely cross-environment float-jitter insurance (e.g. multi-thread reduction order on the CI runner), not run-to-run noise. The default path has no RNG/ML/timestamps. The one non-deterministic field —
melodyDetail.midiFile, a transient per-run artifact path — is the sole exclusion, stored in the golden as a self-documenting placeholder so the key-set check still holds.Test plan
analyze.pyon the fixture ×5 → byte-identical (determinism proof)./venv/bin/python -m unittest tests.test_phase1_golden→ green (main gate + 10 meta-tests)bpm: 140.4 != 135.4); golden restored byte-identicalmelodyDetail.midiFilepath drift surfaced before it was excluded)python -m unittest discover -s tests→ 703 tests OK (1 skip = the regen helper), no regressionshttps://claude.ai/code/session_018hYXgv4LK9Hm1BNGWgMZf8
Generated by Claude Code