test: cross-boundary Phase 1 contract gate — kill the silent-field-drop bug class - #161
Merged
Conversation
…op class The Phase 1 contract lives in three hand-maintained reconstructions (analyze.py output, _build_phase1 in server_phase1.py, parsePhase1Result in backendPhase1Client.ts) plus two declarative mirrors (JSON_SCHEMA.md, types/measurement.ts), and nothing executable defended the boundary — the 2026-05-30 full review traced the dropped reverbDetail/vocalDetail citation fields to exactly this gap (CLAUDE.md tripwires #3/#4). This makes the boundary executable end to end: Backend (tests/test_phase1_golden.py): - The golden snapshot now records a recursive structure-only `keyTree` (dicts recurse; lists of objects record the union of element keys; scalars reduce to type categories; nested null<->scalar flips are tolerated as runner noise, container-shape changes always bite). Regenerated golden arms it (+426 lines, purely additive). - New keytree gate compares every nested key the analyzer emits against the committed tree; goldens predating keyTree skip visibly, not silently. - New superset gate pins that _build_phase1 forwards every golden top-level key — the third reconstruction layer, previously unguarded. - KeyTreeMetaTests prove the comparator bites (and doesn't over-bite). Frontend (apps/ui): - tests/fixtures/phase1FullPayload.ts: the inline validPayload promoted to a shared, enriched envelope-shaped fixture covering all 66 golden top-level keys (9 were missing) with realistic nested values. - tests/services/phase1ContractParity.test.ts: five gates driven by the backend golden — A1 every golden key exists in the fixture; A2 reverse (fixture rot on backend deletions); A3 every nested keyTree path exists in the fixture (auto-arms off the golden); B the fixture survives the real parsePhase1Result with every non-null path intact (the silent-drop check); C the canonical run path strips exactly transcriptionDetail and nothing else. Server-side transforms (stereo hoist, spectral *Mean renames, staged strip) are a declared, cited map — an undeclared transform fails the gate. - Fix the live instance the gate caught on first run: parsePhase1Result silently dropped phase1Version (emitted by the server, declared in the type, never forwarded). Gate A3's arming also surfaced stale segmentLoudness/segmentSpectral fixture shapes — corrected to the real analyzer contract. Docs: tripwires #3/#4, JSON_SCHEMA.md, and apps/ui/AGENTS.md now point at the executable gates. Verified: apps/ui `npm run verify` green (756 unit incl. the 6 new gates active with zero skips, 49 smoke); backend test_phase1_golden (27), test_analyze + test_server (374) green in a fresh py3.11 venv; mutation checks observed red-then-green for a parser-side drop (preDelayMs), a fixture-side gap (vocalDetail), and the live phase1Version drop. https://claude.ai/code/session_014eU638tNAHuV3bXDZStuNU
The review job authenticates with the subscription OAuth token (CLAUDE_CODE_OAUTH_TOKEN), which expires roughly yearly; an expired token 401s the whole job and must not block otherwise-green PRs. https://claude.ai/code/session_014eU638tNAHuV3bXDZStuNU
Job-level continue-on-error greens the workflow run but the job's check run still reports failure in the PR checks list (verified on this PR). Step-level keeps the check itself green — genuinely advisory. https://claude.ai/code/session_014eU638tNAHuV3bXDZStuNU
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The Phase 1 contract lives in three hand-maintained reconstructions —
analyze.pyoutput,_build_phase1inserver_phase1.py, andparsePhase1ResultinbackendPhase1Client.ts— plus two declarative mirrors (JSON_SCHEMA.md,types/measurement.ts), and nothing executable defended the boundary. The 2026-05-30 full review named this the structural root cause of the droppedreverbDetail.perBandRt60/preDelayMsandvocalDetail.stemEnergyRatio/stemOtherCorrelationcitation fields ("no generated source of truth and no cross-app test"); CLAUDE.md tripwires #3/#4 exist because this keeps biting. This PR makes the bug class fail loudly in CI instead of relying on the tripwire being remembered.What
Backend — golden snapshot becomes the generated cross-app source of truth (
apps/backend/tests/test_phase1_golden.py):build_goldennow records a recursive, structure-onlykeyTree: dicts recurse, lists of objects record the union of element keys, scalars reduce to type categories. Comparison tolerates the known runner-noise axes (nested null↔scalar flips, empty↔populated lists) and bites on every container-shape change (key added/removed, dict↔scalar, objects↔scalars).KEYTREE_PRUNE_PATHS(seeded empty) is the documented one-line loosening path if a flaky subtree ever surfaces.test_phase1_keytree_matches_goldengate; goldens predatingkeyTreeskip visibly until the nextUPDATE_PHASE1_GOLDEN=1re-baseline arms it. The golden here is regenerated, so the gate ships armed (+426 purely additive lines).BuildPhase1KeySupersetTests: pins that_build_phase1forwards every golden top-level key — the third reconstruction layer, previously unguarded (a key missed there never reaches the frontend at all).KeyTreeMetaTestsprove the comparator bites and doesn't over-bite.Frontend — the parity gate (
apps/ui):tests/fixtures/phase1FullPayload.ts: the inlinevalidPayloadpromoted to a shared, envelope-shaped fixture, enriched to cover all 66 golden top-level keys (9 were missing:phase1Version,lufsCurve,spectralBalanceTimeSeries,saturationDetail,snareDetail,hihatDetail,transientDensityDetail,pitchDetail,stemAnalysis) with realistic nested values.tests/services/phase1ContractParity.test.ts— five gates driven by the committed backend golden:keyTreepath exists in the fixture. Auto-arms off the golden; active in this PR.parsePhase1Resultwith every non-null input path intact — the silent-drop check itself.transcriptionDetailand nothing else.*Meanrenames, staged-path strip) are a small declared map withserver_phase1.py/analysis_runtime.pycitations — an undeclared future transform fails the gate rather than passing silently.Two real catches before merge (the gate proving itself):
parsePhase1Resultsilently droppedphase1Version— emitted by the server, declared in the type, never forwarded. Fixed (one line).segmentLoudness/segmentSpectralfixture shapes ({start, value}vs the real{segmentIndex, start, end, lufs, lra}contract) — corrected.Docs: tripwires #3/#4,
JSON_SCHEMA.md, andapps/ui/AGENTS.mdnow point at the executable gates.Verification
apps/ui:npm run verifygreen — lint, 756 unit tests (all six new gates active, zero skips), build, 49 smoke tests.apps/backend(fresh py3.11 venv):tests.test_phase1_golden27 tests OK with the keytree gate active;tests.test_analyze+tests.test_server374 tests OK.reverbDetail.preDelayMsforwarding removed → Gate B names exactly that path), fixture-side gap (vocalDetailremoved → Gates A1+A2 name it), and the livephase1Versiondrop (Gate B red on first run, green after the fix).https://claude.ai/code/session_014eU638tNAHuV3bXDZStuNU
Generated by Claude Code