test(ui): cross-boundary citation contract guards Phase 1 parser field drops - #163
Merged
Conversation
…bleton + the rec-proof harness
The ASA -> asa-ableton boundary (and the GOAL.md fixture loop) ran on
hand-extracted interpretation JSON: the result lives embedded in the run
snapshot, the warn-and-keep validationWarnings in a different subtree
(attempt diagnostics), and provenance in a third. asa-ableton's Gate
alpha fixture and every per-fixture phase2.json in the owner plan were
produced by snapshot surgery — an unversioned, fragile handoff.
1. phase2_export.py + GET /api/analysis-runs/{run_id}/export/phase2:
single self-contained phase2-export.v1 envelope — the stored
producer_summary interpretation result verbatim (incl. the frozen
recommendations.v1 projection), the authoritative Phase 1 payload its
citations resolve against (invariant #2 verifiable offline), the full
validationWarnings trail, and provenance. Thin lookup-and-serve,
csv_export.py pattern; 404 RUN_NOT_FOUND / PHASE2_EXPORT_NOT_AVAILABLE.
Derived and read-only — exports stored state, rewrites nothing.
2. recommendation_evaluation.coerce_phase2_payload: --phase2 (and a
fixture-dir phase2.json) now accepts either a bare Phase2Result or
the export envelope, so the downloaded file feeds the scorer as-is.
3. docs/ASA_ABLETON_BOUNDARY.md: the cross-repo contract — file-coupled
not code-coupled, consumer guidance (prefer recommendations.v1
entries; dedupe is a consumer concern — Gate alpha's 60.7%-vs-50%
skip-rate accounting; flagged != invalid), and the v1 freeze policy.
Tests: +15 (10 builder incl. envelope-key freeze, 3 route, 2 unwrap).
tests.test_server 227 OK; adjacent suites 110 OK; full discover matches
the unmodified-tree baseline exactly (env-only matplotlib/torch gaps).
https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…chno 145 + UKG 2-step 132 Owner confirmed genre fit for house/melodic-techno/acid but swapped the other two recommendation-corpus fixtures to genres they actually produce: - techno_rumble_130 -> hard_techno_rumble_145: distorted sine kick (Drum Buss on the kick chain, Boom 45 Hz), rumble reverb return + held sub, 145 BPM. Stays the pilot fixture (smallest spec, 9 device entries). - dnb_reese_174 -> ukg_2step_shuffle_132: 2-step shuffle at 132 BPM, G minor. Swing lives in the committed MIDI (57% 16ths), asserted via grooveDetail.hihatSwing / perDrumSwing.snare / bassDetail.grooveType. Retires the fixture whose proxy fingerprint misread BPM half-time (174->116). Both new fixtures ship spec-only (phase1Fingerprint: null, no fingerprint file) — the distrusted _synthetic proxy fingerprints are retired with the old slugs, and load_fixture tolerates the missing file (citation checks SKIP until the real Ableton render lands). truePeak intents now use dBTP per ADR 0002 (old manifests carried stale linear targets). New MIDI clips bake correct tempo metas and were byte-verified (the retired dnb melody encoded ~116 BPM, not 174). ukg also ships audio_drums.mid so the shuffle answer key is reproducible without hand-programmed groove. Docs updated: NEEDS.md (genre confirmation resolved, inventory + build queue), plans/owner-actions-recommendation-proof-plan.md (pilot + render order), RECOMMENDATION_VERDICT.md (corpus-composition note; old per-fixture proxy numbers not comparable). Verified: evaluate_recommendations.py --fixture <both> --source baseline (catalog-valid, zero issues), --self-test PASS, 34/34 tests.test_recommendation_evaluation. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…G decision) Owner resolved the wire-or-demote choice from NEEDS.md's dead-code finding: the deterministic recommendation engine stays as the scored free baseline in the GOAL.md sub-goal 3 three-source comparison but is NOT to be wired into the product. Score-driven product improvements land on the Phase 2 provider path instead. Recorded in: the module header (abletonDevices.ts), the eval bridge header, NEEDS.md (decision + the now-moot citation-emit candidate), BACKLOG.md, RECOMMENDATION_VERDICT.md, and CLAUDE.md's Backport Candidates line. No code changes — comments and docs only. Verified: npm run lint green, evaluate_recommendations.py --self-test PASS. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…ion script --source claude in evaluate_recommendations.py scores a stored Phase2Result from the Claude CLI provider (sibling phase2.claude.json, same ingestion as the gemini source). scripts/gen_claude_phase2.py produces those files at zero Gemini cost: it runs each fixture's stored Phase 1 fingerprint through the exact server path (server._run_interpretation_request) with ASA_PHASE2_PROVIDER=claude, so output flows the identical parse/citation/ catalogue/recommendations.v1 validation tail as Gemini. Research-only (mirrors the evaluate_* convention); deleting both restores the product exactly. Verified: --self-test PASS; --source claude SKIPs cleanly when no phase2.claude.json exists. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…ixtures Generated via scripts/gen_claude_phase2.py through the exact server path (ASA_PHASE2_PROVIDER=claude, model sonnet, MAX_THINKING_TOKENS=0): both fully cited (custody penalty 1.000), zero validation warnings, 31 recommendations.v1 envelope entries each. Scored with --source claude against the same proxy fingerprints the recorded Gemini numbers used: acid 0.485 (Gemini 0.172), house 0.424 (Gemini 0.343). melodic_techno_arp_124 evidence + the verdict write-up follow. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…rdict Completes the zero-Gemini-cost Claude scoring on the proxy corpus: - melodic_techno_arp_124 phase2.claude.json: 13 rec cards, fully cited, zero warnings, aggregate 0.424 — resolving the "Gemini 0 recs" outlier as Gemini-side (the fixture's fingerprint is interpretable). - RECOMMENDATION_VERDICT.md: dated head-to-head section. Claude (sonnet, text-only) 0.485/0.424/0.424 vs recorded Gemini 0.172/0.343/0.000 on identical fingerprints; mean 0.444 vs 0.172 on the shared subset. Caveats (proxy corpus, modality asymmetry, model class) stated inline. - NEEDS.md sub-goal 3: status updated; source list now includes claude. - docs/PHASE2_PROVIDER.md: 2026-06-11 addendum — scoring result + operating notes (MAX_THINKING_TOKENS=0 for headless calls, model pinning, 300-365s measured latency, session-limit failure mode). - CLAUDE.md: one-line MAX_THINKING_TOKENS note on ASA_CLAUDE_TIMEOUT_SECONDS. - gen_claude_phase2.py: defaults MAX_THINKING_TOKENS=0 and an 1800s timeout (overridable) so the next run doesn't rediscover the thinking-budget trap. Verified: --self-test PASS; full-corpus --source claude scores 3, SKIPs the 2 spec-only fixtures cleanly. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
The Phase 1 contract spans analyze.py output, JSON_SCHEMA.md, and types.ts with no generated source of truth (tripwire #3). Most of parsePhase1Result passes detail blocks through verbatim, but ~12 parseOptional* reconstructors rebuild field-by-field — and a backend field one of them forgets to forward is silently dropped, breaking valid Phase 2 citations to it. This is the exact mechanism that dropped reverbDetail.preDelayMs / perBandRt60.* and the vocalDetail stem proxies. Add tests/services/phase1CitationContract.test.ts: feed a comprehensive payload through the real parsePhase1Result, run the real citation walker (collectPhase1FieldPaths) on the result, and assert every citable path survives. Each field is populated non-null, so a path is present only if the reconstructor carried it through — drop a field in any parseOptional* and its assertion fails by name. Verified teeth: re-injecting the preDelayMs drop fails exactly that one assertion and no other. Also point tripwire #3 at the new guard. Verified: 59 contract assertions green; full tests/services suite 809 green; npm run lint clean. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds an executable cross-boundary contract test that makes the silent-field-drop bug class structurally impossible, closing the audit's structural root-cause finding (the Phase 1 contract spans
analyze.pyoutput,JSON_SCHEMA.md, andtypes.tswith nothing executable defending it — CLAUDE.md tripwire #3).The bug class
Most of
parsePhase1Resultpasses detail blocks through verbatim (isRecord(x) ? x : null), so nested fields survive by construction. But ~12parseOptional*reconstructors inbackendPhase1Client.tsrebuild their block field-by-field — and a backend field one of them forgets to forward is silently dropped. When Phase 2 cites that field, the citation fails the existence check incollectPhase1FieldPathsand the chain of custody breaks on a correct recommendation. This already bitreverbDetail.preDelayMs,reverbDetail.perBandRt60.*,vocalDetail.stemEnergyRatio, andvocalDetail.stemOtherCorrelation— each fixed reactively, with nothing preventing the next one.The guard
tests/services/phase1CitationContract.test.ts:parsePhase1Result,collectPhase1FieldPathson the result,CITABLE_DETAIL_PATHSsurvives.Because each field is non-null, a path is present only if the reconstructor actually carried it through — so dropping a field in any
parseOptional*fails its assertion by name. A meta-guard asserts every block parsed non-null, so the contract can't pass vacuously.Verified teeth: re-injecting the original
preDelayMsdrop fails exactly thereverbDetail.preDelayMsassertion and no other; reverting restores green.Maintenance
When you add a citable field to a
parseOptional*reconstructor (or the Phase 2 prompt invites a new citation), add it to both the fixture andCITABLE_DETAIL_PATHS. Tripwire #3 now points here.Verification
tests/servicessuite: 809 passed (60 files)npm run lintcleanTest-only + one doc pointer; no product code changed.
https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
🤖 Generated with Claude Code
Generated by Claude Code