feat(eval): claude provider source for the recommendation harness (zero-Gemini-cost comparison) - #160
Merged
Merged
Conversation
…bleton + the rec-proof harness
The ASA -> asa-ableton boundary (and the GOAL.md fixture loop) ran on
hand-extracted interpretation JSON: the result lives embedded in the run
snapshot, the warn-and-keep validationWarnings in a different subtree
(attempt diagnostics), and provenance in a third. asa-ableton's Gate
alpha fixture and every per-fixture phase2.json in the owner plan were
produced by snapshot surgery — an unversioned, fragile handoff.
1. phase2_export.py + GET /api/analysis-runs/{run_id}/export/phase2:
single self-contained phase2-export.v1 envelope — the stored
producer_summary interpretation result verbatim (incl. the frozen
recommendations.v1 projection), the authoritative Phase 1 payload its
citations resolve against (invariant #2 verifiable offline), the full
validationWarnings trail, and provenance. Thin lookup-and-serve,
csv_export.py pattern; 404 RUN_NOT_FOUND / PHASE2_EXPORT_NOT_AVAILABLE.
Derived and read-only — exports stored state, rewrites nothing.
2. recommendation_evaluation.coerce_phase2_payload: --phase2 (and a
fixture-dir phase2.json) now accepts either a bare Phase2Result or
the export envelope, so the downloaded file feeds the scorer as-is.
3. docs/ASA_ABLETON_BOUNDARY.md: the cross-repo contract — file-coupled
not code-coupled, consumer guidance (prefer recommendations.v1
entries; dedupe is a consumer concern — Gate alpha's 60.7%-vs-50%
skip-rate accounting; flagged != invalid), and the v1 freeze policy.
Tests: +15 (10 builder incl. envelope-key freeze, 3 route, 2 unwrap).
tests.test_server 227 OK; adjacent suites 110 OK; full discover matches
the unmodified-tree baseline exactly (env-only matplotlib/torch gaps).
https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…chno 145 + UKG 2-step 132 Owner confirmed genre fit for house/melodic-techno/acid but swapped the other two recommendation-corpus fixtures to genres they actually produce: - techno_rumble_130 -> hard_techno_rumble_145: distorted sine kick (Drum Buss on the kick chain, Boom 45 Hz), rumble reverb return + held sub, 145 BPM. Stays the pilot fixture (smallest spec, 9 device entries). - dnb_reese_174 -> ukg_2step_shuffle_132: 2-step shuffle at 132 BPM, G minor. Swing lives in the committed MIDI (57% 16ths), asserted via grooveDetail.hihatSwing / perDrumSwing.snare / bassDetail.grooveType. Retires the fixture whose proxy fingerprint misread BPM half-time (174->116). Both new fixtures ship spec-only (phase1Fingerprint: null, no fingerprint file) — the distrusted _synthetic proxy fingerprints are retired with the old slugs, and load_fixture tolerates the missing file (citation checks SKIP until the real Ableton render lands). truePeak intents now use dBTP per ADR 0002 (old manifests carried stale linear targets). New MIDI clips bake correct tempo metas and were byte-verified (the retired dnb melody encoded ~116 BPM, not 174). ukg also ships audio_drums.mid so the shuffle answer key is reproducible without hand-programmed groove. Docs updated: NEEDS.md (genre confirmation resolved, inventory + build queue), plans/owner-actions-recommendation-proof-plan.md (pilot + render order), RECOMMENDATION_VERDICT.md (corpus-composition note; old per-fixture proxy numbers not comparable). Verified: evaluate_recommendations.py --fixture <both> --source baseline (catalog-valid, zero issues), --self-test PASS, 34/34 tests.test_recommendation_evaluation. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…G decision) Owner resolved the wire-or-demote choice from NEEDS.md's dead-code finding: the deterministic recommendation engine stays as the scored free baseline in the GOAL.md sub-goal 3 three-source comparison but is NOT to be wired into the product. Score-driven product improvements land on the Phase 2 provider path instead. Recorded in: the module header (abletonDevices.ts), the eval bridge header, NEEDS.md (decision + the now-moot citation-emit candidate), BACKLOG.md, RECOMMENDATION_VERDICT.md, and CLAUDE.md's Backport Candidates line. No code changes — comments and docs only. Verified: npm run lint green, evaluate_recommendations.py --self-test PASS. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…ion script --source claude in evaluate_recommendations.py scores a stored Phase2Result from the Claude CLI provider (sibling phase2.claude.json, same ingestion as the gemini source). scripts/gen_claude_phase2.py produces those files at zero Gemini cost: it runs each fixture's stored Phase 1 fingerprint through the exact server path (server._run_interpretation_request) with ASA_PHASE2_PROVIDER=claude, so output flows the identical parse/citation/ catalogue/recommendations.v1 validation tail as Gemini. Research-only (mirrors the evaluate_* convention); deleting both restores the product exactly. Verified: --self-test PASS; --source claude SKIPs cleanly when no phase2.claude.json exists. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…ixtures Generated via scripts/gen_claude_phase2.py through the exact server path (ASA_PHASE2_PROVIDER=claude, model sonnet, MAX_THINKING_TOKENS=0): both fully cited (custody penalty 1.000), zero validation warnings, 31 recommendations.v1 envelope entries each. Scored with --source claude against the same proxy fingerprints the recorded Gemini numbers used: acid 0.485 (Gemini 0.172), house 0.424 (Gemini 0.343). melodic_techno_arp_124 evidence + the verdict write-up follow. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
…rdict Completes the zero-Gemini-cost Claude scoring on the proxy corpus: - melodic_techno_arp_124 phase2.claude.json: 13 rec cards, fully cited, zero warnings, aggregate 0.424 — resolving the "Gemini 0 recs" outlier as Gemini-side (the fixture's fingerprint is interpretable). - RECOMMENDATION_VERDICT.md: dated head-to-head section. Claude (sonnet, text-only) 0.485/0.424/0.424 vs recorded Gemini 0.172/0.343/0.000 on identical fingerprints; mean 0.444 vs 0.172 on the shared subset. Caveats (proxy corpus, modality asymmetry, model class) stated inline. - NEEDS.md sub-goal 3: status updated; source list now includes claude. - docs/PHASE2_PROVIDER.md: 2026-06-11 addendum — scoring result + operating notes (MAX_THINKING_TOKENS=0 for headless calls, model pinning, 300-365s measured latency, session-limit failure mode). - CLAUDE.md: one-line MAX_THINKING_TOKENS note on ASA_CLAUDE_TIMEOUT_SECONDS. - gen_claude_phase2.py: defaults MAX_THINKING_TOKENS=0 and an 1800s timeout (overridable) so the next run doesn't rediscover the thinking-budget trap. Verified: --self-test PASS; full-corpus --source claude scores 3, SKIPs the 2 spec-only fixtures cleanly. https://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the
clauderecommendation source to the recommendation-proof harness (GOAL.md sub-goal 3), enabling a zero-Gemini-cost provider comparison on the existing proxy-fingerprint corpus:evaluate_recommendations.py:--source claudescores a storedPhase2Resultfrom the Claude CLI provider — sibling filephase2.claude.json, same ingestion path as thegeminisource.scripts/gen_claude_phase2.py(new, research-only): produces those files by running each fixture's stored Phase 1 fingerprint through the exact server path (server._run_interpretation_requestwithASA_PHASE2_PROVIDER=claude), so the output carries the full citation/catalogue/recommendations.v1validation tail — identical handling to Gemini output. Text-only by design: the provider grounds on the embedded Phase 1 JSON and never needs rendered audio, which is what makes the pre-render comparison possible.This directly advances agent-work items 3–5 of
plans/owner-actions-recommendation-proof-plan.md("Generate Claude recommendations … at no Gemini cost", "Score Claude, deterministic, and baseline outputs using the same scorer").Status
Generation runs on the three proxy-fingerprint fixtures (
acid_303_128,house_sidechain_pluck_124,melodic_techno_arp_124) are in progress; the generatedphase2.claude.jsonevidence files plus the verdict/NEEDS updates will land as follow-up commits on this PR.Verification
evaluate_recommendations.py --self-testPASS (good 1.000 / bad 0.000)--source claudeSKIPs cleanly when nophase2.claude.jsonexistsevaluate_*convention — deleting both files restores the product exactlyhttps://claude.ai/code/session_01YB5QFBLL4nbqPSdtcnxrLy
Generated by Claude Code