feat(backend): pluggable Phase 2 provider (gemini ↔ MOSS sidecar), default-off - #145
Conversation
…fault-off Add a Phase2Provider abstraction so the producer_summary interpretation can be routed to a self-hosted OpenMOSS MOSS-Audio FastAPI sidecar instead of Gemini, flowing through the IDENTICAL parse/citation/catalogue validators (the "identical recommendation schema" DoD). Default-off (ASA_PHASE2_PROVIDER=gemini); Gemini stays the product default and its path is behavior-preserving (223 server contract tests pass). STEP ONE licence gate (verified from primary sources): MOSS-Audio *weights* are Apache-2.0, but the *modeling code* you must run has no effective licence — the OpenMOSS/MOSS-Audio GitHub repo has no LICENSE file and its pyproject points at a non-existent one (dangling ref). So the sidecar ships NO OpenMOSS code: mock mode for the experiment/eval, real-model path is a 501 stub. Research-only, not promotable; mirrors the MSST backend treatment. - phase2_provider.py: Phase2Provider + MossSidecarProvider + env resolver - server.py: one guarded asymmetric branch in the interpretation request; the Gemini client is still built before the try so its errors still propagate as INTERPRETATION_SETUP_FAILED (most of the diff is re-indentation) - moss_sidecar/: standalone FastAPI sidecar (own venv) + deterministic, schema- valid, Phase-1-grounded mock interpreter - scripts/evaluate_phase2_providers.py + phase2_provider_evaluation.py: citation- accuracy eval reusing the production validators; documented offline-good-enough vs Gemini-wins split (live numbers BLOCKED: no GEMINI_API_KEY, no audio renders, model not runnable under the licence gate) - tests/test_phase2_provider.py: 22 tests incl. a MOSS-branch integration test that executes the modified server.py function end-to-end - docs/PHASE2_PROVIDER.md (STEP ONE findings + design) + CLAUDE.md env vars Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
slittycode
left a comment
There was a problem hiding this comment.
Verdict: COMMENT
Summary
Adds a Phase2Provider protocol and MossSidecarProvider HTTP client that lets _run_interpretation_request_with_profile_config route to a self-hosted MOSS sidecar instead of Gemini, while both paths flow through the identical parse → citation-validate → catalogue-gate → recommendations.v1 tail. Default-off (ASA_PHASE2_PROVIDER=gemini); the Gemini path is byte-for-byte unchanged when no external provider is selected. Frontend and Loudness WASM CI are green; Backend CI was still in progress at review time. The 22 new tests cover the resolver, mock contract, sidecar client, eval scorer, and a live integration branch through the real server function.
Findings
Should fix
stem_summary claims to stay on Gemini but doesn't. Three places in this PR assert it — CLAUDE.md, docs/PHASE2_PROVIDER.md, moss_sidecar/README.md: "stem_summary interpretations stay on Gemini; only the recommendation-emitting producer_summary profile routes to MOSS." The implementation disagrees: resolve_external_phase2_provider() is called unconditionally at the top of _run_interpretation_request_with_profile_config with no guard on profile_id. When ASA_PHASE2_PROVIDER=moss, every stem evaluation for every stem in _run_combined_stem_summary_request is also routed to MOSS. The mock sidecar returns a producer_summary-shaped dict — _is_valid_phase2_shape will reject it for the stem_summary schema, the salvage path kicks in, and you get silently empty stem interpretations. Not a product-path failure (MOSS is default-off) but it is wrong behavior for anyone exercising the experiment. Fix is one line:
# In _run_interpretation_request_with_profile_config, after line 1846:
external_provider = resolve_external_phase2_provider()
if profile_id != "producer_summary": # MOSS is producer_summary-only
external_provider = NoneWorth considering
api_key scoping across two if external_provider is None: blocks (server.py ~L1848, ~L1893). The first block assigns api_key and has three early-return guards. The second block consumes it to construct the Gemini client. At runtime this is always safe (same guard, same condition), but the two-block structure means static analysis will flag api_key as potentially unbound in the second block. The comment "Construct the Gemini client BEFORE the try" explains why the split exists; the scoping awkwardness is worth noting for the next person touching this function.
Test results
Frontend: pass. Loudness WASM: pass. Backend: in progress at review time — no failures visible. The 22 unit/integration tests in test_phase2_provider.py are meaningful (they include a live integration test that executes the modified server function end-to-end with a mocked sidecar), not coverage padding.
Phase boundary check
Clean. Phase 1 (measurement_result) is passed read-only to Phase2ProviderRequest.phase1_result and to build_mock_phase2_result. No new code assigns to or mutates the measurement dict. The mock interpreter reads from phase1 to ground citations but produces an independent return value. The MOSS path flows through the same _validate_phase2_citation_paths + apply_live12_catalogue_gates + build_validated_recommendations tail as Gemini, so invariants #1 and #2 are enforced identically.
Generated by Claude Code
Summary
Adds a
Phase2Providerabstraction so theproducer_summaryinterpretation can be routed to a self-hosted OpenMOSS MOSS-Audio FastAPI sidecar instead of Gemini — fed the same audio + Phase-1 JSON, emitting the identicalPhase2Resultbecause both providers flow through the same parse →_validate_phase2_citation_paths→apply_live12_catalogue_gates→ frozenrecommendations.v1tail. Default-off (ASA_PHASE2_PROVIDER=gemini); Gemini stays the product default and its path is behavior-preserving.STEP ONE — licence gate (verified from primary sources)
The goal made a licence check mandatory and first. Split verdict:
OpenMOSS-Team/MOSS-Audio-*repo islicense: apache-2.0(HF API), ungated, no NonCommercial rider.OpenMOSS/MOSS-Audiorepo has no LICENSE file anywhere and itspyproject.tomldeclareslicense = { file = "LICENSE" }— a dangling reference to a file that doesn't exist. The HF weights repo has nomodeling_*.pyand noAutoModelauto_map, so the model can't run without that unlicensed GitHub code.→ Not promotable to the product path (mirrors the MSST backend, which is even more encumbered). Built as a default-off research experiment: the sidecar ships no OpenMOSS code —
mockmode for the experiment/eval, real-model path is a 501 stub. Full findings indocs/PHASE2_PROVIDER.md+ theasa-moss-audio-licence-gatememory.What's included
phase2_provider.py—Phase2Provider+MossSidecarProvider+ env resolver (ASA_PHASE2_PROVIDER, defaultgemini→None→ native path untouched).server.py— one guarded asymmetric branch in_run_interpretation_request_with_profile_config. The Gemini client is still constructed before thetry, so a construction error still propagates asINTERPRETATION_SETUP_FAILED. Most of the diff is re-indentation (the unchanged Gemini block nested underelse).moss_sidecar/— standalone FastAPI sidecar (own venv, isolated like MSST) + a deterministic, schema-valid, Phase-1-grounded mock interpreter.scripts/evaluate_phase2_providers.py+phase2_provider_evaluation.py— citation-accuracy eval reusing the production validators; documented "offline-good-enough vs Gemini-wins" split.tests/test_phase2_provider.py— 22 tests incl. a MOSS-branch integration test that executes the modifiedserver.pyfunction end-to-end.Honest blockers (the DoD's live numbers)
A live MOSS-vs-Gemini citation eval is blocked 3 ways: no
GEMINI_API_KEY, no audio renders in the fixedrecommendation_tracks/corpus, and MOSS isn't runnable under the licence gate. The harness + scorer + fixed set + split framework ship; the live legs are labelledBLOCKED, not fabricated. The mock's perfect citation score is a contract proof, not a model-quality proxy.Verification
tests/test_phase2_provider.py: 22/22 (incl. MOSS-branch integration + a test that the Gemini default never touches the sidecar).tests/test_server.py+tests/test_recommendations_contract.py: 254/254 (rebased on feat: freeze versioned recommendations.v1 JSON contract for Phase 2 output #143/feat(backend): make MSST separation run for real + ground-truth SDR A/B + licence gate #144) — behavior-preserving for the Gemini path, and MOSS output satisfies the frozenrecommendations.v1contract via the shared tail.analyze.pychange → golden snapshots /EXPECTED_TOP_LEVEL_KEYSunaffected. (Full Essentia-venvunittest discovernot run locally; unrelated to this change — CI covers it.)🤖 Generated with Claude Code