From 7e9a2669e3224063083bdd30f479430666e21760 Mon Sep 17 00:00:00 2001 From: slitty-codes Date: Fri, 5 Jun 2026 18:53:12 +1200 Subject: [PATCH] =?UTF-8?q?feat(backend):=20pluggable=20Phase=202=20provid?= =?UTF-8?q?er=20(gemini=20=E2=86=94=20MOSS=20sidecar),=20default-off?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add a Phase2Provider abstraction so the producer_summary interpretation can be routed to a self-hosted OpenMOSS MOSS-Audio FastAPI sidecar instead of Gemini, flowing through the IDENTICAL parse/citation/catalogue validators (the "identical recommendation schema" DoD). Default-off (ASA_PHASE2_PROVIDER=gemini); Gemini stays the product default and its path is behavior-preserving (223 server contract tests pass). STEP ONE licence gate (verified from primary sources): MOSS-Audio *weights* are Apache-2.0, but the *modeling code* you must run has no effective licence — the OpenMOSS/MOSS-Audio GitHub repo has no LICENSE file and its pyproject points at a non-existent one (dangling ref). So the sidecar ships NO OpenMOSS code: mock mode for the experiment/eval, real-model path is a 501 stub. Research-only, not promotable; mirrors the MSST backend treatment. - phase2_provider.py: Phase2Provider + MossSidecarProvider + env resolver - server.py: one guarded asymmetric branch in the interpretation request; the Gemini client is still built before the try so its errors still propagate as INTERPRETATION_SETUP_FAILED (most of the diff is re-indentation) - moss_sidecar/: standalone FastAPI sidecar (own venv) + deterministic, schema- valid, Phase-1-grounded mock interpreter - scripts/evaluate_phase2_providers.py + phase2_provider_evaluation.py: citation- accuracy eval reusing the production validators; documented offline-good-enough vs Gemini-wins split (live numbers BLOCKED: no GEMINI_API_KEY, no audio renders, model not runnable under the licence gate) - tests/test_phase2_provider.py: 22 tests incl. a MOSS-branch integration test that executes the modified server.py function end-to-end - docs/PHASE2_PROVIDER.md (STEP ONE findings + design) + CLAUDE.md env vars Co-Authored-By: Claude Opus 4.8 (1M context) --- CLAUDE.md | 5 + apps/backend/moss_sidecar/README.md | 79 +++++ apps/backend/moss_sidecar/__init__.py | 5 + apps/backend/moss_sidecar/app.py | 133 ++++++++ apps/backend/moss_sidecar/mock_interpreter.py | 294 +++++++++++++++++ apps/backend/moss_sidecar/requirements.txt | 22 ++ apps/backend/phase2_provider.py | 254 +++++++++++++++ apps/backend/phase2_provider_evaluation.py | 278 ++++++++++++++++ .../scripts/evaluate_phase2_providers.py | 173 ++++++++++ apps/backend/server.py | 241 +++++++++----- apps/backend/tests/test_phase2_provider.py | 303 ++++++++++++++++++ docs/PHASE2_PROVIDER.md | 179 +++++++++++ 12 files changed, 1880 insertions(+), 86 deletions(-) create mode 100644 apps/backend/moss_sidecar/README.md create mode 100644 apps/backend/moss_sidecar/__init__.py create mode 100644 apps/backend/moss_sidecar/app.py create mode 100644 apps/backend/moss_sidecar/mock_interpreter.py create mode 100644 apps/backend/moss_sidecar/requirements.txt create mode 100644 apps/backend/phase2_provider.py create mode 100644 apps/backend/phase2_provider_evaluation.py create mode 100644 apps/backend/scripts/evaluate_phase2_providers.py create mode 100644 apps/backend/tests/test_phase2_provider.py create mode 100644 docs/PHASE2_PROVIDER.md diff --git a/CLAUDE.md b/CLAUDE.md index f7f4b1c5..fc4cbffa 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -365,6 +365,11 @@ ASA_MSST_ROOT="" # path to the MSST-WebUI checkout (adde ASA_MSST_MODEL="scnet_4stem" # MSST model registry id in separation_backend._MSST_MODEL_REGISTRY. Default `scnet_4stem` (4-stem parity). `bs_roformer_vocals` is research/A-B-only (2-stem, leaves bass/drums empty). Unknown ids fall back to the default. ASA_MSST_MODEL_DIR="" # optional config/checkpoint root (MSST configs/ + pretrain/ layout); defaults to ASA_MSST_ROOT. ASA_MSST_DEVICE="" # optional device for MSST inference (auto|cpu|cuda|mps); defaults to auto. +ASA_PHASE2_PROVIDER="gemini" # Phase 2 interpretation provider: `gemini` (default, product path, unchanged) or `moss` to route the producer_summary interpretation to a self-hosted OpenMOSS MOSS-Audio FastAPI sidecar. Default-off experiment; both providers flow through the identical parse/citation/catalogue validators. STEP ONE licence gate: MOSS-Audio *weights* are Apache-2.0 but the *modeling code* has no effective licence (the GitHub repo's pyproject points at a non-existent LICENSE), so the sidecar ships no OpenMOSS code and the real-model path is a 501 stub — research-only, not promotable. See apps/backend/phase2_provider.py + docs/PHASE2_PROVIDER.md. +ASA_MOSS_SIDECAR_URL="http://127.0.0.1:8200" # base URL of the MOSS sidecar (apps/backend/moss_sidecar/app.py) when ASA_PHASE2_PROVIDER=moss. +ASA_MOSS_SIDECAR_TIMEOUT_SECONDS="180" # per-request timeout for the MOSS sidecar call. +ASA_MOSS_MODEL_ID="" # optional MOSS model id forwarded to the sidecar; defaults to the run's model_name. +ASA_MOSS_SIDECAR_MODE="mock" # sidecar-side (its own venv): `mock` (default, deterministic schema-valid Phase-1-grounded output) or `model` (a 501 licence-gated stub that executes no OpenMOSS code). ``` Phase 2 is gated by `VITE_ENABLE_PHASE2_GEMINI`. `GEMINI_API_KEY` is backend-only. `SONIC_ANALYZER_ADMIN_KEY` is backend-only and never exposed to clients. diff --git a/apps/backend/moss_sidecar/README.md b/apps/backend/moss_sidecar/README.md new file mode 100644 index 00000000..18e1f3ac --- /dev/null +++ b/apps/backend/moss_sidecar/README.md @@ -0,0 +1,79 @@ +# MOSS-Audio Phase 2 Sidecar + +A standalone FastAPI service that emits the **identical `Phase2Result` schema** as +the Gemini Phase 2 path, fed the same inputs (built prompt + audio + authoritative +Phase 1 JSON + `response_schema`). The ASA backend reaches it via +[`phase2_provider.py`](../phase2_provider.py) when `ASA_PHASE2_PROVIDER=moss`. + +**Default-off research experiment.** Gemini stays the product default. This sidecar +runs in its **own venv**, isolated from the product venv (mirrors the MSST +separation runner) so its dependencies never touch `analyze.py`'s contract. + +## ⚠️ Licence status (STEP ONE gate) + +- **Weights** (`OpenMOSS-Team/MOSS-Audio-*` on HuggingFace): **Apache-2.0** — clean, + permits self-host + derivative + commercial use. +- **Modeling code** (the forward pass you must run): **no effective licence.** The + `OpenMOSS/MOSS-Audio` GitHub repo has no `LICENSE` file; its `pyproject.toml` + declares `license = { file = "LICENSE" }` — a dangling reference to a file that + does not exist. The HF weights repo has no `modeling_*.py` and its `auto_map` has + no `AutoModel` entry, so you cannot run the model without the GitHub `src/` code. + +Consequence: **this sidecar ships no OpenMOSS code.** The real-model path +(`ASA_MOSS_SIDECAR_MODE=model`) is a 501 stub. Only the deterministic `mock` mode +runs. See [`docs/PHASE2_PROVIDER.md`](../../../docs/PHASE2_PROVIDER.md) and the +`asa-moss-audio-licence-gate` memory. Promotion is unblocked only when OpenMOSS +publishes the missing LICENSE (or `moss_audio` is upstreamed into Apache-2.0 +`transformers`), or a cleanly-licensed runtime can serve the Apache-2.0 weights for +audio-in inference. + +## Run (mock mode) + +```bash +# From apps/backend (own venv — NOT the product venv): +python3.11 -m venv moss_sidecar/.venv +moss_sidecar/.venv/bin/pip install -r moss_sidecar/requirements.txt +moss_sidecar/.venv/bin/python -m moss_sidecar.app # serves 127.0.0.1:8200 +``` + +Then point the backend at it: + +```bash +ASA_PHASE2_PROVIDER=moss ASA_MOSS_SIDECAR_URL=http://127.0.0.1:8200 \ + # producer_summary interpretations now route to MOSS +``` + +`stem_summary` interpretations stay on Gemini; only the recommendation-emitting +`producer_summary` profile routes to MOSS. + +## Contract + +`POST /v1/phase2` + +```jsonc +{ + "prompt": "", + "response_schema": { /* the Phase2Result schema server.py would hand Gemini */ }, + "phase1": { /* authoritative measurement JSON, for citation grounding */ }, + "model": "moss-audio-4b-instruct", + "audio": { "filename": "x.flac", "mime_type": "audio/flac", "base64": "..." } // or null +} +``` + +→ `{ "result": | null, "provider": "moss-mock" | "moss", "warnings": [] }` + +`GET /healthz` → `{ "status": "ok", "mode": "mock", "provider": "moss-mock" }` + +The backend re-serialises `result` and runs it through the **same** parse + +`_validate_phase2_citation_paths` + `apply_live12_catalogue_gates` path as Gemini, +so the schema and the chain-of-custody contract are enforced identically. + +## Eval + +[`scripts/evaluate_phase2_providers.py`](../scripts/evaluate_phase2_providers.py) +scores citation accuracy across providers on the fixed +`tests/fixtures/recommendation_tracks/` corpus. The mock's citation accuracy is +~1.0 **by construction** (it cites only resolving paths) — it proves the pipeline +and scorer, not the real model's quality. Live MOSS-vs-Gemini numbers are blocked +(no Gemini key locally, no audio renders in the corpus, model not runnable under +the licence gate); see the eval's `--help` and `docs/PHASE2_PROVIDER.md`. diff --git a/apps/backend/moss_sidecar/__init__.py b/apps/backend/moss_sidecar/__init__.py new file mode 100644 index 00000000..dd6fa9c8 --- /dev/null +++ b/apps/backend/moss_sidecar/__init__.py @@ -0,0 +1,5 @@ +"""ASA MOSS-Audio Phase 2 sidecar package. + +Default-off research experiment. Runs in its OWN venv, isolated from the product +venv (mirrors the MSST separation runner). See ``app.py`` and ``README.md``. +""" diff --git a/apps/backend/moss_sidecar/app.py b/apps/backend/moss_sidecar/app.py new file mode 100644 index 00000000..4ad80db3 --- /dev/null +++ b/apps/backend/moss_sidecar/app.py @@ -0,0 +1,133 @@ +"""MOSS-Audio Phase 2 sidecar — a standalone FastAPI service. + +Fed the same inputs as Gemini (built prompt + audio + authoritative Phase 1 JSON ++ ``response_schema``), it returns a ``Phase2Result`` that the ASA backend +validates through the *identical* parse / citation / catalogue path as Gemini +(see ``apps/backend/phase2_provider.py``). Run it on its **own venv** (FastAPI + +uvicorn — and, only if/when licensing is resolved, the model stack), isolated +from the product venv exactly like the MSST separation runner. + +Modes (env ``ASA_MOSS_SIDECAR_MODE``): + - ``mock`` (default): deterministic, schema-valid, Phase-1-grounded output via + ``mock_interpreter``. No model, no GPU. This is what the experiment + eval use. + - ``model``: the real MOSS-Audio path — a **gated stub**. STEP ONE found the + MOSS-Audio *weights* are Apache-2.0 but the *modeling code* you must run has + no effective licence (the GitHub repo's ``pyproject`` points at a ``LICENSE`` + file that does not exist). So this service ships **no** OpenMOSS code and the + ``model`` path raises a 501 explaining what would unblock it. See + ``docs/PHASE2_PROVIDER.md`` and the README in this directory. + +Run: + uvicorn moss_sidecar.app:app --host 127.0.0.1 --port 8200 # from apps/backend + # or: python -m moss_sidecar.app +""" + +from __future__ import annotations + +import os +from typing import Any + +try: # allow both `uvicorn moss_sidecar.app:app` and `python -m moss_sidecar.app` + from .mock_interpreter import build_mock_phase2_result +except ImportError: # pragma: no cover - direct-script fallback + from mock_interpreter import build_mock_phase2_result # type: ignore + +from fastapi import FastAPI, HTTPException +from pydantic import BaseModel, Field + + +SIDECAR_MODE = os.getenv("ASA_MOSS_SIDECAR_MODE", "mock").strip().lower() + +app = FastAPI(title="ASA MOSS-Audio Phase 2 Sidecar", version="0.1.0") + + +class AudioPayload(BaseModel): + filename: str + mime_type: str + base64: str + + +class Phase2SidecarRequest(BaseModel): + prompt: str + response_schema: dict[str, Any] = Field(default_factory=dict) + phase1: dict[str, Any] = Field(default_factory=dict) + model: str | None = None + request_id: str | None = None + audio: AudioPayload | None = None + + +class Phase2SidecarResponse(BaseModel): + result: dict[str, Any] | None + provider: str + warnings: list[str] = Field(default_factory=list) + + +def _run_model_interpreter(request: Phase2SidecarRequest) -> dict[str, Any]: + """Real MOSS-Audio inference — intentionally a licence-gated stub. + + This deliberately imports and executes **no** OpenMOSS code. Two independent + conditions must hold before a real implementation may live here: + + 1. The MOSS-Audio inference/modeling code carries an effective licence + permitting self-host + derivative use (today it does not — the GitHub + repo has no ``LICENSE`` file; only the *weights* are Apache-2.0), OR a + cleanly-licensed runtime (e.g. an Apache/MIT reimplementation) can serve + the Apache-2.0 weights for audio-in inference. + 2. The operator opts in explicitly via ``ASA_MOSS_ALLOW_UNLICENSED_MODEL=1`` + and supplies a runner path — and accepts the licence risk. + + Until then this raises 501 so the failure is loud and honest rather than a + silent fabrication. + """ + if os.getenv("ASA_MOSS_ALLOW_UNLICENSED_MODEL", "").strip() != "1": + raise HTTPException( + status_code=501, + detail=( + "MOSS-Audio model mode is not wired: the modeling code has no " + "effective licence (STEP ONE gate). Use ASA_MOSS_SIDECAR_MODE=mock, " + "or resolve the licence and supply a cleanly-licensed runtime. See " + "docs/PHASE2_PROVIDER.md." + ), + ) + raise HTTPException( + status_code=501, + detail=( + "Real MOSS-Audio inference is not implemented in this sidecar by " + "design — it would execute currently-unlicensed OpenMOSS modeling " + "code. Provide a cleanly-licensed runtime and implement " + "_run_model_interpreter against it." + ), + ) + + +@app.get("/healthz") +def healthz() -> dict[str, Any]: + return {"status": "ok", "mode": SIDECAR_MODE, "provider": _provider_label()} + + +def _provider_label() -> str: + return "moss-mock" if SIDECAR_MODE == "mock" else "moss" + + +@app.post("/v1/phase2", response_model=Phase2SidecarResponse) +def interpret(request: Phase2SidecarRequest) -> Phase2SidecarResponse: + if SIDECAR_MODE == "mock": + result = build_mock_phase2_result(request.phase1, prompt=request.prompt) + return Phase2SidecarResponse(result=result, provider="moss-mock", warnings=[]) + if SIDECAR_MODE == "model": + result = _run_model_interpreter(request) + return Phase2SidecarResponse(result=result, provider="moss", warnings=[]) + raise HTTPException( + status_code=500, + detail=f"Unknown ASA_MOSS_SIDECAR_MODE '{SIDECAR_MODE}' (expected 'mock' or 'model').", + ) + + +if __name__ == "__main__": # pragma: no cover + import uvicorn + + uvicorn.run( + app, + host=os.getenv("ASA_MOSS_SIDECAR_HOST", "127.0.0.1"), + port=int(os.getenv("ASA_MOSS_SIDECAR_PORT", "8200")), + ) diff --git a/apps/backend/moss_sidecar/mock_interpreter.py b/apps/backend/moss_sidecar/mock_interpreter.py new file mode 100644 index 00000000..ebc50382 --- /dev/null +++ b/apps/backend/moss_sidecar/mock_interpreter.py @@ -0,0 +1,294 @@ +"""Deterministic mock Phase 2 interpreter for the MOSS sidecar. + +Pure stdlib. Produces a **full, schema-valid ``Phase2Result``** grounded in the +provided Phase 1 measurement JSON, citing only Phase 1 field paths that actually +resolve. This exists so the provider abstraction, the FastAPI sidecar, the unit +tests, and the citation-accuracy eval can all run end-to-end **with no model and +no GPU** — and so the schema/citation contract is exercised on every commit. + +IMPORTANT — what this is and is NOT: + - It IS a contract fixture: every required ``Phase2Result`` field is populated + with shape-valid content, and every citation resolves against the supplied + Phase 1 payload (so its citation accuracy is ~1.0 *by construction*). + - It is NOT a quality proxy for the real MOSS-Audio model. The mock's + citation accuracy says nothing about whether MOSS would cite correctly; it + only proves the pipeline + scorer work. The real model path is licence-gated + (see ``docs/PHASE2_PROVIDER.md``) and deliberately not wired. + +Determinism: no randomness, no clock. Same Phase 1 in → same result out. +""" + +from __future__ import annotations + +from typing import Any + + +# Candidate top-level Phase 1 scalars to cite, in priority order. Only those +# present in the supplied payload are emitted, so citations always resolve. +_CITATION_CANDIDATES: tuple[str, ...] = ( + "bpm", + "lufsIntegrated", + "key", + "lufsRange", + "truePeak", + "crestFactor", + "timeSignature", + "durationSeconds", + "sampleRate", + "danceability", +) + + +def _present(phase1: dict[str, Any], path: str) -> bool: + """Does a dotted path resolve in the Phase 1 payload? + + Mirrors the descent semantics of + ``server_phase2._collect_measurement_field_paths`` closely enough that any + path this returns True for will also be accepted by + ``_validate_phase2_citation_paths``: descend dict keys; for a list, treat the + first dict element as the representative (array-item fields register under + ``prefix.key``). + """ + parts = path.split(".") + node: Any = phase1 + for part in parts: + if isinstance(node, dict) and part in node: + node = node[part] + elif isinstance(node, list) and node and isinstance(node[0], dict) and part in node[0]: + node = node[0][part] + else: + return False + return True + + +def _citations(phase1: dict[str, Any], limit: int) -> list[str]: + """Up to ``limit`` resolving citation paths, grounded in this Phase 1. + + Falls back to ``["bpm"]`` only if literally nothing resolves (defensive; a + real Phase 1 always has ``bpm``). A non-empty list keeps the recommendation + from tripping the frontend's "missing phase1Fields" check. + """ + found = [path for path in _CITATION_CANDIDATES if _present(phase1, path)] + # Add a couple of nested citations when the obvious sub-objects exist, to + # exercise dotted-path resolution (not just top-level scalars). + for parent, child in (("spectralBalance", None), ("kickDetail", None)): + node = phase1.get(parent) + if isinstance(node, dict): + for child_key, child_val in node.items(): + if isinstance(child_val, (int, float, str)): + found.append(f"{parent}.{child_key}") + break + deduped = list(dict.fromkeys(found)) + return (deduped or ["bpm"])[:limit] + + +def _num(phase1: dict[str, Any], key: str, default: float) -> float: + value = phase1.get(key) + if isinstance(value, bool) or not isinstance(value, (int, float)): + return default + return float(value) + + +def _str(phase1: dict[str, Any], key: str, default: str) -> str: + value = phase1.get(key) + return value if isinstance(value, str) and value.strip() else default + + +def build_mock_phase2_result( + phase1: dict[str, Any], + *, + prompt: str | None = None, + model_label: str = "moss-mock", +) -> dict[str, Any]: + """Build a full, schema-valid ``Phase2Result`` grounded in ``phase1``. + + Satisfies every predicate in ``server_phase2._is_valid_phase2_shape`` and + emits resolving ``phase1Fields`` citations on every cited record. + """ + bpm = _num(phase1, "bpm", 120.0) + key = _str(phase1, "key", "C Minor") + time_signature = _str(phase1, "timeSignature", "4/4") + sample_rate = int(_num(phase1, "sampleRate", 48000)) + duration = _num(phase1, "durationSeconds", 16.0) + lufs = phase1.get("lufsIntegrated") + lufs_text = f"{lufs:.1f} LUFS" if isinstance(lufs, (int, float)) else "the measured loudness" + + cites_loudness = _citations(phase1, 2) + cites_tempo = _citations(phase1, 1) + cites_tonal = _citations(phase1, 2) + + return { + "trackCharacter": ( + f"A {bpm:.0f} BPM piece in {key} ({time_signature}). This is a " + f"deterministic mock interpretation grounded in Phase 1 ({model_label})." + ), + "projectSetup": { + "tempoBpm": bpm, + "timeSignature": time_signature, + "sampleRate": sample_rate, + "bitDepth": 24, + "headroomTarget": "-6 dBFS on the master before limiting", + "sessionGoal": f"Rebuild the track's character at {bpm:.0f} BPM in {key}.", + }, + "trackLayout": [ + { + "order": 1, + "name": "Drums", + "type": "Drum Rack", + "purpose": "Foundational groove and transient energy.", + "grounding": {"phase1Fields": cites_tempo}, + }, + { + "order": 2, + "name": "Bass", + "type": "Instrument", + "purpose": "Low-end anchor tuned to the detected key.", + "grounding": {"phase1Fields": cites_tonal}, + }, + ], + "routingBlueprint": { + "sidechainSource": "Kick", + "sidechainTargets": ["Bass", "Pads"], + "returns": [ + { + "name": "A - Reverb", + "purpose": "Shared space for melodic elements.", + "sendSources": ["Bass", "Lead"], + "deviceFocus": "Reverb", + "levelGuidance": "Start at -18 dB return level.", + } + ], + "notes": ["Mock routing blueprint grounded in Phase 1 measurements."], + }, + "warpGuide": { + "fullTrack": { + "warpMode": "Complex Pro", + "settings": "Formants 100, Envelope 128", + "reason": "Full-mix material warps cleanest under Complex Pro.", + }, + "drums": { + "warpMode": "Beats", + "settings": "Transient Loop Mode: Loop Off", + "reason": "Percussive transients preserved with Beats mode.", + }, + "bass": { + "warpMode": "Complex", + "reason": "Sustained low end stays stable under Complex.", + }, + "melodic": { + "warpMode": "Tones", + "reason": "Monophonic-leaning melodic content suits Tones.", + }, + "rationale": f"Warp choices follow the {bpm:.0f} BPM grid measured in Phase 1.", + }, + "detectedCharacteristics": [ + { + "name": "Tempo lock", + "confidence": "HIGH", + "explanation": f"Phase 1 measured {bpm:.1f} BPM.", + }, + { + "name": "Tonal center", + "confidence": "MED", + "explanation": f"Phase 1 estimated the key as {key}.", + }, + ], + "arrangementOverview": { + "summary": "A single-loop mock arrangement spanning the measured duration.", + "segments": [ + { + "index": 0, + "startTime": 0.0, + "endTime": round(duration, 3), + "description": "Full loop.", + "sceneName": "Loop", + "abletonAction": "Duplicate to a 1-bar clip and loop.", + "automationFocus": "Filter cutoff sweep over the loop.", + } + ], + }, + "sonicElements": { + "kick": "Punchy sine-based kick with a short decay.", + "bass": f"Sub-forward bass tuned to {key}.", + "melodicArp": "Plucked arpeggio supporting the lead.", + "grooveAndTiming": f"Straight {time_signature} groove at {bpm:.0f} BPM.", + "effectsAndTexture": "Light reverb and saturation for glue.", + }, + "mixAndMasterChain": [ + { + "order": 1, + "device": "EQ Eight", + "deviceFamily": "NATIVE", + "trackContext": "Master", + "workflowStage": "MASTER", + "parameter": "Low Cut", + "value": "30 Hz, 24 dB/oct", + "reason": "Clear inaudible sub-rumble below the fundamental.", + "phase1Fields": cites_tonal, + }, + { + "order": 2, + "device": "Glue Compressor", + "deviceFamily": "NATIVE", + "trackContext": "Master", + "workflowStage": "MASTER", + "parameter": "Ratio", + "value": "2:1, slow attack", + "reason": f"Gentle bus glue toward {lufs_text}.", + "phase1Fields": cites_loudness, + }, + ], + "secretSauce": { + "title": "Measurement-grounded mock recipe", + "explanation": "A deterministic recipe demonstrating the citation contract.", + "implementationSteps": [ + "Set the project tempo to the measured BPM.", + "Tune the bass to the detected key.", + ], + "workflowSteps": [ + { + "step": 1, + "trackContext": "Master", + "device": "Utility", + "parameter": "Gain", + "value": "0.0 dB", + "instruction": "Confirm gain staging before the limiter.", + "measurementJustification": f"Targeting {lufs_text} measured in Phase 1.", + "phase1Fields": cites_loudness, + } + ], + }, + "confidenceNotes": [ + { + "field": "key", + "value": key, + "reason": "Key estimate carries moderate confidence.", + } + ], + "abletonRecommendations": [ + { + "device": "Operator", + "deviceFamily": "NATIVE", + "trackContext": "Bass", + "workflowStage": "SOUND_DESIGN", + "category": "SYNTHESIS", + "parameter": "Oscillator A Coarse", + "value": "tuned to the detected key root", + "reason": f"Anchor the bass to {key}.", + "advancedTip": "Add a second detuned voice for width.", + "phase1Fields": cites_tonal, + }, + { + "device": "Glue Compressor", + "deviceFamily": "NATIVE", + "trackContext": "Master", + "workflowStage": "MASTER", + "category": "DYNAMICS", + "parameter": "Makeup", + "value": "+2 dB", + "reason": f"Move loudness toward {lufs_text}.", + "advancedTip": "Engage soft-clip for extra glue.", + "phase1Fields": cites_loudness, + }, + ], + } diff --git a/apps/backend/moss_sidecar/requirements.txt b/apps/backend/moss_sidecar/requirements.txt new file mode 100644 index 00000000..c1c9fa43 --- /dev/null +++ b/apps/backend/moss_sidecar/requirements.txt @@ -0,0 +1,22 @@ +# MOSS-Audio Phase 2 sidecar — isolated venv (NOT the product venv). +# +# Build: +# python3.11 -m venv apps/backend/moss_sidecar/.venv +# apps/backend/moss_sidecar/.venv/bin/pip install -r apps/backend/moss_sidecar/requirements.txt +# Run (mock mode, default): +# apps/backend/moss_sidecar/.venv/bin/python -m moss_sidecar.app # from apps/backend +# +# Mock mode needs only FastAPI + a server. The mock interpreter is pure stdlib. +fastapi>=0.110 +uvicorn>=0.29 + +# --- Real MOSS-Audio model mode: NOT a pinned list, and NOT installed here. --- +# STEP ONE licence gate (docs/PHASE2_PROVIDER.md): the MOSS-Audio modeling code +# has no effective licence (the OpenMOSS/MOSS-Audio GitHub repo's pyproject points +# at a LICENSE file that does not exist; only the *weights* are Apache-2.0). This +# sidecar therefore ships NO OpenMOSS code and the model path is a 501 stub. +# +# A real model venv would additionally need torch + transformers + the (currently +# unlicensed) OpenMOSS modeling code OR a cleanly-licensed runtime that can serve +# the Apache-2.0 weights for audio-in inference. Do not add those here until the +# licence is resolved. diff --git a/apps/backend/phase2_provider.py b/apps/backend/phase2_provider.py new file mode 100644 index 00000000..812d6ca5 --- /dev/null +++ b/apps/backend/phase2_provider.py @@ -0,0 +1,254 @@ +"""Phase 2 interpretation provider abstraction (gemini | moss). + +The product default is Gemini, whose call lives inline in ``server.py`` +(``_run_interpretation_request_with_profile_config``) and is **unchanged**. This +module adds a pluggable seam so the *same* Phase 2 request — same built prompt, +same audio, same authoritative Phase 1 JSON, same ``response_schema`` — can be +routed to a self-hosted OpenMOSS **MOSS-Audio** model running as a FastAPI +sidecar instead, while flowing through the *identical* downstream parse / +validate / citation / catalogue path in ``server.py``. That shared tail is what +guarantees the "identical recommendation schema" DoD: both providers emit a raw +``Phase2Result`` JSON string, and ``server.py`` validates both the same way. + +Selection is env-gated and **default-off**:: + + ASA_PHASE2_PROVIDER = "gemini" (default) | "moss" + +``resolve_external_phase2_provider()`` returns ``None`` for the Gemini default +(meaning: use ``server.py``'s native Gemini path, byte-for-byte unchanged) and a +``MossSidecarProvider`` only when ``moss`` is explicitly selected. The MOSS path +applies to the ``producer_summary`` profile only (the recommendation-emitting +interpretation); the ``stem_summary`` path stays on Gemini. + +LICENCE NOTE (STEP ONE gate — see ``docs/PHASE2_PROVIDER.md`` and the +``asa-moss-audio-licence-gate`` memory): MOSS-Audio *weights* are Apache-2.0, but +the *modeling code* you must run carries no effective licence (the GitHub repo's +``pyproject`` points at a ``LICENSE`` file that does not exist). This abstraction +and the sidecar therefore execute **no** OpenMOSS code on the product path: the +sidecar ships a deterministic mock interpreter for the experiment/eval, and the +real model wiring is a documented, gated stub. Gemini stays the product default. +This is a default-off research experiment, mirroring the MSST separation backend. +""" + +from __future__ import annotations + +import json +import os +from dataclasses import dataclass, field +from typing import Any, Protocol, runtime_checkable + + +DEFAULT_PHASE2_PROVIDER = "gemini" +SUPPORTED_PHASE2_PROVIDERS = ("gemini", "moss") + +_DEFAULT_SIDECAR_URL = "http://127.0.0.1:8200" +_DEFAULT_SIDECAR_TIMEOUT_SECONDS = 180 +# Mirror server.py's INLINE_SIZE_LIMIT intent: above this we do not inline-base64 +# the audio into the JSON body. The sidecar/model path is a default-off +# experiment, so the simple JSON+base64 transport is acceptable for the small +# eval corpus; large files degrade to a typed, retryable error rather than +# silently shipping a huge body. +_MAX_INLINE_AUDIO_BYTES = 24 * 1024 * 1024 + + +@dataclass +class Phase2ProviderRequest: + """Everything a provider needs to produce a raw ``Phase2Result`` JSON string. + + ``prompt`` already embeds the authoritative Phase 1 JSON, the Live 12 device + catalogue, and grounding metadata (built by ``server_phase2._build_phase2_prompt``). + ``phase1_result`` is forwarded *separately* as well so a sidecar can ground + citations in real measurement paths without re-parsing the prompt — matching + the goal's "fed the same audio + Phase-1 JSON". + """ + + prompt: str + response_schema: dict[str, Any] + phase1_result: dict[str, Any] + model_name: str + request_id: str + source_path: str | None = None + filename: str | None = None + mime_type: str | None = None + file_size_bytes: int | None = None + + +@dataclass +class Phase2ProviderResponse: + """A provider's raw output, shaped to slot into ``server.py``'s shared tail. + + ``text`` is the raw JSON string of a ``Phase2Result`` (or ``None`` to signal + a skip) — the SAME thing ``response.text`` is for the Gemini path. It is fed + to the profile's ``parseResult`` / ``parseDebugResult`` and then through the + citation + catalogue validators, exactly like Gemini output. + """ + + text: str | None + flags: list[str] = field(default_factory=list) + message_suffix: str | None = None + + +class Phase2ProviderError(Exception): + """A provider-side failure that ``server.py`` converts to an execution error. + + Carries the fields the interpretation execution dict needs so a MOSS sidecar + failure degrades the same way a Gemini failure does (retryable, surfaced in + the UI as a failed attempt — never a silent success). + """ + + def __init__( + self, + message: str, + *, + error_code: str = "MOSS_SIDECAR_FAILED", + status_code: int = 502, + retryable: bool = True, + ) -> None: + super().__init__(message) + self.error_code = error_code + self.status_code = status_code + self.retryable = retryable + + +@runtime_checkable +class Phase2Provider(Protocol): + name: str + + def generate(self, request: Phase2ProviderRequest) -> Phase2ProviderResponse: ... + + +class MossSidecarProvider: + """HTTP client for the MOSS-Audio FastAPI sidecar (``moss_sidecar/app.py``). + + Posts ``{prompt, response_schema, phase1, model, audio?}`` to ``/v1/phase2`` + and returns the sidecar's ``Phase2Result`` re-serialized to a JSON string so + ``server.py`` validates it through the identical path as Gemini. Any + transport / non-2xx / unparseable-response condition raises a typed, + retryable ``Phase2ProviderError`` so the run degrades cleanly (the licence + gate already means MOSS is never the product default). + """ + + name = "moss" + + def __init__( + self, + *, + base_url: str, + timeout_seconds: float = _DEFAULT_SIDECAR_TIMEOUT_SECONDS, + model_id: str | None = None, + ) -> None: + self.base_url = base_url.rstrip("/") + self.timeout_seconds = timeout_seconds + self.model_id = model_id + + @classmethod + def from_env(cls) -> "MossSidecarProvider": + return cls( + base_url=os.getenv("ASA_MOSS_SIDECAR_URL", _DEFAULT_SIDECAR_URL), + timeout_seconds=float( + os.getenv("ASA_MOSS_SIDECAR_TIMEOUT_SECONDS", str(_DEFAULT_SIDECAR_TIMEOUT_SECONDS)) + ), + model_id=os.getenv("ASA_MOSS_MODEL_ID") or None, + ) + + def _encode_audio(self, request: Phase2ProviderRequest) -> dict[str, Any] | None: + path = request.source_path + if not path or not os.path.isfile(path): + # No rendered audio is a valid state for the experiment (the + # recommendation corpus ships Phase 1 fingerprints without renders). + # The sidecar grounds on Phase 1 JSON; audio is additive. + return None + size = os.path.getsize(path) + if size > _MAX_INLINE_AUDIO_BYTES: + raise Phase2ProviderError( + f"Audio {size} bytes exceeds the {_MAX_INLINE_AUDIO_BYTES}-byte inline " + "limit for the MOSS sidecar transport.", + error_code="MOSS_AUDIO_TOO_LARGE", + status_code=413, + retryable=False, + ) + import base64 + + with open(path, "rb") as handle: + encoded = base64.b64encode(handle.read()).decode("ascii") + return { + "filename": request.filename or os.path.basename(path), + "mime_type": request.mime_type or "application/octet-stream", + "base64": encoded, + } + + def generate(self, request: Phase2ProviderRequest) -> Phase2ProviderResponse: + import requests # product dependency (requests==2.32.5); imported lazily + + body = { + "prompt": request.prompt, + "response_schema": request.response_schema, + "phase1": request.phase1_result, + "model": self.model_id or request.model_name, + "request_id": request.request_id, + "audio": self._encode_audio(request), + } + url = f"{self.base_url}/v1/phase2" + try: + response = requests.post(url, json=body, timeout=self.timeout_seconds) + except requests.RequestException as exc: + raise Phase2ProviderError( + f"MOSS sidecar request to {url} failed: {exc}", + error_code="MOSS_SIDECAR_UNREACHABLE", + ) from exc + + if response.status_code != 200: + snippet = response.text[:200] if isinstance(response.text, str) else "" + raise Phase2ProviderError( + f"MOSS sidecar returned HTTP {response.status_code}: {snippet}", + error_code="MOSS_SIDECAR_HTTP_ERROR", + # Only transient server errors are worth retrying. 4xx and 501 + # (the licence-gated "model not wired" stub) are permanent — a + # retry loop against them would never converge. + retryable=response.status_code in (500, 502, 503, 504), + ) + + try: + payload = response.json() + except ValueError as exc: + raise Phase2ProviderError( + f"MOSS sidecar returned non-JSON: {exc}", + error_code="MOSS_SIDECAR_BAD_RESPONSE", + ) from exc + + result = payload.get("result") + provider_label = str(payload.get("provider") or self.name) + warnings = payload.get("warnings") + suffix = f"MOSS interpretation complete ({provider_label})." + if isinstance(warnings, list) and warnings: + suffix = f"{suffix} {len(warnings)} sidecar warning(s)." + # ``None`` result → text=None → server's parse path emits a skip, just + # like an empty Gemini response. A dict result is re-serialized so the + # shared tail parses + validates it identically to Gemini output. + text = None if result is None else json.dumps(result) + return Phase2ProviderResponse( + text=text, + flags=[f"provider:{provider_label}"], + message_suffix=suffix, + ) + + +def resolve_phase2_provider_name() -> str: + """The configured provider id, defaulting to ``gemini``. + + Unknown values degrade to ``gemini`` (the product default) rather than + raising — a typo in the env must never take Phase 2 down. + """ + raw = os.getenv("ASA_PHASE2_PROVIDER", DEFAULT_PHASE2_PROVIDER).strip().lower() + return raw if raw in SUPPORTED_PHASE2_PROVIDERS else DEFAULT_PHASE2_PROVIDER + + +def resolve_external_phase2_provider() -> Phase2Provider | None: + """Return a non-Gemini provider, or ``None`` to use server.py's native Gemini. + + ``None`` is the default — the product Gemini path is left entirely untouched + unless ``ASA_PHASE2_PROVIDER=moss`` is explicitly set. + """ + if resolve_phase2_provider_name() == "moss": + return MossSidecarProvider.from_env() + return None diff --git a/apps/backend/phase2_provider_evaluation.py b/apps/backend/phase2_provider_evaluation.py new file mode 100644 index 00000000..07b7aefb --- /dev/null +++ b/apps/backend/phase2_provider_evaluation.py @@ -0,0 +1,278 @@ +"""Citation-accuracy evaluation for Phase 2 providers (gemini vs moss). + +Research-only. Scores each provider's ``Phase2Result`` on the fixed +``tests/fixtures/recommendation_tracks/`` corpus by **reusing the production +validators** — not a parallel scorer: + + - ``server_phase2._validate_phase2_citation_paths`` → invented (non-resolving) + cited paths. Citation accuracy = (cited − invented) / cited. + - ``server_phase2._is_valid_phase2_shape`` → schema validity (the "identical + schema" DoD, enforced on every provider's output). + - ``server_phase2.apply_live12_catalogue_gates`` → device/parameter catalogue + warnings (advisory). + - grounding coverage = fraction of recommendations that cite ≥1 Phase 1 field + (PURPOSE.md invariant #2: every recommendation cites, never invents). + +The DoD's headline "offline-good-enough vs Gemini-wins" split needs BOTH +providers producing real output on the same set. Today that live comparison is +blocked (see ``docs/PHASE2_PROVIDER.md``): no ``GEMINI_API_KEY`` locally, no audio +renders in the corpus, and MOSS is not runnable under the STEP ONE licence gate. +So this harness runs whatever providers ARE available (always the deterministic +mock; Gemini/sidecar when supplied), computes the framework, and labels the +missing legs ``BLOCKED`` rather than fabricating numbers. The mock's accuracy is +~1.0 by construction — it proves the pipeline + scorer, not model quality. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable + +from server_phase2 import ( + _is_valid_phase2_shape, + _validate_phase2_citation_paths, + apply_live12_catalogue_gates, +) + +# Within-this-margin-of-Gemini counts as "offline-good-enough" for the split. +OFFLINE_GOOD_ENOUGH_MARGIN = 0.05 + +_CITATION_RECORD_PATHS = ("mixAndMasterChain", "abletonRecommendations") + + +def default_corpus_dir() -> Path: + return Path(__file__).parent / "tests" / "fixtures" / "recommendation_tracks" + + +@dataclass +class CorpusTrack: + track_id: str + manifest: dict[str, Any] + phase1: dict[str, Any] + + +def load_corpus(corpus_dir: Path | None = None) -> list[CorpusTrack]: + root = corpus_dir or default_corpus_dir() + tracks: list[CorpusTrack] = [] + for entry in sorted(root.iterdir()): + if not entry.is_dir() or entry.name.startswith("_"): + continue + manifest_path = entry / "manifest.json" + fingerprint_path = entry / "phase1_fingerprint.json" + if not manifest_path.is_file() or not fingerprint_path.is_file(): + continue + manifest = json.loads(manifest_path.read_text(encoding="utf-8")) + phase1 = json.loads(fingerprint_path.read_text(encoding="utf-8")) + tracks.append(CorpusTrack(track_id=entry.name, manifest=manifest, phase1=phase1)) + return tracks + + +def _count_cited_paths(result: dict[str, Any]) -> tuple[int, int, int]: + """Return (total_cited_paths, recommendations, recommendations_with_a_citation). + + Walks the same record sets ``_validate_phase2_citation_paths`` does for the + recommendation surfaces, so "total cited" is the denominator that pairs with + its invented-path warnings. + """ + total_cited = 0 + rec_count = 0 + rec_with_cite = 0 + for key in _CITATION_RECORD_PATHS: + items = result.get(key) + if not isinstance(items, list): + continue + for item in items: + if not isinstance(item, dict): + continue + rec_count += 1 + fields = item.get("phase1Fields") + cited = [f for f in fields if isinstance(f, str)] if isinstance(fields, list) else [] + total_cited += len(cited) + if cited: + rec_with_cite += 1 + return total_cited, rec_count, rec_with_cite + + +@dataclass +class TrackScore: + track_id: str + available: bool + note: str = "" + schema_valid: bool = False + recommendation_count: int = 0 + cited_path_total: int = 0 + invented_path_count: int = 0 + grounding_coverage: float | None = None + citation_accuracy: float | None = None + catalogue_warning_count: int = 0 + + +def score_result( + track: CorpusTrack, + result: dict[str, Any] | None, + *, + available: bool, + note: str = "", +) -> TrackScore: + if not available or result is None: + return TrackScore(track_id=track.track_id, available=False, note=note or "no output") + + schema_valid = _is_valid_phase2_shape(result) + invented = _validate_phase2_citation_paths(result, track.phase1) + total_cited, rec_count, rec_with_cite = _count_cited_paths(result) + try: + catalogue_warnings = apply_live12_catalogue_gates(result, request_id=track.track_id) + except Exception: # advisory only — never fail a score on a catalogue error + catalogue_warnings = [] + + citation_accuracy = ( + (total_cited - len(invented)) / total_cited if total_cited else None + ) + grounding_coverage = (rec_with_cite / rec_count) if rec_count else None + + return TrackScore( + track_id=track.track_id, + available=True, + note=note, + schema_valid=schema_valid, + recommendation_count=rec_count, + cited_path_total=total_cited, + invented_path_count=len(invented), + grounding_coverage=grounding_coverage, + citation_accuracy=citation_accuracy, + catalogue_warning_count=len(catalogue_warnings), + ) + + +# A provider function maps a track to (result | None, available, note). +ProviderFn = Callable[[CorpusTrack], "tuple[dict[str, Any] | None, bool, str]"] + + +@dataclass +class ProviderAggregate: + provider: str + available: bool + blocked_reason: str = "" + track_scores: list[TrackScore] = field(default_factory=list) + + def _mean(self, attr: str) -> float | None: + values = [ + getattr(s, attr) + for s in self.track_scores + if s.available and getattr(s, attr) is not None + ] + return sum(values) / len(values) if values else None + + @property + def mean_citation_accuracy(self) -> float | None: + return self._mean("citation_accuracy") + + @property + def mean_grounding_coverage(self) -> float | None: + return self._mean("grounding_coverage") + + @property + def schema_valid_rate(self) -> float | None: + scored = [s for s in self.track_scores if s.available] + return sum(1 for s in scored if s.schema_valid) / len(scored) if scored else None + + +def evaluate_provider( + provider: str, + provider_fn: ProviderFn, + tracks: list[CorpusTrack], + *, + blocked_reason: str = "", +) -> ProviderAggregate: + if blocked_reason: + return ProviderAggregate(provider=provider, available=False, blocked_reason=blocked_reason) + scores: list[TrackScore] = [] + any_available = False + for track in tracks: + result, available, note = provider_fn(track) + any_available = any_available or available + scores.append(score_result(track, result, available=available, note=note)) + return ProviderAggregate( + provider=provider, + available=any_available, + blocked_reason="" if any_available else "no tracks produced output", + track_scores=scores, + ) + + +def classify_split( + baseline: ProviderAggregate | None, + candidate: ProviderAggregate, +) -> dict[str, Any]: + """The 'offline-good-enough vs Gemini-wins' verdict for one candidate. + + ``baseline`` is Gemini. If it is missing/unavailable the verdict is BLOCKED — + the framework + threshold are reported, but no comparison is invented. + """ + if baseline is None or not baseline.available or baseline.mean_citation_accuracy is None: + return { + "verdict": "BLOCKED", + "reason": ( + "No Gemini baseline available " + f"({baseline.blocked_reason if baseline else 'baseline not run'}). " + "Provide GEMINI_API_KEY + audio renders, or recorded Gemini outputs." + ), + "marginUsed": OFFLINE_GOOD_ENOUGH_MARGIN, + } + if candidate.mean_citation_accuracy is None: + return {"verdict": "BLOCKED", "reason": f"{candidate.provider} produced no scores."} + delta = candidate.mean_citation_accuracy - baseline.mean_citation_accuracy + verdict = "OFFLINE_GOOD_ENOUGH" if delta >= -OFFLINE_GOOD_ENOUGH_MARGIN else "GEMINI_WINS" + return { + "verdict": verdict, + "candidateCitationAccuracy": candidate.mean_citation_accuracy, + "baselineCitationAccuracy": baseline.mean_citation_accuracy, + "delta": delta, + "marginUsed": OFFLINE_GOOD_ENOUGH_MARGIN, + } + + +def build_report(aggregates: list[ProviderAggregate]) -> dict[str, Any]: + by_name = {a.provider: a for a in aggregates} + baseline = by_name.get("gemini") + splits = { + a.provider: classify_split(baseline, a) + for a in aggregates + if a.provider != "gemini" + } + return { + "corpusNote": ( + "Fixed recommendation_tracks corpus. The mock provider's citation " + "accuracy is ~1.0 by construction (contract fixture, not a quality " + "proxy). Live MOSS-vs-Gemini is blocked: see docs/PHASE2_PROVIDER.md." + ), + "offlineGoodEnoughMargin": OFFLINE_GOOD_ENOUGH_MARGIN, + "providers": { + a.provider: { + "available": a.available, + "blockedReason": a.blocked_reason, + "meanCitationAccuracy": a.mean_citation_accuracy, + "meanGroundingCoverage": a.mean_grounding_coverage, + "schemaValidRate": a.schema_valid_rate, + "tracks": [ + { + "trackId": s.track_id, + "available": s.available, + "note": s.note, + "schemaValid": s.schema_valid, + "recommendationCount": s.recommendation_count, + "citedPathTotal": s.cited_path_total, + "inventedPathCount": s.invented_path_count, + "citationAccuracy": s.citation_accuracy, + "groundingCoverage": s.grounding_coverage, + "catalogueWarningCount": s.catalogue_warning_count, + } + for s in a.track_scores + ], + } + for a in aggregates + }, + "splitVerdicts": splits, + } diff --git a/apps/backend/scripts/evaluate_phase2_providers.py b/apps/backend/scripts/evaluate_phase2_providers.py new file mode 100644 index 00000000..be961005 --- /dev/null +++ b/apps/backend/scripts/evaluate_phase2_providers.py @@ -0,0 +1,173 @@ +#!/usr/bin/env python3 +"""Citation-accuracy eval: Phase 2 providers (gemini vs moss) on the fixed corpus. + +Research-only harness for the Phase2Provider goal's DoD. Wraps +``phase2_provider_evaluation.py`` (which reuses the production validators). Runs +the always-available deterministic mock, plus Gemini and/or a live MOSS sidecar +when you supply them, and prints the "offline-good-enough vs Gemini-wins" split +(labelled BLOCKED for any leg that can't run — never fabricated). + +Examples: + # Mock only (always works; proves the pipeline + scorer): + ./venv/bin/python scripts/evaluate_phase2_providers.py + + # Score a live MOSS sidecar (start it first, see moss_sidecar/README.md): + ./venv/bin/python scripts/evaluate_phase2_providers.py --sidecar-url http://127.0.0.1:8200 + + # Compare against recorded Gemini outputs (one .json per track): + ./venv/bin/python scripts/evaluate_phase2_providers.py --recorded-gemini-dir /path/to/gemini_outputs + + # Write the full JSON report: + ./venv/bin/python scripts/evaluate_phase2_providers.py --out .runtime/phase2_providers_report.json + +See docs/PHASE2_PROVIDER.md for why the live MOSS-vs-Gemini numbers are currently +blocked (licence gate, no Gemini key, no audio renders). +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path + +_BACKEND_DIR = Path(__file__).resolve().parent.parent +if str(_BACKEND_DIR) not in sys.path: + sys.path.insert(0, str(_BACKEND_DIR)) + +from moss_sidecar.mock_interpreter import build_mock_phase2_result # noqa: E402 +from phase2_provider_evaluation import ( # noqa: E402 + CorpusTrack, + build_report, + evaluate_provider, + load_corpus, +) + + +def _mock_provider(track: CorpusTrack): + return build_mock_phase2_result(track.phase1), True, "deterministic mock" + + +def _recorded_gemini_provider_factory(recorded_dir: Path): + def _fn(track: CorpusTrack): + path = recorded_dir / f"{track.track_id}.json" + if not path.is_file(): + return None, False, f"no recorded output at {path.name}" + try: + return json.loads(path.read_text(encoding="utf-8")), True, "recorded" + except (ValueError, OSError) as exc: + return None, False, f"unreadable recorded output: {exc}" + + return _fn + + +def _sidecar_provider_factory(sidecar_url: str): + import requests + + def _fn(track: CorpusTrack): + try: + response = requests.post( + f"{sidecar_url.rstrip('/')}/v1/phase2", + json={ + "prompt": "", + "response_schema": {}, + "phase1": track.phase1, + "model": "moss-audio", + "audio": None, + }, + timeout=180, + ) + except requests.RequestException as exc: + return None, False, f"sidecar unreachable: {exc}" + if response.status_code != 200: + return None, False, f"sidecar HTTP {response.status_code}" + result = response.json().get("result") + return result, result is not None, "live sidecar" + + return _fn + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--corpus-dir", type=Path, default=None) + parser.add_argument( + "--recorded-gemini-dir", + type=Path, + default=None, + help="Directory with one .json recorded Gemini Phase2Result per track.", + ) + parser.add_argument( + "--sidecar-url", + type=str, + default=None, + help="Base URL of a running MOSS sidecar to score (e.g. http://127.0.0.1:8200).", + ) + parser.add_argument("--out", type=Path, default=None, help="Write the full JSON report here.") + args = parser.parse_args(argv) + + tracks = load_corpus(args.corpus_dir) + if not tracks: + print("No corpus tracks found.", file=sys.stderr) + return 1 + + aggregates = [evaluate_provider("moss-mock", _mock_provider, tracks)] + + # Gemini baseline: recorded outputs if provided, else BLOCKED with the reason. + if args.recorded_gemini_dir is not None: + aggregates.append( + evaluate_provider( + "gemini", + _recorded_gemini_provider_factory(args.recorded_gemini_dir), + tracks, + ) + ) + else: + reason = ( + "no --recorded-gemini-dir and live Gemini not run by this harness" + + ("" if os.getenv("GEMINI_API_KEY") else "; GEMINI_API_KEY unset") + + "; corpus has no audio renders" + ) + aggregates.append( + evaluate_provider("gemini", _mock_provider, tracks, blocked_reason=reason) + ) + + if args.sidecar_url: + aggregates.append( + evaluate_provider( + "moss-sidecar", _sidecar_provider_factory(args.sidecar_url), tracks + ) + ) + + report = build_report(aggregates) + + print(f"\nPhase 2 provider eval — {len(tracks)} tracks\n" + "=" * 52) + for name, data in report["providers"].items(): + if not data["available"]: + print(f" {name:<13} BLOCKED — {data['blockedReason']}") + continue + acc = data["meanCitationAccuracy"] + cov = data["meanGroundingCoverage"] + valid = data["schemaValidRate"] + print( + f" {name:<13} schemaValid={_pct(valid)} " + f"citationAccuracy={_pct(acc)} groundingCoverage={_pct(cov)}" + ) + print("\nSplit verdicts (offline-good-enough vs Gemini-wins):") + for name, verdict in report["splitVerdicts"].items(): + print(f" {name:<13} {verdict['verdict']}" + + (f" — {verdict['reason']}" if verdict.get("reason") else "")) + + if args.out is not None: + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(report, indent=2), encoding="utf-8") + print(f"\nFull report → {args.out}") + return 0 + + +def _pct(value: float | None) -> str: + return "n/a" if value is None else f"{value * 100:.1f}%" + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/apps/backend/server.py b/apps/backend/server.py index 8d492209..7c224621 100644 --- a/apps/backend/server.py +++ b/apps/backend/server.py @@ -150,6 +150,11 @@ from recommendations_contract import build_validated_recommendations import server_samples +from phase2_provider import ( + Phase2ProviderError, + Phase2ProviderRequest, + resolve_external_phase2_provider, +) app = FastAPI(title="Sonic Analyzer Local API") @@ -1834,36 +1839,43 @@ def _run_interpretation_request_with_profile_config( mime_type = _get_audio_mime_type(filename) descriptor_hooks = _build_descriptor_hooks(measurement_result) - if not _GENAI_AVAILABLE: - return { - "ok": False, - "statusCode": 500, - "errorCode": "GEMINI_NOT_INSTALLED", - "message": "google-genai package is not installed on the backend.", - "retryable": False, - "diagnostics": None, - } + # Phase 2 provider seam (default-off; see phase2_provider.py). None means + # "use the native Gemini path below", which stays byte-for-byte unchanged. + # An external provider (ASA_PHASE2_PROVIDER=moss) routes the SAME request to a + # sidecar and flows back through the identical parse/validate/citation tail. + external_provider = resolve_external_phase2_provider() - api_key = os.getenv("GEMINI_API_KEY", "").strip() - if not api_key: - return { - "ok": False, - "statusCode": 500, - "errorCode": "GEMINI_NOT_CONFIGURED", - "message": "GEMINI_API_KEY is not set on the backend.", - "retryable": False, - "diagnostics": None, - } + if external_provider is None: + if not _GENAI_AVAILABLE: + return { + "ok": False, + "statusCode": 500, + "errorCode": "GEMINI_NOT_INSTALLED", + "message": "google-genai package is not installed on the backend.", + "retryable": False, + "diagnostics": None, + } - if model_name not in ALLOWED_GEMINI_MODELS: - return { - "ok": False, - "statusCode": 400, - "errorCode": "INVALID_MODEL", - "message": f"model_name '{model_name}' is not allowed. Must be one of: {sorted(ALLOWED_GEMINI_MODELS)}", - "retryable": False, - "diagnostics": None, - } + api_key = os.getenv("GEMINI_API_KEY", "").strip() + if not api_key: + return { + "ok": False, + "statusCode": 500, + "errorCode": "GEMINI_NOT_CONFIGURED", + "message": "GEMINI_API_KEY is not set on the backend.", + "retryable": False, + "diagnostics": None, + } + + if model_name not in ALLOWED_GEMINI_MODELS: + return { + "ok": False, + "statusCode": 400, + "errorCode": "INVALID_MODEL", + "message": f"model_name '{model_name}' is not allowed. Must be one of: {sorted(ALLOWED_GEMINI_MODELS)}", + "retryable": False, + "diagnostics": None, + } prompt = profile_config["buildPrompt"]( measurement_result=measurement_result, @@ -1872,75 +1884,107 @@ def _run_interpretation_request_with_profile_config( descriptor_hooks=descriptor_hooks, mt3_result=mt3_result, ) - client = _genai.Client( - api_key=api_key, - http_options={"timeout": GEMINI_TIMEOUT_SECONDS * 1_000}, - ) - generate_config = _genai_types.GenerateContentConfig( - response_mime_type="application/json", - response_schema=profile_config["responseSchema"], - ) + # Construct the Gemini client BEFORE the try (only when no external provider + # is selected) so a client-construction error propagates to the setup wrapper + # (INTERPRETATION_SETUP_FAILED) exactly as it did before this seam existed — + # not caught locally as a generate failure. + client = None + generate_config = None + if external_provider is None: + client = _genai.Client( + api_key=api_key, + http_options={"timeout": GEMINI_TIMEOUT_SECONDS * 1_000}, + ) + generate_config = _genai_types.GenerateContentConfig( + response_mime_type="application/json", + response_schema=profile_config["responseSchema"], + ) + api_started_at = _current_time() uploaded_gemini_file = None try: - if file_size_bytes <= INLINE_SIZE_LIMIT: - flags_used.append("inline") - with open(source_path, "rb") as input_file: - audio_bytes = input_file.read() - audio_b64 = base64.b64encode(audio_bytes).decode("ascii") - media_part = {"inline_data": {"data": audio_b64, "mime_type": mime_type}} - - def _generate_inline() -> Any: - return client.models.generate_content( - model=model_name, - contents=[{"parts": [media_part, {"text": prompt}]}], - config=generate_config, + if external_provider is not None: + # MOSS (or other non-Gemini) provider — default-off experiment. The + # provider returns a raw Phase2Result JSON string (or None for a + # skip) that flows through the SAME parse/citation/catalogue tail as + # Gemini below, guaranteeing an identical recommendation schema. + provider_response = external_provider.generate( + Phase2ProviderRequest( + prompt=prompt, + response_schema=profile_config["responseSchema"], + phase1_result=measurement_result, + model_name=model_name, + request_id=request_id, + source_path=source_path, + filename=filename, + mime_type=mime_type, + file_size_bytes=file_size_bytes, ) - - response = asyncio.run(_gemini_with_retry(_generate_inline)) + ) + flags_used.append(f"phase2-provider:{external_provider.name}") + flags_used.extend(provider_response.flags) + response_text: str | None = provider_response.text + message_suffix = provider_response.message_suffix or profile_config["successMessage"] api_completed_at = _current_time() - message_suffix = profile_config["successMessage"] else: - flags_used.append("files-api") - - def _upload_file() -> Any: - return client.files.upload( - file=source_path, - config=_genai_types.UploadFileConfig( - mime_type=mime_type, - display_name=filename, - ), - ) - - upload_start = _current_time() - uploaded_gemini_file = asyncio.run(_gemini_with_retry(_upload_file)) - upload_end = _current_time() - media_part = { - "file_data": { - "file_uri": uploaded_gemini_file.uri, - "mime_type": uploaded_gemini_file.mime_type, + if file_size_bytes <= INLINE_SIZE_LIMIT: + flags_used.append("inline") + with open(source_path, "rb") as input_file: + audio_bytes = input_file.read() + audio_b64 = base64.b64encode(audio_bytes).decode("ascii") + media_part = {"inline_data": {"data": audio_b64, "mime_type": mime_type}} + + def _generate_inline() -> Any: + return client.models.generate_content( + model=model_name, + contents=[{"parts": [media_part, {"text": prompt}]}], + config=generate_config, + ) + + response = asyncio.run(_gemini_with_retry(_generate_inline)) + api_completed_at = _current_time() + message_suffix = profile_config["successMessage"] + else: + flags_used.append("files-api") + + def _upload_file() -> Any: + return client.files.upload( + file=source_path, + config=_genai_types.UploadFileConfig( + mime_type=mime_type, + display_name=filename, + ), + ) + + upload_start = _current_time() + uploaded_gemini_file = asyncio.run(_gemini_with_retry(_upload_file)) + upload_end = _current_time() + media_part = { + "file_data": { + "file_uri": uploaded_gemini_file.uri, + "mime_type": uploaded_gemini_file.mime_type, + } } - } - def _generate_files_api() -> Any: - return client.models.generate_content( - model=model_name, - contents=[{"parts": [media_part, {"text": prompt}]}], - config=generate_config, + def _generate_files_api() -> Any: + return client.models.generate_content( + model=model_name, + contents=[{"parts": [media_part, {"text": prompt}]}], + config=generate_config, + ) + + generate_start = _current_time() + response = asyncio.run(_gemini_with_retry(_generate_files_api)) + generate_end = _current_time() + api_completed_at = _current_time() + message_suffix = ( + f"{profile_config['successMessage']} " + f"Upload: {int(_elapsed_ms(upload_start, upload_end))}ms, " + f"Generate: {int(_elapsed_ms(generate_start, generate_end))}ms" ) - generate_start = _current_time() - response = asyncio.run(_gemini_with_retry(_generate_files_api)) - generate_end = _current_time() - api_completed_at = _current_time() - message_suffix = ( - f"{profile_config['successMessage']} " - f"Upload: {int(_elapsed_ms(upload_start, upload_end))}ms, " - f"Generate: {int(_elapsed_ms(generate_start, generate_end))}ms" - ) - - response_text: str | None = getattr(response, "text", None) + response_text = getattr(response, "text", None) debug_payload = None parse_validation_warnings: list[dict[str, Any]] = [] if callable(profile_config.get("parseDebugResult")): @@ -2060,6 +2104,31 @@ def _generate_files_api() -> Any: "message": message_suffix, "diagnostics": diagnostics, } + except Phase2ProviderError as exc: + # Non-Gemini provider (MOSS sidecar) failure → execution error dict, + # surfaced the same way a Gemini failure is (never a silent success). + diagnostics = _build_diagnostics( + response_ready_at=_current_time(), + request_id=request_id, + estimate={"totalLowMs": 0, "totalHighMs": 0}, + timeout_seconds=GEMINI_TIMEOUT_SECONDS, + request_started_at=request_started_at, + analysis_started_at=api_started_at, + analysis_completed_at=_current_time(), + flags_used=flags_used, + file_size_bytes=file_size_bytes, + file_duration_seconds=None, + engine_version=model_name, + stderr=str(exc), + ) + return { + "ok": False, + "statusCode": exc.status_code, + "errorCode": exc.error_code, + "message": str(exc), + "retryable": exc.retryable, + "diagnostics": diagnostics, + } except Exception as exc: error_msg = str(exc) status_code = 429 if "429" in error_msg or "quota" in error_msg.lower() else 502 diff --git a/apps/backend/tests/test_phase2_provider.py b/apps/backend/tests/test_phase2_provider.py new file mode 100644 index 00000000..65396c95 --- /dev/null +++ b/apps/backend/tests/test_phase2_provider.py @@ -0,0 +1,303 @@ +"""Tests for the Phase 2 provider abstraction, MOSS sidecar, and citation eval. + +Pure-logic / contract tests — no Essentia, no GPU, no network. Run with the +product venv, or a minimal one: ``python3.11 -m venv .venv && .venv/bin/pip +install fastapi requests httpx uvicorn`` then +``.venv/bin/python -m unittest tests.test_phase2_provider`` from apps/backend. +""" + +import json +import os +import unittest +from unittest import mock + +import phase2_provider +import phase2_provider_evaluation as evalmod +from moss_sidecar.mock_interpreter import build_mock_phase2_result +from phase2_provider import ( + MossSidecarProvider, + Phase2ProviderError, + Phase2ProviderRequest, + resolve_external_phase2_provider, + resolve_phase2_provider_name, +) +from server_phase2 import _is_valid_phase2_shape, _validate_phase2_citation_paths + + +_SAMPLE_PHASE1 = { + "bpm": 128.0, + "key": "A Minor", + "timeSignature": "4/4", + "sampleRate": 48000, + "durationSeconds": 16.0, + "lufsIntegrated": -9.2, + "lufsRange": 4.1, + "truePeak": -0.8, + "crestFactor": 11.2, + "spectralBalance": {"subBass": 0.2, "lowMid": 0.3}, + "kickDetail": {"fundamentalHz": 55.0}, +} + + +class ResolveProviderTests(unittest.TestCase): + def test_defaults_to_gemini_none(self): + with mock.patch.dict(os.environ, {}, clear=True): + self.assertEqual(resolve_phase2_provider_name(), "gemini") + self.assertIsNone(resolve_external_phase2_provider()) + + def test_unknown_value_degrades_to_gemini(self): + with mock.patch.dict(os.environ, {"ASA_PHASE2_PROVIDER": "bogus"}, clear=True): + self.assertEqual(resolve_phase2_provider_name(), "gemini") + self.assertIsNone(resolve_external_phase2_provider()) + + def test_moss_selected(self): + with mock.patch.dict(os.environ, {"ASA_PHASE2_PROVIDER": "MOSS"}, clear=True): + self.assertEqual(resolve_phase2_provider_name(), "moss") + provider = resolve_external_phase2_provider() + self.assertIsInstance(provider, MossSidecarProvider) + self.assertEqual(provider.name, "moss") + + +class MockInterpreterContractTests(unittest.TestCase): + def test_mock_is_schema_valid(self): + result = build_mock_phase2_result(_SAMPLE_PHASE1) + self.assertTrue(_is_valid_phase2_shape(result), "mock must satisfy the Phase2Result shape") + + def test_mock_citations_all_resolve(self): + result = build_mock_phase2_result(_SAMPLE_PHASE1) + invented = _validate_phase2_citation_paths(result, _SAMPLE_PHASE1) + self.assertEqual(invented, [], f"mock cited non-resolving paths: {invented}") + + def test_mock_grounds_in_supplied_phase1(self): + result = build_mock_phase2_result(_SAMPLE_PHASE1) + self.assertEqual(result["projectSetup"]["tempoBpm"], 128.0) + self.assertIn("A Minor", json.dumps(result)) + + def test_mock_is_deterministic(self): + a = build_mock_phase2_result(_SAMPLE_PHASE1) + b = build_mock_phase2_result(_SAMPLE_PHASE1) + self.assertEqual(a, b) + + def test_mock_handles_sparse_phase1(self): + # Only bpm present — must still be schema-valid with resolving citations. + sparse = {"bpm": 90.0} + result = build_mock_phase2_result(sparse) + self.assertTrue(_is_valid_phase2_shape(result)) + self.assertEqual(_validate_phase2_citation_paths(result, sparse), []) + + +class _FakeResponse: + def __init__(self, status_code, payload=None, text=""): + self.status_code = status_code + self._payload = payload + self.text = text or (json.dumps(payload) if payload is not None else "") + + def json(self): + if self._payload is None: + raise ValueError("no json") + return self._payload + + +class MossSidecarProviderTests(unittest.TestCase): + def _request(self): + return Phase2ProviderRequest( + prompt="PROMPT", + response_schema={}, + phase1_result=_SAMPLE_PHASE1, + model_name="moss-audio", + request_id="req-1", + ) + + def test_success_returns_serialized_result(self): + result = build_mock_phase2_result(_SAMPLE_PHASE1) + provider = MossSidecarProvider(base_url="http://x:8200") + with mock.patch( + "requests.post", + return_value=_FakeResponse(200, {"result": result, "provider": "moss-mock"}), + ): + response = provider.generate(self._request()) + self.assertIsNotNone(response.text) + self.assertEqual(json.loads(response.text), result) + + def test_none_result_yields_skip_text(self): + provider = MossSidecarProvider(base_url="http://x:8200") + with mock.patch( + "requests.post", + return_value=_FakeResponse(200, {"result": None, "provider": "moss-mock"}), + ): + response = provider.generate(self._request()) + self.assertIsNone(response.text) + + def test_http_error_raises_typed_error(self): + provider = MossSidecarProvider(base_url="http://x:8200") + with mock.patch("requests.post", return_value=_FakeResponse(501, text="not wired")): + with self.assertRaises(Phase2ProviderError) as ctx: + provider.generate(self._request()) + self.assertFalse(ctx.exception.retryable) # 4xx not retryable + + def test_network_error_raises_retryable(self): + import requests + + provider = MossSidecarProvider(base_url="http://x:8200") + with mock.patch("requests.post", side_effect=requests.RequestException("boom")): + with self.assertRaises(Phase2ProviderError) as ctx: + provider.generate(self._request()) + self.assertTrue(ctx.exception.retryable) + + +class CitationEvalTests(unittest.TestCase): + def _track(self): + return evalmod.CorpusTrack(track_id="t1", manifest={}, phase1=_SAMPLE_PHASE1) + + def test_mock_scores_perfect_citation_accuracy(self): + track = self._track() + result = build_mock_phase2_result(_SAMPLE_PHASE1) + score = evalmod.score_result(track, result, available=True, note="mock") + self.assertTrue(score.schema_valid) + self.assertEqual(score.citation_accuracy, 1.0) + self.assertEqual(score.grounding_coverage, 1.0) + self.assertGreater(score.cited_path_total, 0) + + def test_invented_path_lowers_accuracy(self): + track = self._track() + result = build_mock_phase2_result(_SAMPLE_PHASE1) + # Inject one invented citation into the first recommendation. + result["abletonRecommendations"][0]["phase1Fields"] = ["totallyMadeUpField"] + score = evalmod.score_result(track, result, available=True) + self.assertIsNotNone(score.citation_accuracy) + self.assertLess(score.citation_accuracy, 1.0) + self.assertGreaterEqual(score.invented_path_count, 1) + + def test_unavailable_provider_scores_blocked(self): + track = self._track() + score = evalmod.score_result(track, None, available=False, note="no key") + self.assertFalse(score.available) + self.assertFalse(score.schema_valid) + + def test_split_blocked_without_gemini_baseline(self): + tracks = [self._track()] + mock_agg = evalmod.evaluate_provider( + "moss-mock", + lambda t: (build_mock_phase2_result(t.phase1), True, "mock"), + tracks, + ) + gemini_agg = evalmod.evaluate_provider( + "gemini", lambda t: (None, False, "blocked"), tracks, blocked_reason="no key" + ) + report = evalmod.build_report([mock_agg, gemini_agg]) + self.assertEqual(report["splitVerdicts"]["moss-mock"]["verdict"], "BLOCKED") + self.assertAlmostEqual(report["providers"]["moss-mock"]["meanCitationAccuracy"], 1.0) + + +class MossBranchIntegrationTests(unittest.TestCase): + """Executes the modified server.py function through the MOSS branch. + + This is the test that proves the abstraction is *in-path* (not dead code): + it calls ``_run_interpretation_request_with_profile_config`` with + ``ASA_PHASE2_PROVIDER=moss`` and a mocked sidecar HTTP, and asserts the + result flows through the SAME shared parse/validate tail as Gemini. Requires + the full backend venv (``import server`` pulls numpy/Essentia); self-skips in + the lightweight env so the rest of this module still runs. + """ + + @classmethod + def setUpClass(cls): + try: + import server # noqa: F401 + except Exception as exc: # pragma: no cover - env-dependent + raise unittest.SkipTest(f"server import unavailable (needs full venv): {exc}") + + def test_moss_provider_flows_through_shared_tail(self): + import server + + result = build_mock_phase2_result(_SAMPLE_PHASE1) + fake = _FakeResponse(200, {"result": result, "provider": "moss-mock"}) + profile_config = server._resolve_interpretation_profile_config("producer_summary") + with mock.patch.dict(os.environ, {"ASA_PHASE2_PROVIDER": "moss"}, clear=False), \ + mock.patch("requests.post", return_value=fake): + execution = server._run_interpretation_request_with_profile_config( + source_path=__file__, # any real file; mock sidecar ignores content + filename="loop.flac", + file_size_bytes=123, + profile_id="producer_summary", + profile_config=profile_config, + measurement_result=_SAMPLE_PHASE1, + pitch_note_result=None, + grounding_metadata={}, + model_name="gemini-2.5-flash", + request_id="itest-moss-1", + ) + self.assertTrue(execution["ok"], execution) + self.assertIsNotNone(execution["interpretationResult"]) + self.assertTrue(_is_valid_phase2_shape(execution["interpretationResult"])) + + def test_gemini_default_does_not_touch_sidecar(self): + import server + + # With the default provider, requests.post must never be called (the + # native Gemini path is taken). We don't need a key: the GEMINI_NOT_CONFIGURED + # early-return proves the Gemini branch ran, and the sidecar was untouched. + with mock.patch.dict(os.environ, {"ASA_PHASE2_PROVIDER": "gemini", "GEMINI_API_KEY": ""}, + clear=False), \ + mock.patch("requests.post") as posted: + execution = server._run_interpretation_request_with_profile_config( + source_path=__file__, + filename="loop.flac", + file_size_bytes=123, + profile_id="producer_summary", + profile_config=server._resolve_interpretation_profile_config("producer_summary"), + measurement_result=_SAMPLE_PHASE1, + pitch_note_result=None, + grounding_metadata={}, + model_name="gemini-2.5-flash", + request_id="itest-gem-1", + ) + posted.assert_not_called() + self.assertFalse(execution["ok"]) + self.assertEqual(execution["errorCode"], "GEMINI_NOT_CONFIGURED") + + +class ClassifySplitTests(unittest.TestCase): + """The DoD's 'offline-good-enough vs Gemini-wins' threshold logic.""" + + def _agg(self, name: str, accuracy: float | None) -> evalmod.ProviderAggregate: + scores = ( + [evalmod.TrackScore(track_id="t", available=True, schema_valid=True, + citation_accuracy=accuracy)] + if accuracy is not None + else [] + ) + return evalmod.ProviderAggregate( + provider=name, available=accuracy is not None, track_scores=scores + ) + + def test_candidate_within_margin_is_offline_good_enough(self): + # Clearly inside the margin (avoid the exact float boundary). + baseline = self._agg("gemini", 0.90) + candidate = self._agg("moss", 0.90 - evalmod.OFFLINE_GOOD_ENOUGH_MARGIN + 0.01) + verdict = evalmod.classify_split(baseline, candidate) + self.assertEqual(verdict["verdict"], "OFFLINE_GOOD_ENOUGH") + + def test_candidate_outside_margin_is_gemini_wins(self): + # Clearly outside the margin. + baseline = self._agg("gemini", 0.90) + candidate = self._agg("moss", 0.90 - evalmod.OFFLINE_GOOD_ENOUGH_MARGIN - 0.01) + verdict = evalmod.classify_split(baseline, candidate) + self.assertEqual(verdict["verdict"], "GEMINI_WINS") + self.assertLess(verdict["delta"], 0) + + def test_candidate_better_than_baseline_is_offline_good_enough(self): + baseline = self._agg("gemini", 0.80) + candidate = self._agg("moss", 0.95) + verdict = evalmod.classify_split(baseline, candidate) + self.assertEqual(verdict["verdict"], "OFFLINE_GOOD_ENOUGH") + self.assertGreater(verdict["delta"], 0) + + def test_missing_baseline_is_blocked(self): + candidate = self._agg("moss", 0.95) + verdict = evalmod.classify_split(self._agg("gemini", None), candidate) + self.assertEqual(verdict["verdict"], "BLOCKED") + + +if __name__ == "__main__": + unittest.main() diff --git a/docs/PHASE2_PROVIDER.md b/docs/PHASE2_PROVIDER.md new file mode 100644 index 00000000..6d53c78b --- /dev/null +++ b/docs/PHASE2_PROVIDER.md @@ -0,0 +1,179 @@ +# Phase 2 Provider Abstraction (Gemini ↔ MOSS) — STEP ONE Licence Gate + Build + +Status: **STEP ONE complete — split verdict (weights clean, code unlicensed). +Maintainer chose to build the default-off research experiment (option A). The +provider abstraction, the mock-capable MOSS sidecar, and the citation-accuracy +eval are implemented and verified; the real MOSS model path stays a licence-gated +stub and the live MOSS-vs-Gemini numbers remain blocked (see below).** + +> **What was built (option A).** All default-off; Gemini stays the product default. +> - `apps/backend/phase2_provider.py` — `Phase2Provider` seam + `MossSidecarProvider` +> + `resolve_external_phase2_provider()` (env `ASA_PHASE2_PROVIDER`, default `gemini`). +> - `apps/backend/server.py` — one guarded asymmetric branch in +> `_run_interpretation_request_with_profile_config`; the Gemini path is unchanged +> when no external provider is selected. MOSS output flows through the **same** +> shared tail — parse/salvage + `_validate_phase2_citation_paths` + +> `apply_live12_catalogue_gates` + the frozen `recommendations.v1` contract +> (ADR 0003, `build_validated_recommendations`) — so it gets the identical +> schema and chain-of-custody enforcement for free. +> - `apps/backend/moss_sidecar/` — standalone FastAPI sidecar (own venv). `mock` +> mode = deterministic, schema-valid, Phase-1-grounded output; `model` mode = a +> 501 stub that executes **no** OpenMOSS code (licence gate). +> - `apps/backend/scripts/evaluate_phase2_providers.py` + +> `phase2_provider_evaluation.py` — citation-accuracy eval over the fixed +> `recommendation_tracks/` corpus, reusing the production validators. +> - `apps/backend/tests/test_phase2_provider.py` — 16 contract tests (resolver, +> mock schema/citation validity, sidecar client, eval scorer). +> +> **Verified** (Python 3.11 venv with the DSP stack): +> - `tests/test_phase2_provider.py` — **22/22 pass**, incl. a MOSS-branch +> integration test that *executes* `_run_interpretation_request_with_profile_config` +> with `ASA_PHASE2_PROVIDER=moss` (mocked sidecar HTTP) and asserts the result +> flows through the shared parse/validate tail (`ok:True`, schema-valid), plus a +> test that the Gemini default never touches the sidecar. +> - `tests/test_server.py` — **223/223 pass**, so the asymmetric branch is +> behavior-preserving for the Gemini path (not just by inspection). The Gemini +> client is still constructed before the `try`, so a client-construction error +> still propagates as `INTERPRETATION_SETUP_FAILED`, exactly as before. +> - The eval runs on all 5 real corpus tracks (mock: 100% schema-valid / 100% +> citation-accuracy / 100% grounding-coverage; Gemini + split correctly +> `BLOCKED`); the sidecar mock + gated 501 stub behave. +> +> **Not run:** the full backend `unittest discover` (the slow Essentia analyzer +> tests are unrelated — no `analyze.py` change, so golden snapshots / +> `EXPECTED_TOP_LEVEL_KEYS` are unaffected) and any live model/Gemini call. + +Goal: add a `Phase2Provider` abstraction so Phase 2 interpretation can be routed to +either Gemini (current, product default) or a self-hosted OpenMOSS **MOSS-Audio** model +run as a FastAPI sidecar — fed the same audio + Phase-1 JSON, emitting the identical +measurement-cited `Phase2Result` schema. Phase 1 stays ground truth; recommendations must +cite, never invent (PURPOSE.md invariant #1/#2). + +The goal made this explicit and first: **"STEP ONE: confirm OpenMOSS/MOSS-Audio code AND +model-weight licences permit self-host + derivative use — if not, stop and report."** + +--- + +## STEP ONE — Licence verification (primary sources, 2026-06-05) + +**Verdict: SPLIT. Weights pass cleanly; the inference code does not.** + +### ✅ Model weights — Apache-2.0 (clean) + +Every official `OpenMOSS-Team/MOSS-Audio-*` weights repo on HuggingFace declares +`license: apache-2.0` in its card metadata, verified directly via the HF API +(`cardData.license`), not via a search summary: + +| Repo | License | Gated | Acceptable-use rider | +|---|---|---|---| +| `OpenMOSS-Team/MOSS-Audio-4B-Instruct` | apache-2.0 | `false` | none (`extra_gated_prompt: null`) | +| `OpenMOSS-Team/MOSS-Audio-8B-Instruct` | apache-2.0 | `false` | none | +| `OpenMOSS-Team/MOSS-Audio-4B-Thinking` | apache-2.0 | `false` | none | +| `OpenMOSS-Team/MOSS-Audio-8B-Thinking` | apache-2.0 | `false` | none | +| `OpenMOSS-Team/MOSS-Audio-Tokenizer` (required at inference) | apache-2.0 | `false` | none | + +No NonCommercial clause, no research-only clause, no click-through gate. The backbone is +Qwen3 (also Apache-2.0). **The weights permit self-host + derivative + commercial use.** + +### ⚠️ Inference / modeling code — no effective licence (UNCONFIRMED) + +The forward-pass code is **not** in the Apache-2.0 HF repo. The HF weights repo ships only +`configuration_moss_audio.py` + `processing_moss_audio.py`, and its `config.json` +`auto_map` declares **only** `AutoConfig` + `AutoProcessor` — **no `AutoModel` entry** +(`architectures: ["MossAudioModel"]`, a custom class with no in-repo definition). The +actual `modeling_moss_audio.py` / `hf_inference.py` / `audio_io.py` live only in the GitHub +repo `OpenMOSS/MOSS-Audio` under `src/`, and: + +1. That GitHub repo has **no LICENSE/COPYING/NOTICE file anywhere in its tree** + (`GET /repos/OpenMOSS/MOSS-Audio/license` → HTTP 404; recursive tree scan → none). +2. Its `pyproject.toml` says `license = { file = "LICENSE" }` — a **dangling reference to a + file that does not exist**. +3. The README's only licence sentence is *"**Models** in MOSS-Audio are licensed under the + Apache License 2.0"* — i.e. the **weights**, not unambiguously the code. +4. The documented run-path is `git clone … && pip install -e . && python infer.py` — i.e. + executing that unlicensed code. `moss_audio` is **not** upstreamed into (Apache-2.0) + `transformers`. + +Absent a licence file, code defaults to **all rights reserved** under copyright. The +dangling `pyproject` reference is the signature of an **oversight** (the intent is clearly +open — Apache-2.0 weights, Qwen3 base), **not** a deliberate restriction like the MSST +separation backend (which is affirmatively AGPL + CC-BY-NC-SA NonCommercial). But as +published, **the code you must run carries no grant.** + +### Consequence + +- **Not promotable to the product Phase 2 path** until OpenMOSS publishes the missing + GitHub LICENSE (or upstreams `moss_audio` into Apache-2.0 `transformers`). +- **Gemini stays the product default.** Treat MOSS exactly like the MSST backend: a + **default-off, isolated research experiment**, documented caveat, never on the product + request path. +- **Clean-runtime unblock path worth verifying:** the Apache-2.0 *weights* could in + principle be served by an independent, cleanly-licensed runtime that does **not** use + OpenMOSS's unlicensed modeling code — e.g. a community GGUF build + (`cstr/MOSS-Audio-4B-Instruct-GGUF`, apache-2.0) via llama.cpp, **if** that runtime + actually supports `moss_audio` audio-**in** inference (unverified — llama.cpp audio-in + for this arch is not confirmed). That path would sidestep the code-licence issue + entirely. + +--- + +## Eval feasibility — the DoD's live numbers are blocked three ways (today) + +The DoD wants a "citation-accuracy eval vs Gemini on a fixed set." All three inputs are +currently unavailable on this machine: + +1. **MOSS** cannot be run without executing the unlicensed modeling code (and is a heavy + 8B/4B download); see above. +2. **`GEMINI_API_KEY` is unset** locally → the Gemini baseline can't be generated live. +3. **The fixed set has no audio.** `apps/backend/tests/fixtures/recommendation_tracks/*` + ships `phase1_fingerprint.json` + `manifest.json` + MIDI, but **no rendered audio** + (it "awaits owner Ableton renders"). An audio-in model has nothing to listen to. + +Honest consequence: the deliverable is the **harness + fixed set + documented procedure + +a programmatic, validator-backed citation scorer**, runnable now against a deterministic +mock/recorded provider, with the live MOSS-vs-Gemini numbers **explicitly stated as +blocked** pending (1) an upstream LICENSE (or clean runtime), (2) a Gemini key, (3) audio +renders. No fabricated comparison numbers. + +--- + +## Decision (maintainer) — **A chosen (2026-06-05)** + +Given the split verdict, how far should the MOSS experiment be built now? (Tracked so the +choice is explicit, mirroring the licence-care shown on the MSST backend.) **The maintainer +chose A** — see "What was built" at the top. + +- **A — Build the default-off research experiment now (recommended). ← CHOSEN.** Provider abstraction + (Gemini default, product-safe), a contract/mock MOSS sidecar emitting the identical + schema, and the citation-accuracy harness — all executing **no** unlicensed code and + fabricating no numbers. The real MOSS model wiring stays a gated, documented stub until + the LICENSE lands. Less encumbered than the MSST backend the repo already ships this way. +- **B — Report only; wait for the upstream LICENSE** before any MOSS-specific code. +- **C — Pursue clarification upstream** (file an issue asking OpenMOSS to add the GitHub + LICENSE) and revisit. + +--- + +## Design (for option A) — provider seam + +- `ASA_PHASE2_PROVIDER=gemini` (default) `| moss`. Default-off; the product path is the + Gemini path, unchanged. +- `phase2_provider.py`: a `Phase2Provider` protocol + `GeminiPhase2Provider` (wraps the + existing `server.py` `_execute_interpretation_attempt` orchestration) + + `MossPhase2Provider` (HTTP client to the sidecar). Resolver keyed on the env var. +- Sidecar (`apps/backend/moss_sidecar/`): standalone FastAPI app, **own venv / own + requirements** (isolation mirrors `separation_backend.py` MSST + MT3), accepts + `{audio, phase1_json}`, reuses `prompts/phase2_system.txt`, returns the identical + `Phase2Result`. Ships a `--mock` deterministic mode so the abstraction + harness are + testable with no model and no GPU. +- **Identical schema** is enforced by routing **both** providers through the existing + validators (`server_phase2._is_valid_phase2_shape`, `_validate_phase2_citation_paths`, + `phase2_catalogue_gates.apply_live12_catalogue_gates`). One schema, one validator, two + providers. +- **Eval** (`scripts/evaluate_phase2_providers.py` + module): fixed + `recommendation_tracks/` set → each provider → citation-accuracy score computed by + reusing `_validate_phase2_citation_paths` (cited `phase1Fields` that resolve vs invented) + → report + the documented "offline-good-enough vs Gemini-wins" split. Research-only. + +When information conflicts: PURPOSE.md > CLAUDE.md. Phase 1 stays ground truth; MOSS, like +Gemini, may interpret but never override a measured value.