From 280f402e22dedcbe05fa126cad08388e6c3336b3 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 5 Jun 2026 20:40:59 +0000 Subject: [PATCH] docs: sync CLAUDE.md and backend AGENTS.md with current codebase MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Four items existed in the repo but were missing from documentation: - `phase2_provider.py` + `moss_sidecar/` (selectable Phase 2 provider, default-off experiment) — added as numbered item #24 in CLAUDE.md backend core files and to the File Map in apps/backend/AGENTS.md - `phase2_provider_evaluation.py` — added to the eval harnesses list (item 16) in CLAUDE.md and the scripts section of AGENTS.md - `evaluate_phase2_providers.py` (the runner script in scripts/) — added to the backend scripts listing in both files - `docs/PHASE2_PROVIDER.md` — added to the docs/ key contents in the repo layout section of CLAUDE.md - `incorporations/msst-separation-licence-gate-2026-06-05.md` — added to the incorporations/ example list in CLAUDE.md https://claude.ai/code/session_0159nkJoHK5shz8USWWpiwrf --- CLAUDE.md | 10 ++++++---- apps/backend/AGENTS.md | 3 ++- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fc4cbffa..bc28af6d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -191,7 +191,7 @@ Operational and one-shot scripts live in two places. They are not on the request `apps/backend/scripts/`: 1. `bootstrap.sh` — recreate the Python 3.11 venv (covered above). `dev.sh` here is a thin shim that `exec`s the repo-root `scripts/dev.sh` full-stack launcher. 2. `render_upload_limit_contract.py` — regenerate the operator-facing upload-limit contract text from `upload_limits.py`. Run after changing the canonical limits. -3. `evaluate_phase1.py`, `evaluate_polyphonic.py`, `evaluate_structure_sweep.py`, `evaluate_beats.py`, `evaluate_loudness_recs.py`, `evaluate_recommendations.py` — offline evaluation harnesses for the Phase 1 detector battery, the research polyphonic transcriber, structure-segmentation parameter sweeps, the beat/downbeat measurement gate, loudness-recommendation reachability, and the recommendation-quality scorer (`GOAL.md`'s recommendation-proof campaign — scores Phase 2 recs against the `tests/fixtures/recommendation_tracks/` answer-key corpus). Wired to `phase1_evaluation.py` / `polyphonic_evaluation.py` / `beat_evaluation.py` / `loudness_rec_evaluation.py` / `recommendation_evaluation.py`. `build_beat_manifest.py` assembles the beat-eval corpus manifest. `emit_deterministic_recs.ts` is the recommendation harness's deterministic-source bridge (Node 23+ native TS; wraps `apps/ui/src/data/abletonDevices.ts` to score the free path). All research-only. The beat gate's optional neural deps (`beat_this`, `mir_eval`) live in `apps/backend/requirements-eval.txt` — install into a separate venv, never the product venv. The optional MSST separation backend follows the same isolation rule but more strictly: `apps/backend/requirements-msst.txt` is a setup-pointer (not a pinned list) because MSST brings its own conflicting torch/numpy/librosa — it lives in its own venv built from MSST-WebUI's own `requirements.txt`, reached only via the `scripts/msst_separate_runner.py` subprocess (see `separation_backend.py`), never imported into the product venv. +3. `evaluate_phase1.py`, `evaluate_polyphonic.py`, `evaluate_structure_sweep.py`, `evaluate_beats.py`, `evaluate_loudness_recs.py`, `evaluate_recommendations.py`, `evaluate_phase2_providers.py` — offline evaluation harnesses for the Phase 1 detector battery, the research polyphonic transcriber, structure-segmentation parameter sweeps, the beat/downbeat measurement gate, loudness-recommendation reachability, the recommendation-quality scorer (`GOAL.md`'s recommendation-proof campaign — scores Phase 2 recs against the `tests/fixtures/recommendation_tracks/` answer-key corpus), and the Phase 2 provider comparison harness (benchmarks Gemini vs MOSS output quality and latency). Wired to `phase1_evaluation.py` / `polyphonic_evaluation.py` / `beat_evaluation.py` / `loudness_rec_evaluation.py` / `recommendation_evaluation.py` / `phase2_provider_evaluation.py`. `build_beat_manifest.py` assembles the beat-eval corpus manifest. `emit_deterministic_recs.ts` is the recommendation harness's deterministic-source bridge (Node 23+ native TS; wraps `apps/ui/src/data/abletonDevices.ts` to score the free path). All research-only. The beat gate's optional neural deps (`beat_this`, `mir_eval`) live in `apps/backend/requirements-eval.txt` — install into a separate venv, never the product venv. The optional MSST separation backend follows the same isolation rule but more strictly: `apps/backend/requirements-msst.txt` is a setup-pointer (not a pinned list) because MSST brings its own conflicting torch/numpy/librosa — it lives in its own venv built from MSST-WebUI's own `requirements.txt`, reached only via the `scripts/msst_separate_runner.py` subprocess (see `separation_backend.py`), never imported into the product venv. 4. `audit_pass1.py`, `genre_check.py`, `replay_catalog_validation.py` — corpus auditing and Live 12 device-catalog validation. `genre_corpus.md` is the corpus manifest. 5. `import_midi_to_ground_truth.py`, `score_polyphonic_clip.py` — corpus-building helpers for the transcription/polyphonic ground-truth fixtures (`tests/fixtures/transcription_tracks/`, `tests/fixtures/polyphonic_tracks/`). Research-only; see those fixtures' READMEs and `docs/LAYER2_EVALUATION.md` / `docs/POLYPHONIC_TRANSCRIPTION_SPIKE.md`. 6. `parity_probe_synth_backends.py` — maintainer probe that renders the same `ClipPlan` through FluidSynth and `symusic.Synthesizer` and reports RMS/peak/spectral-centroid deltas before flipping the `ASA_SAMPLE_SYNTH_BACKEND` auto default. Research/operator-only; not invoked by the runtime. @@ -207,8 +207,8 @@ Operational and one-shot scripts live in two places. They are not on the request 1. `data/` — generated, source-extracted Live 12 catalogue (`live12_catalogue.json` + `live12_catalogue.schema.json`) consumed at runtime by `apps/backend/live12_catalogue.py`. Regenerated via `scripts/build_live12_catalogue.py` from the upstream `gluon/AbletonLive12_MIDIRemoteScripts` checkout. Static device/parameter metadata only; no upstream code is committed. 2. `packages/loudness-spectro-wasm/` — browser-first WebAssembly DSP (ITU-R BS.1770-5 / EBU R128 loudness, A-weighted spectrum, and spectral-reassignment spectrogram). Rust lifted from [openmeters](https://github.com/httpsworldview/openmeters) and compiled via `wasm-bindgen`, GPL-3.0-or-later. **Phase 1: standalone, not yet imported by `apps/ui` or `apps/backend`.** Has its own Cargo workspace, `npm`/`cargo` build, and EBU/ebur128/pyloudnorm validation layers — see [`packages/loudness-spectro-wasm/README.md`](packages/loudness-spectro-wasm/README.md). When integration lands, the canonical Phase 1 LUFS contract still comes from the Essentia path until this is wired in and proven at parity. (The reassigned spectrogram on the product path today comes from librosa via the `reassigned` spectral-enhancement endpoint, not this package.) 3. `audits/` — dated, automated audit reports (e.g. `nightly-2026-05-19.md`, `phase2-recommendation-surface-2026-05-24.md`, `full-review-2026-05-30.md`). Past-tense paper trail; not imported by either app. (Older one-shot advisory deliverables like the phase 1 audit have been archived under `docs/history/phase1-audit/`.) -4. `incorporations/` — planning docs for incorporating upstream projects (e.g. `forking-plans-2026-05-14.md`, `beat-this-measurement-gate-2026-05-20.md`). Planning notes only; not code. -5. `docs/` — long-form rationale and architectural records. Key contents: `ARCHITECTURE_STRATEGY.md` (three-layer design rationale), `SETUP.md` (first-run setup), `LAYER2_EVALUATION.md` and `POLYPHONIC_TRANSCRIPTION_SPIKE.md` (Layer 2 research), `SAMPLE_GENERATION.md` (Phase 3 design), `adr/` (Architecture Decision Records — `0001-phase1-json-schema-v1.md` declares the v1 schema stability contract; `0002-phase1-loudness-units-v2.md` bumped `truePeak` to dBTP and `bpmConfidence` to normalized 0–1, schema now at `phase1.v2`; `0003-recommendations-contract-v1.md` freezes the Phase 2 `recommendations` contract (`recommendations.v1`) — a normalized, schema-validated, citation-gated projection of the device cards), and `history/` (completed plans, one-shot audits, and deliverables — past-tense). Read the relevant doc *before* structural changes; do not treat them as living API docs. +4. `incorporations/` — planning docs for incorporating upstream projects (e.g. `forking-plans-2026-05-14.md`, `beat-this-measurement-gate-2026-05-20.md`, `msst-separation-licence-gate-2026-06-05.md`). Planning notes only; not code. +5. `docs/` — long-form rationale and architectural records. Key contents: `ARCHITECTURE_STRATEGY.md` (three-layer design rationale), `SETUP.md` (first-run setup), `LAYER2_EVALUATION.md` and `POLYPHONIC_TRANSCRIPTION_SPIKE.md` (Layer 2 research), `SAMPLE_GENERATION.md` (Phase 3 design), `PHASE2_PROVIDER.md` (Phase 2 provider selection — Gemini vs MOSS sidecar, licence status, provider contract), `adr/` (Architecture Decision Records — `0001-phase1-json-schema-v1.md` declares the v1 schema stability contract; `0002-phase1-loudness-units-v2.md` bumped `truePeak` to dBTP and `bpmConfidence` to normalized 0–1, schema now at `phase1.v2`; `0003-recommendations-contract-v1.md` freezes the Phase 2 `recommendations` contract (`recommendations.v1`) — a normalized, schema-validated, citation-gated projection of the device cards), and `history/` (completed plans, one-shot audits, and deliverables — past-tense). Read the relevant doc *before* structural changes; do not treat them as living API docs. 6. `tests/ground_truth/` — labeled-corpus fixtures consumed by `scripts/calibrate_confidence.py` (see `tests/ground_truth/README.md`). Not a test suite — each app owns its own (`apps/backend/tests/`, `apps/ui/tests/`). ### Three-Layer Model @@ -283,7 +283,7 @@ Artifact access goes through `artifact_storage.py` rather than direct disk paths 13. **`stage_status.py`**: Collapses the eight internal stage statuses into the additive client-facing `publicStatus` field on every stage snapshot. 14. **`server_samples.py` + `sample_generation.py`, `sample_theory.py`, `sample_synthesis.py`, `sample_drums.py`**: Phase 3 audition-sample generation. `sample_theory.py` builds the PyTheory musical plan, `sample_synthesis.py` renders audio (FluidSynth with sine-additive fallback), `sample_drums.py` synthesizes drum one-shots, `sample_generation.py` orchestrates and emits the citation manifest. On-demand only. 15. **`dsp_bandbank.py` + `dsp_utils.py`**: Shared DSP primitives — `BatchedBandpass` (4th-order Butterworth bandpass bank) and cross-module utility functions. -16. **`phase1_evaluation.py` + `phase1_report_html.py`, `polyphonic_evaluation.py`, `beat_evaluation.py` + `beat_report_html.py`, `loudness_rec_evaluation.py`, `recommendation_evaluation.py`**: Offline evaluation harnesses (deterministic-metric / detector-stability reporting, research-only polyphonic transcription, the beat/downbeat measurement gate that benchmarks CPJKU/beat_this against the shipping kick-accent heuristic, loudness-recommendation reachability, and the recommendation-quality scorer that grades Phase 2 recommendations against known-settings fixtures — see [`apps/backend/NEEDS.md`](apps/backend/NEEDS.md)). Not on the product path; driven by `scripts/evaluate_*.py`. Deleting them restores the product exactly. +16. **`phase1_evaluation.py` + `phase1_report_html.py`, `polyphonic_evaluation.py`, `beat_evaluation.py` + `beat_report_html.py`, `loudness_rec_evaluation.py`, `recommendation_evaluation.py`, `phase2_provider_evaluation.py`**: Offline evaluation harnesses (deterministic-metric / detector-stability reporting, research-only polyphonic transcription, the beat/downbeat measurement gate that benchmarks CPJKU/beat_this against the shipping kick-accent heuristic, loudness-recommendation reachability, the recommendation-quality scorer that grades Phase 2 recommendations against known-settings fixtures, and the Phase 2 provider comparison harness that benchmarks Gemini vs MOSS output quality — see [`apps/backend/NEEDS.md`](apps/backend/NEEDS.md)). Not on the product path; driven by `scripts/evaluate_*.py`. Deleting them restores the product exactly. 17. **`utils/cleanup.py`**: Periodic artifact-cleanup helpers used by the server background-task loop. 18. **`transcription_pianoroll.py`**: Renders the pitch-note translation stage's `transcriptionDetail` as a velocity-encoded `(pitch, time)` uint8 matrix via [`symusic`](https://github.com/Yikai-Liao/symusic). Backs `GET /api/analysis-runs/{run_id}/transcription/pianoroll`. Derived view, never overrides Phase 1; the response cites Phase 1's `bpm` + `timeSignature` so chain of custody is preserved. Note: `transcriptionDetail` is *stripped* from `measurement.result` (see `analysis_runtime.py` ~L800) and lives in the `pitchNoteTranslation` stage instead — the route reads from there. 19. **`mt3_transcription.py`**: Optional polyphonic transcription via Google MT3 (T5X). Additive only — never overrides measurement (PURPOSE.md invariant #1). Gated on the env var `ASA_ENABLE_MT3=1` for the legacy CLI path and the run-level form field `mt3_mode='enabled'` for the canonical staged API. Driven from `_execute_mt3_attempt`/`_mt3_worker_loop` in `server.py`; emits per-stem MIDI as artifacts and an `mt3` namespace on the run snapshot. Heavy dependencies pinned in `requirements-mt3.txt`. See [`JSON_SCHEMA.md` "Optional MT3 Namespace"](apps/backend/JSON_SCHEMA.md#optional-mt3-namespace). @@ -295,6 +295,8 @@ Artifact access goes through `artifact_storage.py` rather than direct disk paths 23. **`recommendations_contract.py` + `schemas/recommendations.v1.schema.json`**: Frozen, versioned Phase 2 recommendation contract (ADR 0003). `project_recommendations` normalizes the three Phase 2 device-card arrays (`abletonRecommendations`, `mixAndMasterChain`, `secretSauce.workflowSteps`) into a flat `{device, parameter, value, unit, range, cited_measurements[]}` envelope (`version: "recommendations.v1"`); `validate_envelope` checks it against the committed JSON Schema via `jsonschema` (the real file, not a hand-rolled mirror — see ADR 0001's drift warning). Derived and additive — it never overrides Phase 1 (invariant #1) and admits ONLY cited cards (`cited_measurements.minItems: 1`); uncited cards stay in the raw arrays where the warn-and-keep catalogue gate flags them. `server.py` attaches the validated envelope to the producer_summary interpretation result as a `recommendations` field (degrades to absent on error). TS mirror: `RecommendationsContract` in [src/types/interpretation.ts](apps/ui/src/types/interpretation.ts). CI gate: `tests/test_recommendations_contract.py` (schema validity + projection-validates + round-trip + freeze). +24. **`phase2_provider.py` + `moss_sidecar/`**: Selectable Phase 2 interpretation provider (default-off experiment). `ASA_PHASE2_PROVIDER=moss` routes the producer_summary interpretation to a self-hosted OpenMOSS MOSS-Audio FastAPI sidecar instead of Gemini, while both paths flow through the identical parse/citation/catalogue validators. STEP ONE licence gate: MOSS-Audio *weights* are Apache-2.0 but the *modeling code* has no effective licence, so the sidecar ships no OpenMOSS code and the real-model path is a 501 stub — research-only, not promotable. `moss_sidecar/app.py` is the FastAPI sidecar; `moss_sidecar/mock_interpreter.py` generates deterministic schema-valid Phase-1-grounded output for the default `ASA_MOSS_SIDECAR_MODE=mock` path. See `docs/PHASE2_PROVIDER.md` for the full design rationale and licence analysis. + The subprocess isolation means `analyze.py` works as a standalone CLI. Check `apps/backend/JSON_SCHEMA.md` before adding new analyzer output fields. Check `apps/backend/ARCHITECTURE.md` for the full HTTP flow and contract details. **Phase 2 (`POST /api/phase2`, legacy compat):** Uploads audio to Gemini inline if ≤100 MiB, or via the Gemini Files API if larger. Phase 1 JSON is appended to the system prompt from `prompts/phase2_system.txt`. Also relevant: `prompts/stem_summary_system.txt` and `prompts/live12_device_catalog.json` (the *prompt-injected* device catalogue — distinct from the runtime-validation `data/live12_catalogue.json` consumed by `phase2_catalogue_gates.py`). Backend defense-in-depth: `server_phase2.py`'s `_validate_phase2_citation_paths` mirrors the frontend citation-existence check and emits `validationWarnings` when a recommendation cites a Phase 1 path that doesn't exist — it flags invented citations rather than failing, since Phase 1 stays authoritative (invariant #1). `phase2_catalogue_gates.apply_live12_catalogue_gates` runs immediately after for source-catalogue checks (see backend file #20 above). diff --git a/apps/backend/AGENTS.md b/apps/backend/AGENTS.md index 97d1e1b3..33799203 100644 --- a/apps/backend/AGENTS.md +++ b/apps/backend/AGENTS.md @@ -131,6 +131,7 @@ python3.11 -m venv venv - `loudness_backend.py`: Selectable Phase 1 loudness backend (default-off experiment, WS3b). `ASA_LOUDNESS_BACKEND=wasm` overrides the four integrated/range/momentary-max/short-term-max LUFS scalars with readings from the native `measure-cli` binary (source-identical to `packages/loudness-spectro-wasm`). `truePeak` and `lufsCurve` stay on Essentia. Any failure degrades back to Essentia silently. Default is `essentia` (no-op). Covered by `tests/test_loudness_backend.py`. - `separation_backend.py`: Selectable Phase 1 stem-separation backend (default-off experiment). `ASA_SEPARATION_BACKEND=msst` swaps torchaudio Hybrid Demucs for MSST/BS-RoFormer from a `SUC-DriverOld/MSST-WebUI` checkout, keeping the `{stem_name: wav_path}` contract (`vocals/bass/drums/other`, 44.1 kHz) unchanged. MSST runs in its own venv via `scripts/msst_separate_runner.py` subprocess under `ASA_MSST_PYTHON`; any failure degrades back to Demucs silently. Default is `demucs` (no-op). See `separation_ab.py` for the research-only A/B harness. Covered by `tests/test_separation_backend.py`. - `separation_ab.py`: Research-only A/B harness for comparing Demucs and MSST separation backends on quality (SI-SDR) and runtime. Driven by `scripts/ab_separation_backends.py`. Deleting this file restores the product exactly. +- `phase2_provider.py` + `moss_sidecar/`: Selectable Phase 2 interpretation provider (default-off experiment). `ASA_PHASE2_PROVIDER=moss` routes the producer_summary to a self-hosted MOSS-Audio sidecar instead of Gemini; both paths run through the same parse/citation/catalogue validators. The real-model path is a 501 licence-gated stub — research-only. See `docs/PHASE2_PROVIDER.md`. Covered by `tests/test_phase2_provider.py`. - `phase2_catalogue_gates.py`: **Warn-and-keep** Live 12 source-catalogue annotation of Phase 2 recommendations. Cross-checks every `{device, parameter, value, phase1Fields}` record against `Live12Catalogue` and emits `RECOMMENDATION_UNVERIFIED` events on `validationWarnings` for `device_unknown`, `parameter_unknown`, `value_out_of_range`, `citation_missing`. NEVER drops or rewrites — an earlier fuzzy-rewrite path produced confidently-wrong output (wrong EQ band + wrong A/B curve), so the contract is warn-only. Wired into `server.py` after `_validate_phase2_citation_paths`. Lives separately from `server_phase2.py` so unit tests can run without the FastAPI/pydantic import chain. - `utils/cleanup.py`: Periodic artifact cleanup helpers used by the server background-task loop. - `tests/test_server.py`: OpenAPI and envelope contract tests. @@ -167,7 +168,7 @@ Under `apps/backend/scripts/` (not on the product path): - `bootstrap.sh`: create/recreate the venv with the pinned Python 3.11 dependencies. - `dev.sh`: convenience shim that `exec`s the monorepo root `./scripts/dev.sh` full-stack launcher. It does not start the backend on its own — the root launcher boots both backend and UI. - `render_upload_limit_contract.py`: re-renders the operator-facing upload-limit contract whenever `upload_limits.py` numbers change. -- `evaluate_phase1.py`, `evaluate_structure_sweep.py`, `evaluate_polyphonic.py`, `evaluate_beats.py`, `evaluate_loudness_recs.py`, `evaluate_recommendations.py`, `build_beat_manifest.py`, `genre_check.py`, `audit_pass1.py`, `replay_catalog_validation.py`: research and audit harnesses for measurement quality, beat/downbeat and loudness-recommendation gates, the recommendation-quality scorer, and prompt-output review. Outputs land under `.runtime/` and are intentionally not wired into the live API. The beat gate's optional neural deps (`beat_this`, `mir_eval`) live in `requirements-eval.txt` — install into a separate venv, never the product venv. +- `evaluate_phase1.py`, `evaluate_structure_sweep.py`, `evaluate_polyphonic.py`, `evaluate_beats.py`, `evaluate_loudness_recs.py`, `evaluate_recommendations.py`, `evaluate_phase2_providers.py`, `build_beat_manifest.py`, `genre_check.py`, `audit_pass1.py`, `replay_catalog_validation.py`: research and audit harnesses for measurement quality, beat/downbeat and loudness-recommendation gates, the recommendation-quality scorer, the Phase 2 provider comparison harness (benchmarks Gemini vs MOSS quality and latency via `phase2_provider_evaluation.py`), and prompt-output review. Outputs land under `.runtime/` and are intentionally not wired into the live API. The beat gate's optional neural deps (`beat_this`, `mir_eval`) live in `requirements-eval.txt` — install into a separate venv, never the product venv. - `emit_deterministic_recs.ts`: Node 23+ native-TS bridge for `evaluate_recommendations.py --source deterministic`. Wraps `apps/ui/src/data/abletonDevices.ts` so the deterministic recommendation path can be scored against the same fixtures as Gemini without a TypeScript build step. No `npm install` needed. - `parity_probe_synth_backends.py`: Maintainer probe that renders the same deterministic `ClipPlan` through both FluidSynth and `symusic.Synthesizer` and reports RMS / peak / spectral-centroid deltas, with a documented tolerance and pass/fail verdict written to `.runtime/parity/synth_parity.json`. Use before flipping the `ASA_SAMPLE_SYNTH_BACKEND` auto default away from FluidSynth — never invoked by the runtime. - `msst_separate_runner.py`: Runs in the MSST venv (not the product venv) under `ASA_MSST_PYTHON`. Invoked by `separation_backend.py` as a subprocess; imports `MSSeparator` from a `SUC-DriverOld/MSST-WebUI` checkout, writes canonical `vocals/bass/drums/other` 44.1 kHz WAVs, and prints a one-line JSON manifest to stdout. Operator tooling only — never imported by the product venv.