feat(backend): pluggable stem-separation backend (Demucs ↔ MSST/BS-RoFormer)

CLAUDE.md

@@ -191,10 +191,12 @@ Operational and one-shot scripts live in two places. They are not on the request
 `apps/backend/scripts/`:
 1. `bootstrap.sh` — recreate the Python 3.11 venv (covered above). `dev.sh` here is a thin shim that `exec`s the repo-root `scripts/dev.sh` full-stack launcher.
 2. `render_upload_limit_contract.py` — regenerate the operator-facing upload-limit contract text from `upload_limits.py`. Run after changing the canonical limits.
-3. `evaluate_phase1.py`, `evaluate_polyphonic.py`, `evaluate_structure_sweep.py`, `evaluate_beats.py`, `evaluate_loudness_recs.py`, `evaluate_recommendations.py` — offline evaluation harnesses for the Phase 1 detector battery, the research polyphonic transcriber, structure-segmentation parameter sweeps, the beat/downbeat measurement gate, loudness-recommendation reachability, and the recommendation-quality scorer (`GOAL.md`'s recommendation-proof campaign — scores Phase 2 recs against the `tests/fixtures/recommendation_tracks/` answer-key corpus). Wired to `phase1_evaluation.py` / `polyphonic_evaluation.py` / `beat_evaluation.py` / `loudness_rec_evaluation.py` / `recommendation_evaluation.py`. `build_beat_manifest.py` assembles the beat-eval corpus manifest. `emit_deterministic_recs.ts` is the recommendation harness's deterministic-source bridge (Node 23+ native TS; wraps `apps/ui/src/data/abletonDevices.ts` to score the free path). All research-only. The beat gate's optional neural deps (`beat_this`, `mir_eval`) live in `apps/backend/requirements-eval.txt` — install into a separate venv, never the product venv.
+3. `evaluate_phase1.py`, `evaluate_polyphonic.py`, `evaluate_structure_sweep.py`, `evaluate_beats.py`, `evaluate_loudness_recs.py`, `evaluate_recommendations.py` — offline evaluation harnesses for the Phase 1 detector battery, the research polyphonic transcriber, structure-segmentation parameter sweeps, the beat/downbeat measurement gate, loudness-recommendation reachability, and the recommendation-quality scorer (`GOAL.md`'s recommendation-proof campaign — scores Phase 2 recs against the `tests/fixtures/recommendation_tracks/` answer-key corpus). Wired to `phase1_evaluation.py` / `polyphonic_evaluation.py` / `beat_evaluation.py` / `loudness_rec_evaluation.py` / `recommendation_evaluation.py`. `build_beat_manifest.py` assembles the beat-eval corpus manifest. `emit_deterministic_recs.ts` is the recommendation harness's deterministic-source bridge (Node 23+ native TS; wraps `apps/ui/src/data/abletonDevices.ts` to score the free path). All research-only. The beat gate's optional neural deps (`beat_this`, `mir_eval`) live in `apps/backend/requirements-eval.txt` — install into a separate venv, never the product venv. The optional MSST separation backend follows the same isolation rule but more strictly: `apps/backend/requirements-msst.txt` is a setup-pointer (not a pinned list) because MSST brings its own conflicting torch/numpy/librosa — it lives in its own venv built from MSST-WebUI's own `requirements.txt`, reached only via the `scripts/msst_separate_runner.py` subprocess (see `separation_backend.py`), never imported into the product venv.
 4. `audit_pass1.py`, `genre_check.py`, `replay_catalog_validation.py` — corpus auditing and Live 12 device-catalog validation. `genre_corpus.md` is the corpus manifest.
 5. `import_midi_to_ground_truth.py`, `score_polyphonic_clip.py` — corpus-building helpers for the transcription/polyphonic ground-truth fixtures (`tests/fixtures/transcription_tracks/`, `tests/fixtures/polyphonic_tracks/`). Research-only; see those fixtures' READMEs and `docs/LAYER2_EVALUATION.md` / `docs/POLYPHONIC_TRANSCRIPTION_SPIKE.md`.
 6. `parity_probe_synth_backends.py` — maintainer probe that renders the same `ClipPlan` through FluidSynth and `symusic.Synthesizer` and reports RMS/peak/spectral-centroid deltas before flipping the `ASA_SAMPLE_SYNTH_BACKEND` auto default. Research/operator-only; not invoked by the runtime.
+7. `msst_separate_runner.py` — the MSST separation runner invoked by `separation_backend.py` as a subprocess under `ASA_MSST_PYTHON`. **Runs in the MSST venv, not the product venv** — it imports `MSSeparator` from a `SUC-DriverOld/MSST-WebUI` checkout, writes canonical `vocals/bass/drums/other` 44.1 kHz WAVs, and prints a single-line JSON manifest to stdout. Operator tooling; not a product-venv module.
+8. `ab_separation_backends.py` — A/B harness CLI (wraps `separation_ab.py`) comparing the Demucs vs MSST separation backends on quality and runtime: an always-available synthetic SI-SDR smoke-test plus optional real-track reference-free proxies. Writes `.runtime/separation_ab/report.json`. Research-only; deleting `separation_ab.py` restores the product exactly.
 
 ## Architecture
 
@@ -289,6 +291,8 @@ Artifact access goes through `artifact_storage.py` rather than direct disk paths
 
 21. **`loudness_backend.py`**: Selectable Phase 1 loudness backend (default-off experiment). `ASA_LOUDNESS_BACKEND=wasm` overrides the four integrated/range/momentary-max/short-term-max LUFS scalars with readings from the native `measure-cli` binary (source-identical to `packages/loudness-spectro-wasm`). `truePeak` and `lufsCurve` always stay on Essentia. Any failure degrades back to Essentia. Default is `essentia` (no-op).
 
+22. **`separation_backend.py`**: Selectable Phase 1 stem-separation backend (default-off experiment). `ASA_SEPARATION_BACKEND=msst` swaps torchaudio Hybrid Demucs (`analyze_audio_io.separate_stems`) for a stronger MSST/BS-RoFormer model from a `SUC-DriverOld/MSST-WebUI` checkout, while keeping the `{stem_name: wav_path}` contract (`vocals/bass/drums/other`, 44.1 kHz) unchanged. Because MSST pins deps that conflict with ASA's (`librosa==0.9.2`, `numpy<2`, its own torch), it runs in its **own** venv and ASA shells out to `scripts/msst_separate_runner.py` under `ASA_MSST_PYTHON` — mirroring how `loudness_backend.py` shells out to `measure-cli`. The subprocess boundary also keeps MSST's stdout/logging off `analyze.py`'s JSON contract (tripwire #1) and isolates its `utils`/`inference` packages. A small model registry (`ASA_MSST_MODEL`, default `scnet_4stem`) maps an id → `{model_type, config, checkpoint}`. Any failure (missing venv/checkout/checkpoint, non-zero exit, unparseable output) degrades back to Demucs. `separate_stems_backend` is the entry point called from `analyze.py`'s three separation sites (measurement `--separate`, `--pitch-note-only`, `--mt3-only`).
+
 The subprocess isolation means `analyze.py` works as a standalone CLI. Check `apps/backend/JSON_SCHEMA.md` before adding new analyzer output fields. Check `apps/backend/ARCHITECTURE.md` for the full HTTP flow and contract details.
 
 **Phase 2 (`POST /api/phase2`, legacy compat):** Uploads audio to Gemini inline if ≤100 MiB, or via the Gemini Files API if larger. Phase 1 JSON is appended to the system prompt from `prompts/phase2_system.txt`. Also relevant: `prompts/stem_summary_system.txt` and `prompts/live12_device_catalog.json` (the *prompt-injected* device catalogue — distinct from the runtime-validation `data/live12_catalogue.json` consumed by `phase2_catalogue_gates.py`). Backend defense-in-depth: `server_phase2.py`'s `_validate_phase2_citation_paths` mirrors the frontend citation-existence check and emits `validationWarnings` when a recommendation cites a Phase 1 path that doesn't exist — it flags invented citations rather than failing, since Phase 1 stays authoritative (invariant #1). `phase2_catalogue_gates.apply_live12_catalogue_gates` runs immediately after for source-catalogue checks (see backend file #20 above).
@@ -353,6 +357,12 @@ ASA_ENABLE_MT3="0"                       # set to "1" on the legacy analyze.py C
 ASA_SAMPLE_SYNTH_BACKEND="auto"          # Phase 3 audition-sample synth backend: `auto` (default, prefers FluidSynth and falls back to symusic), `symusic`, or `fluidsynth` to pin one. See apps/backend/sample_synthesis.py.
 ASA_LOUDNESS_BACKEND="essentia"          # Phase 1 loudness source for the LUFS scalars: `essentia` (default, authoritative) or `wasm` to override lufsIntegrated/lufsRange/lufsMomentaryMax/lufsShortTermMax with the asa-dsp (loudness-spectro-wasm) reading via the native measure-cli binary. truePeak + lufsCurve stay Essentia; degrades back to Essentia if measure-cli is unbuilt. Default-off experiment. See apps/backend/loudness_backend.py.
 ASA_MEASURE_CLI=""                       # optional absolute path to the measure-cli binary used by ASA_LOUDNESS_BACKEND=wasm; defaults to packages/loudness-spectro-wasm/target/release/measure-cli.
+ASA_SEPARATION_BACKEND="demucs"          # Phase 1 stem-separation backend: `demucs` (default, torchaudio Hybrid Demucs) or `msst` to drive a stronger MSST/BS-RoFormer model from a SUC-DriverOld/MSST-WebUI checkout. The stems contract (vocals/bass/drums/other WAVs) is unchanged. Runs MSST as a subprocess under its own venv (see ASA_MSST_*); any failure degrades back to Demucs. Default-off experiment. See apps/backend/separation_backend.py + requirements-msst.txt.
+ASA_MSST_PYTHON=""                       # path to the MSST venv interpreter (built from MSST-WebUI's own requirements.txt) used to run scripts/msst_separate_runner.py. Required when ASA_SEPARATION_BACKEND=msst.
+ASA_MSST_ROOT=""                         # path to the MSST-WebUI checkout (added to the runner's sys.path for `from inference.msst_infer import MSSeparator`). Required when ASA_SEPARATION_BACKEND=msst.
+ASA_MSST_MODEL="scnet_4stem"             # MSST model registry id in separation_backend._MSST_MODEL_REGISTRY. Default `scnet_4stem` (4-stem parity). `bs_roformer_vocals` is research/A-B-only (2-stem, leaves bass/drums empty). Unknown ids fall back to the default.
+ASA_MSST_MODEL_DIR=""                    # optional config/checkpoint root (MSST configs/ + pretrain/ layout); defaults to ASA_MSST_ROOT.
+ASA_MSST_DEVICE=""                       # optional device for MSST inference (auto|cpu|cuda|mps); defaults to auto.
 ```
 
 Phase 2 is gated by `VITE_ENABLE_PHASE2_GEMINI`. `GEMINI_API_KEY` is backend-only. `SONIC_ANALYZER_ADMIN_KEY` is backend-only and never exposed to clients.

apps/backend/JSON_SCHEMA.md

@@ -274,7 +274,9 @@ Sibling time-series partner for `spectralBalance`. Each row carries all seven ba
 
 Type: `object \| null`
 
-Phase 1.B per-stem analytical surface — populated only when Demucs stem separation ran successfully (`--separate`). Null when separation wasn't requested or failed. Phase 2 cites individual stems for element-specific recommendations.
+Phase 1.B per-stem analytical surface — populated only when stem separation ran successfully (`--separate`). Null when separation wasn't requested or failed. Phase 2 cites individual stems for element-specific recommendations.
+
+The separation backend is **selectable** (`ASA_SEPARATION_BACKEND`, default `demucs`; `msst` drives a stronger MSST/BS-RoFormer model — see `separation_backend.py`), but this schema is **backend-agnostic**: stems are always the canonical `drums`/`bass`/`other`/`vocals` at 44.1 kHz, so `stemAnalysis` is unchanged regardless of which backend produced them.
 
 For each available stem (`drums` / `bass` / `other` / `vocals`), the same high-value full-mix analyzers run on the stem's audio:
 

apps/backend/analyze.py

@@ -61,10 +61,10 @@
     _demucs_chunked_inference,
     _load_stem_mono,
     _load_stem_stereo,
-    separate_stems,
     analyze_crepe_pitch,
     cleanup_stems,
 )
+from separation_backend import separate_stems_backend  # noqa: E402
 from analyze_estimate import (  # noqa: E402
     _format_duration_label,
     _estimate_stage_seconds,
@@ -1162,7 +1162,7 @@ def _run_pitch_note_translation(
 
     if need_separation:
         temp_dir = stem_output_dir or tempfile.mkdtemp(prefix="asa_pitch_note_stems_")
-        separated = separate_stems(audio_path, output_dir=temp_dir)
+        separated = separate_stems_backend(audio_path, output_dir=temp_dir)
         if isinstance(separated, dict) and separated:
             stem_paths = separated
 
@@ -1224,9 +1224,9 @@ def _run_mt3_transcription(
     # handover convention so both stages can short-circuit Demucs.
     if stems_dir_path is None and stem_output_dir is not None:
         os.makedirs(stem_output_dir, exist_ok=True)
-        separated = separate_stems(audio_path, output_dir=stem_output_dir)
+        separated = separate_stems_backend(audio_path, output_dir=stem_output_dir)
         if isinstance(separated, dict) and separated:
-            # separate_stems returns {"bass": "/path/...wav", ...}. Recover
+            # separate_stems_backend returns {"bass": "/path/...wav", ...}. Recover
             # the shared parent — all stem files live in one directory.
             for stem_path in separated.values():
                 if isinstance(stem_path, str) and os.path.isfile(stem_path):
@@ -1555,7 +1555,7 @@ def main():
             "Running source separation (this may take 30-60 seconds)...",
             file=sys.stderr,
         )
-        stems = separate_stems(audio_path)
+        stems = separate_stems_backend(audio_path)
         print("@@SEPARATION_COMPLETE", file=sys.stderr)
 
     # Run torchcrepe pitch extraction on separated stems (if available)

apps/backend/requirements-msst.txt

@@ -0,0 +1,41 @@
+# Optional MSST / BS-RoFormer separation backend.
+#
+# These deps are NOT part of the product baseline and must NOT be added to
+# requirements.txt or installed into the product venv. The base ASA install
+# runs separation on torchaudio Hybrid Demucs (analyze_audio_io.separate_stems);
+# the MSST backend is a default-off experiment selected by ASA_SEPARATION_BACKEND=msst.
+#
+# Unlike requirements-mt3.txt / requirements-eval.txt, this file does NOT pin a
+# package list. MSST-WebUI pins OLDER, conflicting versions of ASA's own deps
+# (librosa==0.9.2 vs ASA 0.11.0, numpy<2 vs 2.4.3, protobuf==3.20.3 vs 7.34.0,
+# demucs==4.0.0, transformers~=4.35, lightning, …) and brings its own torch.
+# They cannot coexist with the product venv. So the MSST backend runs in a
+# SEPARATE venv built from MSST-WebUI's OWN requirements.txt, and ASA shells out
+# to it as a subprocess (apps/backend/scripts/msst_separate_runner.py) — exactly
+# how loudness_backend.py shells out to measure-cli. ASA never imports MSST.
+#
+# Setup:
+#
+#   # 1. Clone MSST-WebUI and create its own venv from ITS requirements.
+#   git clone https://github.com/SUC-DriverOld/MSST-WebUI.git /opt/MSST-WebUI
+#   python3.11 -m venv /opt/MSST-WebUI/venv
+#   /opt/MSST-WebUI/venv/bin/pip install -r /opt/MSST-WebUI/requirements.txt
+#
+#   # 2. Download a 4-stem checkpoint + its config from the Sucial/MSST-WebUI
+#   #    Hugging Face repo into the MSST layout (configs/ + pretrain/). The
+#   #    default model in separation_backend._MSST_MODEL_REGISTRY ('scnet_4stem')
+#   #    expects:
+#   #      pretrain/multi_stem_models/model_scnet_sdr_9.3244.ckpt
+#   #      configs/multi_stem_models/config_musdb18_scnet.yaml
+#
+#   # 3. Point ASA at the checkout + interpreter and select the backend:
+#   export ASA_SEPARATION_BACKEND=msst
+#   export ASA_MSST_PYTHON=/opt/MSST-WebUI/venv/bin/python
+#   export ASA_MSST_ROOT=/opt/MSST-WebUI
+#   # optional:
+#   #   ASA_MSST_MODEL_DIR  (config/checkpoint root; defaults to ASA_MSST_ROOT)
+#   #   ASA_MSST_MODEL      (registry id; defaults to 'scnet_4stem')
+#   #   ASA_MSST_DEVICE     (auto|cpu|cuda|mps; default auto)
+#
+# Any failure (missing venv/checkout/checkpoint, non-zero exit, unparseable
+# output) degrades gracefully back to Demucs — see separation_backend.py.

apps/backend/scripts/ab_separation_backends.py

@@ -0,0 +1,106 @@
+#!/usr/bin/env python3
+"""A/B separation backends CLI — EVAL / RESEARCH ONLY.
+
+Thin wrapper over ``separation_ab.run_ab`` (like ``scripts/evaluate_*.py`` wrap
+their ``*_evaluation.py`` modules). Compares the default Demucs backend against
+the optional MSST backend (``ASA_SEPARATION_BACKEND=msst``) on separation quality
+and runtime, writes a JSON report, and prints a compact table.
+
+The synthetic smoke-test always runs. The MSST column is filled only when
+``ASA_MSST_PYTHON`` + ``ASA_MSST_ROOT`` are configured (otherwise reported as
+``skipped_no_msst``). This is observational, not a gate — exit code is 0 for any
+completed/skipped run, 1 only on a hard error.
+
+Usage::
+
+    ./venv/bin/python scripts/ab_separation_backends.py \\
+        [-i /dir/of/real/tracks] [-o .runtime/separation_ab] [--model scnet_4stem]
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+REPO_DIR = Path(__file__).resolve().parent.parent
+if str(REPO_DIR) not in sys.path:
+    sys.path.insert(0, str(REPO_DIR))
+
+import separation_ab  # noqa: E402
+
+
+def _fmt(value: object) -> str:
+    return "—" if value is None else str(value)
+
+
+def _print_table(report: dict) -> None:
+    print("\nSeparation A/B — synthetic smoke-test (NOT a real-music quality ranking)")
+    print(f"{'backend':<10} {'status':<16} {'meanSI-SDR(dB)':>15} {'runtime(s)':>11} {'device':>8}")
+    synthetic = report.get("syntheticSmokeTest", {}).get("perBackend", {})
+    for backend in ("demucs", "msst"):
+        block = synthetic.get(backend, {})
+        status = block.get("status", "—")
+        quality = block.get("quality", {}) if isinstance(block.get("quality"), dict) else {}
+        mean = quality.get("meanSiSdrDb")
+        runtime = block.get("runtimeSeconds")
+        device = block.get("device")
+        print(f"{backend:<10} {status:<16} {_fmt(mean):>15} {_fmt(runtime):>11} {_fmt(device):>8}")
+
+    real = report.get("realTracks", [])
+    if real:
+        print(f"\nReal-track reference-free proxies ({len(real)} track(s)):")
+        for entry in real:
+            print(f"  {entry.get('track')}:")
+            for backend in ("demucs", "msst"):
+                block = entry.get("perBackend", {}).get(backend, {})
+                status = block.get("status", "—")
+                runtime = block.get("runtimeSeconds")
+                residual = block.get("mixReconstructionResidualDb")
+                print(
+                    f"    {backend:<8} status={status:<16} "
+                    f"runtime={_fmt(runtime)}s reconResidual={_fmt(residual)}dB"
+                )
+    for caveat in report.get("caveats", []):
+        print(f"\n[caveat] {caveat}")
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "-i", "--input-dir", default=None,
+        help="Optional directory of real audio tracks for reference-free proxies.",
+    )
+    parser.add_argument(
+        "-o", "--out-dir", type=Path, default=REPO_DIR / ".runtime" / "separation_ab",
+        help="Report output directory. Defaults to apps/backend/.runtime/separation_ab/.",
+    )
+    parser.add_argument(
+        "--model", default=None,
+        help="MSST model registry id (sets ASA_MSST_MODEL). Defaults to scnet_4stem.",
+    )
+    parser.add_argument("--repeats", type=int, default=2, help="Timed runs per backend (min wins).")
+    args = parser.parse_args()
+
+    try:
+        report = separation_ab.run_ab(
+            input_dir=args.input_dir,
+            out_dir=str(args.out_dir),
+            model=args.model,
+            repeats=args.repeats,
+        )
+    except Exception as exc:  # noqa: BLE001
+        print(json.dumps({"status": "error", "error": str(exc)}, indent=2))
+        return 1
+
+    args.out_dir.mkdir(parents=True, exist_ok=True)
+    report_path = args.out_dir / "report.json"
+    report_path.write_text(json.dumps(report, indent=2), encoding="utf-8")
+    _print_table(report)
+    print(f"\nWrote {report_path}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())