From efa96ef3cca84371734bd2b7001e3b78f7fdeead Mon Sep 17 00:00:00 2001 From: slitty-codes Date: Fri, 5 Jun 2026 18:45:36 +1200 Subject: [PATCH] feat(backend): make MSST separation run for real + ground-truth SDR A/B + licence gate Follows PR #141's pluggable separation backend. Confirms the licence position (the campaign's stated prerequisite), makes the MSST runner work end-to-end, and adds a ground-truth-stems SDR mode so Demucs-vs-MSST can be measured on real isolated stems. Default path untouched: ASA_SEPARATION_BACKEND still defaults to demucs; analyze.py's JSON contract is unchanged. Licence gate (previously unconfirmed): - MSST-WebUI code is AGPL-3.0; both registry checkpoints (scnet_4stem, bs_roformer_vocals) are CC-BY-NC-SA-4.0 (NonCommercial). That is a strictly worse posture than the (MUSDB-grey) Demucs incumbent, so the MSST backend is research / NonCommercial-only and NOT promotable to a commercial default -- promotion is licence-gated, not quality-gated. - Durable in-code guardrails: LICENCE GATE notes on _MSST_MODEL_REGISTRY and in requirements-msst.txt. Full map + rationale + A/B results: incorporations/msst-separation-licence-gate-2026-06-05.md msst_separate_runner.py -- 3 integration fixes (never exercised before; PR #141 had no MSST install): - chdir into the checkout (MSSeparator import reads data_backup/webui_config.json via a relative path); caller paths resolved to absolute first so they survive. - drop the injected stdlib logger (MSSeparator relies on MSST's own get_logger with a console_handler attribute); its handler targets stderr and the existing redirect_stdout keeps the stdout JSON contract clean. - sample-rate read: regex fallback for configs carrying !!python/tuple tags. separation_ab.py -- ground-truth reference-set mode (--ref-dir): loads MUSDB-style {mixture,vocals,bass,drums,other}.wav, runs each backend, scores true per-stem SI-SDR + aggregate; warmup-skip for the subprocess MSST backend (it reloads per call). New tests/test_separation_ab.py (loader + aggregation + si_sdr). Existing separation tests stay green. NonCommercial research A/B (5 MUSDB18 test tracks, 7s, CPU, mono gain-aligned SI-SDR -- a preliminary proxy, NOT museval): Demucs 8.08 dB @1.3s vs scnet_4stem 6.22 dB @40.8s (~32x slower on CPU). MSST wins vocals (10.77 vs 8.62) but trails overall; this contradicts published museval SDR, so the short-clip proxy is the limiter -- a full-length re-run is the honest basis for any quality claim. The gate conclusion is unchanged either way: do not promote. Co-Authored-By: Claude Opus 4.8 (1M context) --- apps/backend/requirements-msst.txt | 11 ++ .../backend/scripts/ab_separation_backends.py | 39 +++++ apps/backend/scripts/msst_separate_runner.py | 65 +++++--- apps/backend/separation_ab.py | 156 +++++++++++++++++- apps/backend/separation_backend.py | 12 ++ apps/backend/tests/test_separation_ab.py | 109 ++++++++++++ ...msst-separation-licence-gate-2026-06-05.md | 90 ++++++++++ 7 files changed, 453 insertions(+), 29 deletions(-) create mode 100644 apps/backend/tests/test_separation_ab.py create mode 100644 incorporations/msst-separation-licence-gate-2026-06-05.md diff --git a/apps/backend/requirements-msst.txt b/apps/backend/requirements-msst.txt index b5a3c7da..6f17494c 100644 --- a/apps/backend/requirements-msst.txt +++ b/apps/backend/requirements-msst.txt @@ -5,6 +5,17 @@ # runs separation on torchaudio Hybrid Demucs (analyze_audio_io.separate_stems); # the MSST backend is a default-off experiment selected by ASA_SEPARATION_BACKEND=msst. # +# LICENCE GATE (read before enabling, and especially before any promotion): +# - MSST-WebUI code is AGPL-3.0. ASA ships none of it and never imports it; the +# operator installs it into a SEPARATE venv and ASA shells out to it. That +# arms-length subprocess boundary is what keeps ASA's MIT licence intact, so +# it must stay a subprocess — never vendored or imported. +# - The model CHECKPOINTS (Sucial/MSST-WebUI HF repo, incl. the registry's +# scnet_4stem + bs_roformer_vocals) are CC-BY-NC-SA-4.0 — NonCommercial. +# => This backend is research / personal / NonCommercial-only and is NOT +# promotable to a commercial default. Full map + rationale: +# incorporations/msst-separation-licence-gate-2026-06-05.md +# # Unlike requirements-mt3.txt / requirements-eval.txt, this file does NOT pin a # package list. MSST-WebUI pins OLDER, conflicting versions of ASA's own deps # (librosa==0.9.2 vs ASA 0.11.0, numpy<2 vs 2.4.3, protobuf==3.20.3 vs 7.34.0, diff --git a/apps/backend/scripts/ab_separation_backends.py b/apps/backend/scripts/ab_separation_backends.py index 4a95f232..90aa20a9 100644 --- a/apps/backend/scripts/ab_separation_backends.py +++ b/apps/backend/scripts/ab_separation_backends.py @@ -35,7 +35,40 @@ def _fmt(value: object) -> str: return "—" if value is None else str(value) +def _print_reference_table(report: dict) -> None: + ref = report.get("referenceSet") + if not isinstance(ref, dict): + return + print( + f"\nReference-set SI-SDR — HEADLINE A/B ({ref.get('trackCount')} track(s), " + f"ground-truth stems, mono gain-aligned SI-SDR; NOT museval-comparable)" + ) + aggregate = ref.get("aggregate", {}) if isinstance(ref.get("aggregate"), dict) else {} + print(f"{'backend':<10} {'meanSI-SDR(dB)':>15} {'meanRuntime(s)':>15} {'tracksScored':>13}") + for backend in ("demucs", "msst"): + block = aggregate.get(backend, {}) if isinstance(aggregate.get(backend), dict) else {} + print( + f"{backend:<10} {_fmt(block.get('meanSiSdrDb')):>15} " + f"{_fmt(block.get('meanRuntimeSeconds')):>15} {_fmt(block.get('tracksScored')):>13}" + ) + # Per-stem aggregate, demucs vs msst side by side. + print(f"\n per-stem meanSI-SDR(dB): {'stem':<8} {'demucs':>10} {'msst':>10}") + d_stems = aggregate.get("demucs", {}).get("perStemMeanSiSdrDb", {}) if isinstance(aggregate.get("demucs"), dict) else {} + m_stems = aggregate.get("msst", {}).get("perStemMeanSiSdrDb", {}) if isinstance(aggregate.get("msst"), dict) else {} + for stem in ("vocals", "bass", "drums", "other"): + print(f" {'':<24} {stem:<8} {_fmt(d_stems.get(stem)):>10} {_fmt(m_stems.get(stem)):>10}") + # Per-track headline (mean SI-SDR) so per-song wins/losses are visible. + print("\n per-track meanSI-SDR(dB):") + for track in ref.get("tracks", []): + per = track.get("perBackend", {}) + d = per.get("demucs", {}).get("quality", {}).get("meanSiSdrDb") if isinstance(per.get("demucs"), dict) else None + m = per.get("msst", {}).get("quality", {}).get("meanSiSdrDb") if isinstance(per.get("msst"), dict) else None + status = "" if (d is not None or m is not None) else f" ({track.get('status','')})" + print(f" {str(track.get('track'))[:36]:<38} demucs={_fmt(d):>8} msst={_fmt(m):>8}{status}") + + def _print_table(report: dict) -> None: + _print_reference_table(report) print("\nSeparation A/B — synthetic smoke-test (NOT a real-music quality ranking)") print(f"{'backend':<10} {'status':<16} {'meanSI-SDR(dB)':>15} {'runtime(s)':>11} {'device':>8}") synthetic = report.get("syntheticSmokeTest", {}).get("perBackend", {}) @@ -72,6 +105,11 @@ def main() -> int: "-i", "--input-dir", default=None, help="Optional directory of real audio tracks for reference-free proxies.", ) + parser.add_argument( + "-r", "--ref-dir", default=None, + help="Optional reference set: a dir of /{mixture,vocals,bass,drums,other}.wav " + "with ground-truth stems, scored as true per-stem SI-SDR (the headline A/B signal).", + ) parser.add_argument( "-o", "--out-dir", type=Path, default=REPO_DIR / ".runtime" / "separation_ab", help="Report output directory. Defaults to apps/backend/.runtime/separation_ab/.", @@ -89,6 +127,7 @@ def main() -> int: out_dir=str(args.out_dir), model=args.model, repeats=args.repeats, + ref_dir=args.ref_dir, ) except Exception as exc: # noqa: BLE001 print(json.dumps({"status": "error", "error": str(exc)}, indent=2)) diff --git a/apps/backend/scripts/msst_separate_runner.py b/apps/backend/scripts/msst_separate_runner.py index 28f5f2aa..6af36642 100644 --- a/apps/backend/scripts/msst_separate_runner.py +++ b/apps/backend/scripts/msst_separate_runner.py @@ -32,7 +32,6 @@ import argparse import contextlib import json -import logging import os import sys import time @@ -53,29 +52,29 @@ def _eprint(message: str) -> None: print(message, file=sys.stderr) -def _stderr_logger() -> logging.Logger: - """A logger that writes to stderr only, so MSST never pollutes stdout.""" - logger = logging.getLogger("asa_msst_runner") - logger.handlers.clear() - handler = logging.StreamHandler(sys.stderr) - handler.setFormatter(logging.Formatter("[msst] %(message)s")) - logger.addHandler(handler) - logger.setLevel(logging.INFO) - logger.propagate = False - return logger - - def _read_target_sample_rate(config_path: str) -> int: - """Best-effort read of ``audio.sample_rate`` from the MSST config YAML.""" - try: - import yaml + """Best-effort read of ``audio.sample_rate`` from the MSST config YAML. + MSST configs can carry ``!!python/tuple`` tags that ``yaml.safe_load`` + rejects, so fall back to a line regex before defaulting to 44.1 kHz. + """ + try: with open(config_path, "r") as handle: - config = yaml.safe_load(handle) - audio = (config or {}).get("audio", {}) if isinstance(config, dict) else {} - sr = audio.get("sample_rate") - if isinstance(sr, int) and sr > 0: - return sr + text = handle.read() + try: + import yaml + + config = yaml.safe_load(text) + audio = (config or {}).get("audio", {}) if isinstance(config, dict) else {} + sr = audio.get("sample_rate") + if isinstance(sr, int) and sr > 0: + return sr + except Exception: # noqa: BLE001 - tagged YAML; fall back to a line regex + import re + + match = re.search(r"^\s*sample_rate:\s*(\d+)", text, re.MULTILINE) + if match: + return int(match.group(1)) except Exception as exc: # noqa: BLE001 _eprint(f"[msst] could not read sample_rate from config ({exc}); using 44100.") return _TARGET_SAMPLE_RATE_FALLBACK @@ -153,14 +152,30 @@ def main() -> int: parser.add_argument("--device", default="auto") args = parser.parse_args() + # Resolve every caller-supplied path to absolute BEFORE we chdir below, so + # they stay valid once cwd moves into the MSST checkout. + args.input = os.path.abspath(args.input) + args.output_dir = os.path.abspath(args.output_dir) + args.config = os.path.abspath(args.config) + args.checkpoint = os.path.abspath(args.checkpoint) + # Append (not prepend) the MSST checkout so its top-level ``utils`` / # ``inference`` packages don't shadow anything in this process unexpectedly. if args.msst_root not in sys.path: sys.path.append(args.msst_root) + # MSST reads some of its own files via paths RELATIVE to the checkout root + # (e.g. utils.constant loads ``data_backup/webui_config.json`` at import + # time), so the import fails unless cwd is the checkout. Run from there; the + # caller paths were just made absolute, so they survive the chdir. + try: + os.chdir(args.msst_root) + except OSError as exc: + _eprint(f"[msst] could not chdir to msst-root {args.msst_root} ({exc}).") + return 2 + os.makedirs(args.output_dir, exist_ok=True) target_sr = _read_target_sample_rate(args.config) - logger = _stderr_logger() # Force every MSST print()/tqdm/log onto stderr — stdout is the JSON contract. with contextlib.redirect_stdout(sys.stderr): @@ -180,7 +195,11 @@ def main() -> int: device=args.device, output_format="wav", store_dirs="", # we write stems ourselves; don't let MSST emit files - logger=logger, + # No custom logger: MSSeparator builds its own via MSST's + # utils.logger.get_logger() (it relies on a non-stdlib + # ``console_handler`` attribute). Its handler targets stderr and + # the redirect_stdout above catches any stray stdout, so the + # stdout JSON contract stays clean. ) load_seconds = time.perf_counter() - load_start diff --git a/apps/backend/separation_ab.py b/apps/backend/separation_ab.py index 92452b34..e4e6b0ee 100644 --- a/apps/backend/separation_ab.py +++ b/apps/backend/separation_ab.py @@ -169,11 +169,16 @@ def run_backend_separation( out_dir: str, *, repeats: int = 2, + warmup: bool = True, ) -> dict[str, Any]: """Separate ``mix_path`` with ``backend`` (demucs|msst); time it fairly. - Does a warm-up pass (amortizes model download/load), then takes the min - wall-clock over ``repeats`` timed runs. Returns the stems + runtime + status. + With ``warmup`` (default), does an untimed warm-up pass (amortizes model + download/load) then takes the min wall-clock over ``repeats`` timed runs. + Set ``warmup=False`` for the subprocess MSST backend, which reloads its model + on every call — a warm-up there buys nothing and just doubles a slow run, so + the single timed run honestly includes the per-call cold cost both backends + actually pay. Returns the stems + runtime + status. """ from separation_backend import separate_stems_backend @@ -182,9 +187,10 @@ def run_backend_separation( try: device = "cuda" if _torch_cuda_available() else "cpu" # Warm-up (untimed) — pulls weights / builds the graph. - warm = separate_stems_backend(mix_path, output_dir=os.path.join(out_dir, "warmup")) - if not warm: - return {"status": "error", "error": "separation returned no stems", "stems": {}} + if warmup: + warm = separate_stems_backend(mix_path, output_dir=os.path.join(out_dir, "warmup")) + if not warm: + return {"status": "error", "error": "separation returned no stems", "stems": {}} timings: list[float] = [] stems: dict[str, str] = {} @@ -335,14 +341,143 @@ def _reconstruction_residual_db(mix: np.ndarray | None, recovered: np.ndarray | return round(10.0 * math.log10(max(res_energy, 1e-12) / mix_energy), 3) +# --------------------------------------------------------------------------- # +# Ground-truth reference-set evaluation (true per-stem SI-SDR) +# --------------------------------------------------------------------------- # +# A "reference set" is a directory of per-track subdirectories, each laid out +# MUSDB-style with WAV files: +# +# //mixture.wav (the input the backend separates) +# //vocals.wav (ground-truth stems, any subset of the +# //bass.wav canonical four; missing stems are skipped) +# //drums.wav +# //other.wav +# +# Unlike the synthetic smoke-test (toy sources), this scores each backend on +# REAL music with REAL isolated stems, so the SI-SDR is a genuine quality +# signal for a Demucs-vs-MSST comparison. Two honesty caveats remain: +# * It is gain-aligned SI-SDR on a mono downmix, NOT museval/BSSEval-v4 SDR — +# so the absolute numbers are NOT comparable to published MUSDB leaderboards. +# They ARE internally consistent (same metric, same tracks, both backends), +# which is exactly what an A/B needs. +# * WAV only (reuses _load_wav_mono); the MUSDB extraction writes WAVs. +_MIX_BASENAMES = ("mixture", "mix") + + +def _load_reference_track(track_dir: str) -> tuple[str, dict[str, np.ndarray]] | None: + """Load a MUSDB-style track dir: ``(mixture_path, {stem: mono ground-truth})``. + + Returns ``None`` if no mixture file or no ground-truth stems are present. + """ + mixture_path: str | None = None + for base in _MIX_BASENAMES: + candidate = os.path.join(track_dir, f"{base}.wav") + if os.path.isfile(candidate): + mixture_path = candidate + break + if mixture_path is None: + return None + + known: dict[str, np.ndarray] = {} + for stem in _CANONICAL_STEMS: + stem_path = os.path.join(track_dir, f"{stem}.wav") + if os.path.isfile(stem_path): + mono = _load_wav_mono(stem_path) + if mono is not None: + known[stem] = mono + if not known: + return None + return mixture_path, known + + +def evaluate_reference_track(track_dir: str, out_dir: str, *, repeats: int = 2) -> dict[str, Any]: + """True per-stem SI-SDR + runtime for both backends on one ground-truth track.""" + name = os.path.basename(os.path.normpath(track_dir)) + loaded = _load_reference_track(track_dir) + if loaded is None: + return {"track": name, "status": "skipped_no_reference"} + mixture_path, known = loaded + + work = tempfile.mkdtemp(prefix="asa_sep_ab_ref_", dir=out_dir) + entry: dict[str, Any] = {"track": name, "perBackend": {}} + for backend in ("demucs", "msst"): + if backend == "msst" and not _msst_configured(): + entry["perBackend"][backend] = {"status": "skipped_no_msst"} + continue + backend_dir = os.path.join(work, backend) + os.makedirs(backend_dir, exist_ok=True) + # warmup=False: one honest cold-start timed run (MSST reloads per call). + run = run_backend_separation( + backend, mixture_path, backend_dir, repeats=repeats, warmup=False + ) + if run["status"] != "completed": + entry["perBackend"][backend] = run + continue + entry["perBackend"][backend] = { + "status": "completed", + "runtimeSeconds": run["runtimeSeconds"], + "device": run["device"], + "quality": _score_against_known(run["stems"], known), + } + return entry + + +def _aggregate_reference(tracks: list[dict[str, Any]]) -> dict[str, Any]: + """Per-backend mean SI-SDR (overall + per stem) and mean runtime across tracks.""" + aggregate: dict[str, Any] = {} + for backend in ("demucs", "msst"): + means: list[float] = [] + runtimes: list[float] = [] + per_stem: dict[str, list[float]] = {stem: [] for stem in _CANONICAL_STEMS} + for track in tracks: + block = track.get("perBackend", {}).get(backend, {}) + if block.get("status") != "completed": + continue + quality = block.get("quality", {}) if isinstance(block.get("quality"), dict) else {} + if isinstance(quality.get("meanSiSdrDb"), (int, float)): + means.append(float(quality["meanSiSdrDb"])) + for stem in _CANONICAL_STEMS: + value = quality.get(stem, {}).get("siSdrDb") if isinstance(quality.get(stem), dict) else None + if isinstance(value, (int, float)): + per_stem[stem].append(float(value)) + if isinstance(block.get("runtimeSeconds"), (int, float)): + runtimes.append(float(block["runtimeSeconds"])) + aggregate[backend] = { + "meanSiSdrDb": round(sum(means) / len(means), 3) if means else None, + "perStemMeanSiSdrDb": { + stem: (round(sum(v) / len(v), 3) if v else None) for stem, v in per_stem.items() + }, + "meanRuntimeSeconds": round(sum(runtimes) / len(runtimes), 3) if runtimes else None, + "tracksScored": len(means), + } + return aggregate + + +def evaluate_reference_set(ref_dir: str, out_dir: str, *, repeats: int = 2) -> dict[str, Any]: + """Evaluate every track subdirectory under ``ref_dir`` and aggregate.""" + track_dirs = sorted( + os.path.join(ref_dir, name) + for name in os.listdir(ref_dir) + if os.path.isdir(os.path.join(ref_dir, name)) + ) + tracks = [evaluate_reference_track(td, out_dir, repeats=repeats) for td in track_dirs] + return { + "refDir": ref_dir, + "trackCount": len(track_dirs), + "tracks": tracks, + "aggregate": _aggregate_reference(tracks), + } + + def run_ab( *, input_dir: str | None, out_dir: str, model: str | None = None, repeats: int = 2, + ref_dir: str | None = None, ) -> dict[str, Any]: - """Top-level A/B run: synthetic smoke-test + optional real-track proxies.""" + """Top-level A/B run: ground-truth reference set (if given) + synthetic smoke-test + optional real-track proxies.""" from datetime import datetime, timezone os.makedirs(out_dir, exist_ok=True) @@ -359,6 +494,10 @@ def run_ab( "repeats": repeats, }, "caveats": [ + "Reference-set SI-SDR (when a --ref-dir is given) is the HEADLINE " + "signal: gain-aligned SI-SDR on a mono downmix vs real ground-truth " + "stems. Internally consistent for Demucs-vs-MSST, but NOT comparable " + "to published museval/BSSEval-v4 MUSDB leaderboard numbers.", "Synthetic SI-SDR is a PLUMBING smoke-test, not a real-music quality " "ranking: RoFormer/SCNet are trained on real spectra and may score " "poorly on toy synthetic sources. Do not read it as Demucs>MSST.", @@ -366,6 +505,11 @@ def run_ab( "correctness (no ground-truth stems). Runtime is end-to-end wall-clock " "(min of repeats after a warm-up); compare same-device rows only.", ], + "referenceSet": ( + evaluate_reference_set(ref_dir, out_dir, repeats=repeats) + if ref_dir and os.path.isdir(ref_dir) + else None + ), "syntheticSmokeTest": evaluate_synthetic(out_dir, repeats=repeats), "realTracks": [], } diff --git a/apps/backend/separation_backend.py b/apps/backend/separation_backend.py index 89740e36..94d97ef9 100644 --- a/apps/backend/separation_backend.py +++ b/apps/backend/separation_backend.py @@ -59,10 +59,21 @@ # (default: ``ASA_MSST_ROOT``), matching MSST-WebUI's own ``configs/`` + ``pretrain/`` # layout. Checkpoints are NOT vendored — the operator downloads weights from the # ``Sucial/MSST-WebUI`` Hugging Face repo into ``ASA_MSST_MODEL_DIR``. +# +# LICENCE GATE — read before promoting any of these to a product default. +# Every checkpoint in this registry is distributed under CC-BY-NC-SA-4.0 +# (NonCommercial; the Sucial/MSST-WebUI HF repo, and MUSDB18-trained), and the +# MSST-WebUI driver code is AGPL-3.0. This backend is therefore research / +# personal / NonCommercial-only and is NOT promotable to a commercial default — +# it is a strictly *worse* licence posture than the (already MUSDB-grey) Demucs +# incumbent. The full map and rationale live in +# incorporations/msst-separation-licence-gate-2026-06-05.md. Promotion is +# licence-gated, not quality-gated; the gate is closed. _DEFAULT_MSST_MODEL = "scnet_4stem" _MSST_MODEL_REGISTRY: dict[str, dict[str, str]] = { # 4-stem parity model — fills vocals/bass/drums/other, so the product path # (stemAnalysis overlay, pitch/note translation, MT3) behaves like Demucs. + # Weights: CC-BY-NC-SA-4.0 (NonCommercial) — see the LICENCE GATE above. "scnet_4stem": { "model_type": "scnet", "config_relpath": "configs/multi_stem_models/config_musdb18_scnet.yaml", @@ -71,6 +82,7 @@ # Strong 2-stem vocal BS-RoFormer (vocals + instrumental->other). RESEARCH / # A-B ONLY: it leaves bass/drums absent, so pitch/note translation + MT3 fall # back to the full mix. Never select this as a product-path default. + # Weights: CC-BY-NC-SA-4.0 (NonCommercial) — see the LICENCE GATE above. "bs_roformer_vocals": { "model_type": "bs_roformer", "config_relpath": "configs/vocal_models/config_vocals_bs_roformer.yaml", diff --git a/apps/backend/tests/test_separation_ab.py b/apps/backend/tests/test_separation_ab.py new file mode 100644 index 00000000..2d6b39bc --- /dev/null +++ b/apps/backend/tests/test_separation_ab.py @@ -0,0 +1,109 @@ +"""Focused tests for the ground-truth reference-set logic in separation_ab. + +Research-only harness, but the new SI-SDR aggregation math and the on-disk +MUSDB-style loader carry real logic worth pinning. These tests need only numpy + +the stdlib ``wave`` writer the harness already uses — no torch, no Demucs, no +MSST install. +""" + +from __future__ import annotations + +import os +import tempfile +import unittest + +import numpy as np + +import separation_ab as sab + + +def _write_wav(path: str, mono: np.ndarray) -> None: + sab._write_stereo_wav(path, np.asarray(mono, dtype=np.float32)) + + +class LoadReferenceTrackTests(unittest.TestCase): + def test_loads_mixture_and_present_stems(self) -> None: + with tempfile.TemporaryDirectory() as root: + track = os.path.join(root, "Some - Track") + os.makedirs(track) + n = sab.SAMPLE_RATE # 1 second + _write_wav(os.path.join(track, "mixture.wav"), np.zeros(n)) + _write_wav(os.path.join(track, "vocals.wav"), 0.3 * np.ones(n)) + _write_wav(os.path.join(track, "bass.wav"), -0.2 * np.ones(n)) + # drums/other deliberately absent — must be skipped, not invented. + + loaded = sab._load_reference_track(track) + self.assertIsNotNone(loaded) + mixture_path, known = loaded + self.assertTrue(mixture_path.endswith("mixture.wav")) + self.assertEqual(set(known), {"vocals", "bass"}) + self.assertEqual(len(known["vocals"]), n) + + def test_returns_none_without_mixture(self) -> None: + with tempfile.TemporaryDirectory() as root: + track = os.path.join(root, "no_mix") + os.makedirs(track) + _write_wav(os.path.join(track, "vocals.wav"), np.ones(sab.SAMPLE_RATE)) + self.assertIsNone(sab._load_reference_track(track)) + + def test_returns_none_without_any_stems(self) -> None: + with tempfile.TemporaryDirectory() as root: + track = os.path.join(root, "mix_only") + os.makedirs(track) + _write_wav(os.path.join(track, "mixture.wav"), np.ones(sab.SAMPLE_RATE)) + self.assertIsNone(sab._load_reference_track(track)) + + +class AggregateReferenceTests(unittest.TestCase): + def test_means_over_completed_tracks_only(self) -> None: + tracks = [ + { + "track": "t1", + "perBackend": { + "demucs": { + "status": "completed", + "runtimeSeconds": 10.0, + "quality": { + "vocals": {"siSdrDb": 6.0, "present": True}, + "bass": {"siSdrDb": 4.0, "present": True}, + "meanSiSdrDb": 5.0, + }, + }, + "msst": {"status": "skipped_no_msst"}, + }, + }, + { + "track": "t2", + "perBackend": { + "demucs": { + "status": "completed", + "runtimeSeconds": 20.0, + "quality": { + "vocals": {"siSdrDb": 8.0, "present": True}, + "bass": {"siSdrDb": 6.0, "present": True}, + "meanSiSdrDb": 7.0, + }, + }, + "msst": {"status": "error", "error": "boom"}, + }, + }, + ] + agg = sab._aggregate_reference(tracks) + self.assertEqual(agg["demucs"]["meanSiSdrDb"], 6.0) # mean(5,7) + self.assertEqual(agg["demucs"]["meanRuntimeSeconds"], 15.0) # mean(10,20) + self.assertEqual(agg["demucs"]["tracksScored"], 2) + self.assertEqual(agg["demucs"]["perStemMeanSiSdrDb"]["vocals"], 7.0) # mean(6,8) + self.assertEqual(agg["demucs"]["perStemMeanSiSdrDb"]["bass"], 5.0) # mean(4,6) + self.assertIsNone(agg["demucs"]["perStemMeanSiSdrDb"]["drums"]) # never scored + # MSST never completed → all-None aggregate, zero tracksScored. + self.assertIsNone(agg["msst"]["meanSiSdrDb"]) + self.assertEqual(agg["msst"]["tracksScored"], 0) + + def test_perfect_estimate_scores_high_sdr(self) -> None: + # Sanity: scoring a stem against itself yields a high (finite or inf) SI-SDR. + ref = np.sin(np.linspace(0, 50, sab.SAMPLE_RATE)).astype(np.float32) + self.assertGreater(sab.si_sdr(ref, ref), 100.0) + + +if __name__ == "__main__": + unittest.main() diff --git a/incorporations/msst-separation-licence-gate-2026-06-05.md b/incorporations/msst-separation-licence-gate-2026-06-05.md new file mode 100644 index 00000000..1ff62f2a --- /dev/null +++ b/incorporations/msst-separation-licence-gate-2026-06-05.md @@ -0,0 +1,90 @@ +# MSST / BS-RoFormer separation backend — licence gate + +**Date:** 2026-06-05 · **Status:** GATE — documented blocker, default stays Demucs · **Scope:** the optional `ASA_SEPARATION_BACKEND=msst` path landed in [PR #141](https://github.com/slittycode/ableton-sonic-analyzer/pull/141) (`separation_backend.py`, `scripts/msst_separate_runner.py`, the A/B harness). + +> *Not legal advice.* This records the licences as distributed and ASA's containment posture so a promotion decision is made with eyes open — it is not a legal opinion. + +## Why this doc exists + +The separation-backend campaign's first instruction was *"confirm the repo AND per-model weight licences first."* The infrastructure (PR #141) shipped before that confirmation was on record, and MSST-WebUI was never entered in the repo's existing licence-map discipline ([`forking-plans-2026-05-14.md`](forking-plans-2026-05-14.md)). This closes that gap. **Bottom line: the MSST backend as currently wired is research / personal / NonCommercial-only and must not be promoted to a commercial default.** + +## Licence map (verified 2026-06-05) + +| Component | Licence *as distributed* | Source | Implication for ASA | +|---|---|---|---| +| **ASA** (this repo) | MIT ([LICENSE](../LICENSE)) | repo | May *shell out* to AGPL tools at arms length; may **not** vendor or import AGPL code. | +| **MSST-WebUI** code | **AGPL-3.0** | [raw LICENSE](https://raw.githubusercontent.com/SUC-DriverOld/MSST-WebUI/main/LICENSE) — "GNU AFFERO GENERAL PUBLIC LICENSE Version 3" | Strong network-copyleft. ASA ships **zero** MSST code; the operator installs it into its own venv and ASA invokes it as a subprocess. See "AGPL containment" below. | +| **`model_scnet_sdr_9.3244.ckpt`** (registry id `scnet_4stem`) | **CC-BY-NC-SA-4.0** (explicit) | [`Sucial/MSST-WebUI`](https://huggingface.co/Sucial/MSST-WebUI) HF repo tag + per-file; trained on **MUSDB18-HQ** (`config_musdb18_scnet.yaml`) | **NonCommercial.** Attribution + ShareAlike. NC is inherited twice — explicit repo licence *and* MUSDB18 training data. Not promotable to a commercial default. | +| **`model_bs_roformer_ep_368_sdr_12.9628.ckpt`** (registry id `bs_roformer_vocals`) | **CC-BY-NC-SA-4.0** (explicit) | same HF repo (lucidrains BS-RoFormer *code* is MIT; these *weights* are not) | **NonCommercial.** Research / A-B only regardless (2-stem; leaves bass/drums empty). | +| **Demucs incumbent** — torchaudio `HDEMUCS_HIGH_MUSDB_PLUS` (what `analyze_audio_io.separate_stems` actually loads) | code **MIT/BSD**; weights carry **no explicit licence**, trained on MUSDB18-HQ + Meta-internal data | [facebookresearch/demucs README](https://github.com/facebookresearch/demucs) ("released under the MIT license"); [torchaudio pipeline doc](https://docs.pytorch.org/audio/stable/generated/torchaudio.pipelines.HDEMUCS_HIGH_MUSDB_PLUS.html) | The grey baseline MSST would replace — see the comparison below. | + +## The finding that actually matters: this is a downgrade, not an upgrade (on licence terms) + +It is tempting to frame the gate as "MSST is NonCommercial, Demucs is clean." **That is wrong and would mislead by omission.** The incumbent Demucs weights are *also* MUSDB18-derived — neither backend rests on commercially-clean weights. The real, honest difference is in how *explicit and binding* the encumbrance is: + +| Axis | Demucs (incumbent) | MSST (candidate) | Direction | +|---|---|---|---| +| Driver-code licence | MIT / BSD (permissive) | **AGPL-3.0** (network copyleft) | ⬇ worse | +| Weights licence *as distributed* | **no explicit licence** (upstream licenses only the code) | **explicit CC-BY-NC-SA-4.0** | ⬇ worse | +| Training-data provenance | MUSDB18-HQ + Meta internal (NC provenance, unforegrounded) | MUSDB18-HQ (NC) | ≈ same | + +So adopting MSST trades an *implicit, unforegrounded* NC-provenance risk for an **explicit, contractually-binding NonCommercial + ShareAlike licence**, and swaps a permissive driver for an AGPL one. On licence posture it is **strictly worse on both axes.** The quality win (higher published SDR) may still justify it for **NonCommercial use**, but it cannot be the answer to "make ASA's separation commercially shippable." + +**Corollary (the strategic conclusion):** there is no commercially-promotable separation upgrade *in this repo's model set*. A genuinely commercial default would need weights trained on open/cleared data and distributed under a permissive or commercial licence — not these checkpoints, and not the current Demucs weights either. That is a sourcing problem, not a wiring problem. + +## AGPL containment (why the AGPL *code* is acceptable, default-off) + +ASA's MIT licence is preserved because the AGPL boundary is a process boundary, not a link: +1. **Zero AGPL code is shipped or vendored.** `requirements-msst.txt` is a *setup pointer*, not a dependency; the operator clones MSST-WebUI themselves into a separate venv. +2. **Invocation is arms-length** — `separation_backend.py` shells out to `scripts/msst_separate_runner.py` under `ASA_MSST_PYTHON` (a different interpreter), exchanging only a file path in and a one-line JSON manifest out. No import, no linking. This is the mainstream reading of "mere aggregation / separate program." +3. **Hosted + networked use** (the `hosted` runtime profile, were MSST ever enabled there) triggers AGPL §13's obligation to offer the running source to users. Because ASA runs **unmodified** upstream MSST-WebUI, that obligation is satisfied by pointing users at the upstream repo — but it must be a conscious operator decision, not a silent default. + +This containment is exactly why PR #141's subprocess-in-its-own-venv design is load-bearing and must not be "simplified" into an in-process import. + +## Decision + +1. **Default stays Demucs.** `ASA_SEPARATION_BACKEND=demucs` remains the only product-path default (PURPOSE.md invariant #1: measurement is authoritative; the stems contract is unchanged either way). +2. **MSST stays a default-off, research / personal / NonCommercial-only experiment.** Acceptable for an operator analysing audio they own, for quality research, and for the A/B harness. Not acceptable as a shipped/commercial default. +3. **Do not promote any MSST model to default on quality grounds alone.** Promotion is licence-gated, not quality-gated, and the gate is closed. +4. **Durable in-code guardrail added** so the gate travels with the code, not just this doc: licence warnings on each `_MSST_MODEL_REGISTRY` entry (`separation_backend.py`) and a Licence section in `requirements-msst.txt`. + +## What would unblock promotion (definition of done for a future "commercial separation" effort) + +- [ ] Identify a separation model whose **weights** are distributed under a permissive or commercial licence (not CC-BY-NC, not MUSDB-only-trained). Candidates to investigate: models trained on cleared/commissioned or fully-open stem corpora. +- [ ] Re-run this licence map against that model's *weights* (not just its code) and its training-data provenance. +- [ ] Only then is an SDR + runtime A/B vs Demucs a *promotion* input rather than a *research* input. + +## A/B results — NonCommercial research run (2026-06-05) + +The owner approved a NonCommercial-research install, so MSST was stood up on the local Apple-Silicon box and the A/B was run for real: + +- **Install:** MSST-WebUI checkout + a *minimal* SCNet-inference venv (CPU/MPS torch; the CUDA-only `sageattention` / `bitsandbytes` / `asteroid` in MSST's full `requirements.txt` are not needed for inference and were omitted — a much smaller, arm64-friendly install). Weights: `model_scnet_sdr_9.3244.ckpt` (CC-BY-NC-SA-4.0) from the `Sucial/MSST-WebUI` HF repo. +- **Reference set:** the first 5 MUSDB18 **test-split** tracks (7-second previews via the `musdb` package — both backends were trained on MUSDB *train*, so a fair A/B must use the test split), ground-truth stems, both backends on **CPU**. +- **Metric:** gain-aligned SI-SDR on a mono downmix (the harness's new `--ref-dir` mode). Internally consistent for Demucs-vs-MSST, but **NOT** museval/BSSEval-v4 — not comparable to published MUSDB leaderboard numbers. + +| Backend | mean SI-SDR (dB) | mean runtime (s/clip) | +|---|---|---| +| **Demucs** (`HDEMUCS_HIGH_MUSDB_PLUS`, incumbent) | **8.08** | **1.26** | +| **MSST** (`scnet_4stem`) | 6.22 | 40.78 | + +Per-stem mean SI-SDR (dB): vocals **D 8.62 / M 10.77**, bass D 9.87 / M 6.21, drums D 9.13 / M 8.33, other D 4.72 / M −0.42. Per-track, Demucs won 3/5 (one big swing on *Arise* — 12.0 vs 3.0), MSST won 2/5 (both narrow). Full report: `apps/backend/.runtime/separation_ab/report.json` (git-ignored runtime artifact). + +**Reading it — caveats first:** +1. **Runtime: MSST is ~32× slower than Demucs on CPU** (40.8 s vs 1.3 s per 7 s clip; model load is trivial at ~0.2 s — the cost is all chunked CPU inference). For ASA's CPU-local deployment that is a serious practical cost. +2. **Quality: `scnet_4stem` did NOT beat Demucs overall on this proxy** (6.22 vs 8.08 mean). MSST's **vocals are better** (10.77 vs 8.62 — consistent with the model family's reputation), but its **`other` collapses** (−0.42) and bass trails. +3. **This contradicts the published museval SDR** (`scnet` ~9.32 vs `htdemucs` ~9.0) — which means the **proxy, not necessarily the model, is the limiter**: 7-second clips are too short for chunked SCNet to shine, mono SI-SDR ≠ museval, and N=5 is tiny. Treat this as a **preliminary signal, not a verdict.** The full-length 30 s re-run (full MUSDB18 download was in progress) is the honest basis for any strong quality claim. + +**What it changes:** nothing about the gate — promotion stays licence-blocked regardless of SDR. But it removes any *urgency*: on this box `scnet_4stem` is both far slower and not clearly better, so even a NonCommercial "quality upgrade" rationale is unproven here. The vocals win is the one thread worth pulling later (a NonCommercial vocals-focused BS-RoFormer for a vocals-only research path). + +**Reproduce:** with the MSST venv + checkout + weights in place (see `requirements-msst.txt`): +``` +ASA_MSST_PYTHON=/bin/python ASA_MSST_ROOT= ASA_MSST_DEVICE=cpu \ + ./venv/bin/python scripts/ab_separation_backends.py --ref-dir --repeats 1 +``` + +## Sources + +- MSST-WebUI code licence: [GNU AGPL-3.0, raw LICENSE](https://raw.githubusercontent.com/SUC-DriverOld/MSST-WebUI/main/LICENSE) +- Model weights licence: [`Sucial/MSST-WebUI` model card — `cc-by-nc-sa-4.0`](https://huggingface.co/Sucial/MSST-WebUI); per-file confirmation for `model_scnet_sdr_9.3244.ckpt` and `model_bs_roformer_ep_368_sdr_12.9628.ckpt` +- Demucs: [facebookresearch/demucs README (code MIT)](https://github.com/facebookresearch/demucs); [torchaudio `HDEMUCS_HIGH_MUSDB_PLUS`](https://docs.pytorch.org/audio/stable/generated/torchaudio.pipelines.HDEMUCS_HIGH_MUSDB_PLUS.html) (training: MUSDB-HQ + 150 internal songs) +- MUSDB18 dataset terms: NonCommercial ([source-separation.github.io](https://source-separation.github.io/tutorial/data/musdb18.html))