feat(backend): make MSST separation run for real + ground-truth SDR A/B + licence gate

[slittycode]

Summary

Follows #141 (pluggable Demucs↔MSST separation backend). This pass does the three things #141 deferred: confirms the licences, makes the MSST runner actually work end-to-end, and adds a ground-truth-stems SDR mode to the A/B harness so Demucs-vs-MSST is measurable on real isolated stems.

Default path is untouched — ASA_SEPARATION_BACKEND still defaults to demucs; analyze.py's JSON contract is unchanged.

Licence gate (the headline)

Component	Licence (as distributed)
MSST-WebUI code	AGPL-3.0
scnet_4stem + bs_roformer_vocals weights	CC-BY-NC-SA-4.0 (NonCommercial)
Demucs incumbent (HDEMUCS_HIGH_MUSDB_PLUS)	code MIT/BSD; weights carry no explicit licence (MUSDB-grey)

MSST is a strictly worse licence posture on both axes → research / NonCommercial-only, NOT promotable to a commercial default. Promotion is licence-gated, not quality-gated. Full map + rationale + results: incorporations/msst-separation-licence-gate-2026-06-05.md. Durable in-code guardrails added to _MSST_MODEL_REGISTRY and requirements-msst.txt.

Runner fixes (3 integration bugs only a real install surfaces)

scripts/msst_separate_runner.py had never been run (#141 had no MSST install):

1. chdir into the checkout — MSSeparator import reads data_backup/webui_config.json via a relative path; caller paths resolved to absolute first.
2. drop the injected stdlib logger — MSSeparator needs MSST's own get_logger (with a console_handler attribute); its handler targets stderr, and the existing redirect_stdout keeps the stdout JSON contract clean.
3. sample-rate read — regex fallback for configs carrying !!python/tuple tags.

Ground-truth SDR A/B

separation_ab.py gains a --ref-dir mode: loads MUSDB-style {mixture,vocals,bass,drums,other}.wav, runs each backend, scores true per-stem SI-SDR + aggregate. New tests/test_separation_ab.py; existing separation tests green.

NonCommercial research run (5 MUSDB18 test tracks, 7 s, CPU, mono gain-aligned SI-SDR — a preliminary proxy, NOT museval)

Backend	mean SI-SDR	runtime
Demucs	8.08 dB	1.26 s/clip
MSST scnet_4stem	6.22 dB	40.8 s/clip (~32× slower)

MSST wins vocals (10.77 vs 8.62) but trails overall and collapses on other (−0.42). This contradicts published museval SDR (scnet ~9.32 > htdemucs ~9.0) → the short-clip mono proxy is the limiter; a full-length re-run (full MUSDB18 download pending) is the honest basis for any quality claim. Conclusion unchanged either way: do not promote.

Not included

• MSST venv, checkout, weights, MUSDB data — all local/external, never committed.
• .runtime/separation_ab/report.json — git-ignored runtime artifact.

Test plan

• cd apps/backend && ./venv/bin/python -m unittest tests.test_separation_ab tests.test_separation_backend — 22 green.
• MSST runner + full A/B exercised live on the local Apple-Silicon box (CPU).

🤖 Generated with Claude Code

[slittycode]

Verdict: APPROVE

Summary

Three scoped changes in one PR: runner bugfixes that make the default-off MSST subprocess actually work end-to-end, a ground-truth reference-set mode for the A/B harness (the thing needed to get a real SDR number), and a licence-gate doc + durable in-code guardrails. Product path is untouched — ASA_SEPARATION_BACKEND still defaults to demucs, analyze.py's JSON contract is unchanged, separation_backend.py gets only comments. CI is still running at review time; changes are off-path enough that a failure here would be unrelated infra noise, not a logic regression in this diff.

Findings

Worth considering (not blocking)

evaluate_reference_track passes warmup=False, repeats=repeats (default 2) to run_backend_separation. warmup=False skips the amortising warm pass, but the timing loop still runs repeats times and takes the min — so the reported runtime is min(2 cold-start runs), not "a single honest cold-start" as the comment says. For MSST the second subprocess invocation benefits from OS page-cache warming on the config/checkpoint files, which shaves a fraction of the per-call cold cost. Not wrong enough to block, just slightly off from the claimed semantics. Easy fix: repeats=1 in that call, or update the comment.

_score_against_known buries meanSiSdrDb at the same dict level as the per-stem entries (pre-existing shape). The new _aggregate_reference reads quality.get("meanSiSdrDb") correctly, and the test fixture matches — no bug — but if a future caller iterates quality.items() expecting only stem names it'll trip over meanSiSdrDb as a key. Worth noting if the shape ever gets serialised more broadly.

Test results

Can't run the suite from this environment (no venv). I verified the three symbols the new test file pulls from pre-existing code: SAMPLE_RATE = 44100 (line 43), _write_stereo_wav (line 90, handles 1-D mono input correctly), si_sdr (line 117, gain-aligned SI-SDR — the assertGreater(si_sdr(ref, ref), 100.0) sanity test is correct because zero-noise → inf). Aggregation math in _aggregate_reference tested against a hand-wired fixture; the means are verifiable by inspection. PR description reports 22 green.

Phase boundary check

Clean. No changes to analyze.py, any analyze_*.py module, or the Phase 1 JSON schema. separation_backend.py is comments-only. No Phase 2/3 files touched.

---

Generated by Claude Code