feat: optional MT3 polyphonic-transcription stage + interpretation grounding gate

[slittycode]

Summary

Adds MT3 (Magenta Multi-Task Multitrack) as an optional, flag-gated, additive analysis stage that emits per-instrument MIDI, and makes the interpretation stage actually consume that grounding. MT3 never overrides Phase 1 measurements — it is additive context only (PURPOSE.md invariant #1). The whole feature is opt-in per run (mt3_mode=enabled) and inert otherwise; its heavy JAX/t5x deps live in a separate optional venv and never touch the product install or the request path.

4 commits, MT3-only diff (~33 files):

1. feat(mt3) — optional MT3 stage (F1–F4). New mt3_transcription.py (lazy-imported so JAX never loads at startup); mt3_attempts table + lifecycle/recovery/interrupt wiring in analysis_runtime.py; _execute_mt3_attempt subprocess executor + _mt3_worker_loop + per-run opt-in route in server.py; requirements-mt3.txt optional extra. Follow-ups: bidirectional Demucs stems handover with pitch/note (F1), estimate prices the MT3 stage when requested (F2), Phase 2 grounding forwards mt3Result to the prompt + citation validator with a new OPTIONAL_MT3_TRANSCRIPTION_RESULT_JSON input (F3), and a frontend Mt3TranscriptionPanel with lazy per-track MIDI download wired into the Session Musician suite (F4).
2. fix(mt3) — strict mt3_mode validation. An unknown mt3_mode now returns 400 MT3_MODE_UNSUPPORTED instead of silently disabling MT3; install docs mandate a separate venv (JAX/t5x pull older numpy/protobuf that would break Essentia/torch on the request path).
3. feat(ui) — flag-gated MT3 opt-in toggle.
4. feat(backend) — interpretation grounding gate (the fix that makes F3 actually deliver).

The interpretation grounding gate

mt3, pitch_note, and interpretation were peers — all reservable the moment measurement completed — so they raced. MT3 loads multi-GB weights and usually lost, leaving mt3Result/pitchNoteResult None by the time Gemini ran. Now:

• reserve_next_interpretation_attempt blocks a queued interpretation while an mt3 or pitch_note attempt for the same run is in-flight (queued/running). Terminal states (completed/failed/interrupted) unblock it, so failure/interrupt/restart-recovery can never deadlock — interpretation then runs with whatever grounding is available, exactly as before.
• _enqueue_requested_followups creates the interpretation attempt last (after mt3 + pitch_note), closing the cross-transaction window where it could reserve ungrounded.
• _execute_mt3_attempt/_execute_pitch_note_attempt resolve get_source_artifact inside the try (it sat before it) and null-safe the except diagnostics, so a pre-subprocess raise terminalizes rather than leaving the attempt stuck running (which, under the gate, would block the run's interpretation indefinitely).

Pure scheduling — changes when interpretation runs, never what any stage produces. camelCase contract and types.ts untouched; snapshot keeps emitting queued while waiting.

Testing

• Backend: 1008 passed, 3 skipped (stdlib unittest). New coverage: MT3 module/runtime/executor, estimate, citation-path + grounding, the interpretation-gating suite (incl. an enqueue-order regression guard), an end-to-end test proving a non-null mt3_result reaches the prompt, and mt3 timeout + get_source_artifact-failure terminalization tests (the no-deadlock linchpins).
• Frontend: tsc clean, build OK, 704 unit tests pass.

Reviewer notes

• MT3 deps are not installed in the product venv — see requirements-mt3.txt (separate venv + out-of-band checkpoint download; apps/backend/models/ is gitignored).
• The stage is fully absent/inert unless mt3_mode=enabled per run.
• This work was independently reviewed (two Opus passes on the ordering gate + an advisor pass that surfaced the get_source_artifact hardening, now included).

🤖 Generated with Claude Code

[slittycode]

Verdict: APPROVE

Summary

Adds MT3 as an optional, flag-gated, purely additive fourth stage behind mt3_mode=enabled per run — inert unless explicitly requested, heavy deps isolated to a separate venv, subprocess-isolated so JAX never enters the server process. The headline correctness change is the interpretation grounding gate: reserve_next_interpretation_attempt now blocks on in-flight MT3 and pitch-note rows, and _enqueue_requested_followups creates the interpretation row last so the gate can't be raced. Phase 1 contract is untouched. I cannot run the test suites in this environment (venv/node_modules not set up), but I read every new test file and the coverage of the gate logic is thorough: 8 scenarios including the no-deadlock and restart-recovery paths. The PR description's claim of 1008 backend / 704 frontend tests passing is consistent with the test structure I verified.

Findings

Should fix

server.py:1571 — the _Mt3UnavailableError handler accesses source_artifact["artifactId"] directly, not null-safe:

except _Mt3UnavailableError as exc:
    runtime.fail_mt3_attempt(
        attempt_id,
        ...
        diagnostics={
            "sourceArtifactId": source_artifact["artifactId"],   # ← not null-safe

The generic except Exception handler four lines below correctly uses (source_artifact or {}).get("artifactId") (line 1588), with a comment explaining why. _Mt3UnavailableError can only be raised after subprocess.run() completes, which comes after get_source_artifact succeeds, so source_artifact is never None at that point — safe today by invariant. But the whole point of the get_source_artifact hardening commit was to make all except-handler paths null-safe. This is the one asymmetry and it's a one-line fix.

Worth considering

requirements-mt3.txt:45 — mt3 @ git+https://github.com/magenta/mt3.git is unpinned to a commit hash. The file comment acknowledges this explicitly, calls pinning "RECOMMENDED", and explains the checkpoint can't be verified automatically from inside this repo. For a library described as unmaintained upstream, this is a supply-chain risk for any deployment that actually runs pip install -r requirements-mt3.txt. Raising it only because "unmaintained" + "unpinned" is a combination that tends to bite later; the comment already tells operators to pin, so this is a policy call.

server.py:1381-1385 — _MT3_NOT_AVAILABLE_MARKERS is a hardcoded substring match against the exact error phrasing in mt3_transcription.py's Mt3NotAvailableError. The comment already flags this as fragile and says "update both sides together" if phrasing changes. No action needed, just confirming I saw it.

Test results

Could not execute. Venv and node_modules not installed in the review environment. Test structure verified manually for all new files:

• test_mt3_transcription.py: unit tests for to_payload() camelCase contract, _resolve_sources() stems-first/full-mix fallback, discover_stems_dir(), typed Mt3NotAvailableError propagation, subprocess gate tests, slow integration test gated on RUN_SLOW_TESTS=1.
• test_analysis_runtime.py InterpretationGatingTests: 8 cases covering MT3-only block, pitch-note-only block, both-must-complete, fail-is-terminal, neither-requested-is-immediate, recovery-clears-stale-running, and the enqueue-order regression guard (monkey-patches create methods, asserts ["pitch_note", "mt3", "interpretation"] call order).
• test_server.py Mt3ExecutorTests: happy path artifact-ref swap (midiB64 → midiArtifactId/midiSizeBytes), MT3_NOT_AVAILABLE → retryable: False, generic failure → retryable: True, timeout terminalization, get_source_artifact failure terminalization, prompt grounding presence/absence.
• test_phase2_citation_paths.py Mt3CitationPathsTests: paths rejected without mt3_result, accepted with it, array-item paths (tracks.instrument, tracks.noteCount) validated, invented paths still flagged.

Phase boundary check

Clean. In detail:

• complete_measurement() pops transcription from the raw analyzer output before persisting, so the ASA_ENABLE_MT3=1 direct-CLI path cannot leak MT3 data into stages.measurement.
• _mt3_stage_snapshot() hardcodes "authoritative": False; the TypeScript Mt3StageSnapshot type enforces authoritative: false as a literal (compile-time guard).
• Phase 2 system prompt rule 5 explicitly prohibits MT3 from overriding BPM/key/loudness/duration/structure and prohibits it from voting against pitch-note translation data.
• _validate_phase2_citation_paths only adds transcription.mt3.* paths to the allowed citation set when mt3_result is actually provided — fabricated MT3 citations without a real result are flagged.
• EXPECTED_TOP_LEVEL_KEYS in test_analyze.py correctly excludes transcription (the MT3 namespace lives on snapshot.stages.mt3, not the analyze.py envelope), with a comment explaining why.

---

Generated by Claude Code