docs(genre): coverage audit + genre-generalization program plan (PR-G1)

audits/genre-coverage-2026-07-13.md

@@ -0,0 +1,192 @@
+# Genre coverage audit — where the measurements are only proven on techno/house (2026-07-13)
+
+Coverage audit for the genre-generalization program
+(`plans/genre-generalization-program.md`). Maps the three genre-aware
+surfaces against the priority electronic genres and names the gaps. No code
+changes; every claim below carries the file it was read from.
+
+The three surfaces:
+
+1. **Genre classifier signatures** — `_GENRE_SIGNATURES`,
+   `apps/backend/analyze_detection.py:1031` (34 signatures) plus
+   `_GENRE_FAMILY_MAP` at `analyze_detection.py:1078`.
+2. **Synthetic fundamentals corpus** —
+   `apps/backend/scripts/build_synthetic_corpus.py` (29 clips,
+   `fundamentals_eval_manifest.synthetic.json`).
+3. **Mix-doctor genre profiles** — `apps/ui/src/data/genreProfiles.ts`
+   (34 profiles, same id set as the backend signatures), resolved by
+   `apps/ui/src/services/mixDoctor.ts` via `genreDetail.genre` →
+   `genreDetail.genreFamily` → default.
+
+Priority genres (program mandate): UK bass / garage / speed garage / 2-step;
+halftime / trap / dubstep / bass; hard dance / hardstyle / gabber; IDM /
+ambient / electro; plus footwork, breakbeat, downtempo.
+
+## Summary verdict
+
+The accuracy gates are real but narrow: the synthetic corpus proves the
+pipeline on **steady kick-per-beat 4/4 material, mostly 96–140 BPM**, and
+the genre surfaces enumerate house/techno lineages in depth while entire
+electronic families (trap, hardstyle, gabber, footwork, electro, halftime,
+2-step-as-rhythm) have **no signature, no profile, and no corpus clip**.
+Nothing in the active gate set exercises a broken kick pattern, a halftime
+feel, 16th-note shuffle, beatless material, or any tempo above 174.
+
+## Surface 1 — `_GENRE_SIGNATURES` (analyze_detection.py:1031)
+
+### What exists (34 signatures)
+
+| Family | Signatures | BPM span |
+|---|---|---|
+| Ambient/downtempo | ambient-drone, ambient-techno, dub-techno, ambient | 40–125 |
+| House | deep/organic/classic/tech/progressive/afro/bass-house, house | 115–132 |
+| Techno | minimal/melodic/driving/industrial/hard/acid/detroit, techno | 122–160 |
+| Trance | trance, psytrance | 136–148 |
+| Bass | dubstep | 138–145 |
+| D&B/breaks | drum-bass, neurofunk, dnb, breaks | 125–180 |
+| UK garage | uk-garage, bassline, garage | 128–142 |
+| Legacy broad | edm, hiphop, rock, pop, acoustic | 70–160 |
+
+Feature axes per signature: `bpm`, `subBassDb`, `crestFactor`,
+`onsetDensity`, `spectralCentroid`, `sidechainStrength`, `bassDecay`,
+optional `rt60` and `kickDistortion` (kickDistortion used by exactly two:
+industrial-techno, hard-techno). Weights (`analyze_detection.py:1180`):
+sidechain 0.95 and bass decay 0.85 are the primary discriminators — a
+**four-on-the-floor-centric assumption**, since both features encode the
+kick-pumping relationship that broken-beat genres don't have.
+
+### Priority-genre gaps
+
+1. **trap** — nothing covers it. `hiphop` tops out at 110 BPM and assumes
+   the felt (half-time) tempo; a trap track measured at its notated ~140
+   BPM (see the tempo-octave trap below) lands nearest `dubstep`
+   (138–145), whose sub/decay/centroid ranges then feed wrong mix advice.
+2. **footwork / juke** (155–165) — falls between `hard-techno` (145–160,
+   wrong character: footwork has low sidechain, sparse sub-heavy toms) and
+   `dnb` (160–180, wrong rhythm). No signature.
+3. **hardstyle** (150–160) — `hard-techno` will absorb it, but hardstyle's
+   reverse-bass kick and much higher kick THD are a different production
+   blueprint; the Phase 2 advice chain treats them identically today.
+4. **gabber / hardcore** (170–200+) — **no signature of any kind above
+   180 BPM.** A 190 BPM gabber track's best match is `dnb`/`drum-bass`,
+   which is flatly wrong (different kick, different sub, different swing).
+5. **electro** (100–135, broken 808 machine funk) — no signature. The
+   broken-beat + high-crest + moderate-tempo combination scores weakly
+   everywhere; likely misread as `breaks` or `pop`.
+6. **IDM** — no signature. Deliberately heterogeneous; the honest behavior
+   is a *confident abstain* (`genreDetail: None` / low confidence), which
+   the classifier already supports (`primary_score < 0.25` →
+   abstain, `analyze_detection.py:1217`). The audit recommendation is to
+   verify abstention on IDM-like material rather than force a signature.
+7. **2-step** — `uk-garage` (128–136) exists, but its
+   `sidechainStrength: (0.35, 0.65)` encodes 4×4-garage pumping; classic
+   2-step has a broken kick and little sidechain, so it scores *against*
+   the one signature meant to catch it. No separate 2-step signature; no
+   **speed garage** signature (4×4 130–140, warehouse sub) either.
+8. **halftime (D&B tempo, half feel)** — no signature; at the measured
+   ~85–87 BPM (post-octave-halving) it reads as `hiphop`.
+9. **breakbeat** — `breaks` (125–135) exists; adequate for nu-skool/big
+   beat. The gap is *measurement-level* (downbeats on syncopated kicks),
+   not signature-level.
+10. **downtempo / trip-hop** (60–110 with beats) — falls in the crack
+    between `ambient` (onsetDensity 0–3, i.e. nearly beatless) and
+    `hiphop` (centroid ≤ 2500). No dedicated signature.
+
+### Family-map and cross-boundary gaps
+
+1. `_GENRE_FAMILY_MAP` (`analyze_detection.py:1078`) has **no entries for
+   uk-garage, bassline, or garage** — all three fall to `"other"`, so the
+   mix-doctor family fallback (`mixDoctor.ts:77`) never fires for garage.
+2. The TS side pins `genreFamily` to a closed union of 8 values —
+   `apps/ui/src/types/measurement.ts:652` and the `validFamilies` allowlist
+   in `apps/ui/src/services/backendPhase1Client.ts:1134`. **Any new family
+   must land on both sides** (tripwire #3) or it silently degrades to
+   `"other"`.
+3. The vocal boost list (`analyze_detection.py:1205`) and supersaw/acid
+   boosts don't reference any of the missing genres — fine today, but new
+   signatures for trap (808 + vocal) and hardstyle (kick THD) should reuse
+   the existing `kickDistortion` axis rather than invent new ones.
+
+## Surface 2 — synthetic corpus span (build_synthetic_corpus.py)
+
+### What the 29 clips cover
+
+| Axis | Coverage |
+|---|---|
+| Tempo (grid clips) | 70, 90, 110, 128, 140, 174 BPM |
+| Tempo (other kinds) | chords 96–140, swing 124, multi 122/128 |
+| Meter | 4/4, 3/4, 6/8, 7/8 (odd meters are registered `knownGaps`) |
+| Rhythm pattern | **one pattern only: kick on every beat, accented downbeat** (`render_grid_pattern`) |
+| Swing | straight-vs-swung **8th-note** hats, 50–66%, at 124 BPM only |
+| Key/chords | 12 roots × major/minor, clean sine triads |
+| Texture | drums-only, chords-only, bass-only, 2 multi-layer 4/4 clips |
+
+Already-registered `knownGaps` (`_KNOWN_GAPS_BY_ID`,
+`build_synthetic_corpus.py:395`): odd meters read as 4/4 (downbeats follow),
+7/8 tempo smears, and **174 BPM halves to 86.9** (octave preference).
+
+### What is absent — the priority-genre rhythm surface
+
+1. **Broken kick patterns (2-step, breakbeat, electro).** Every grid clip
+   places a kick on every beat. No active check exercises the downbeat
+   heuristic (`_compute_downbeat_phase`, `analyze_rhythm.py:160` — it
+   *documents* that four-on-the-floor carries no phase information) or the
+   meter autocorrelation on syncopation.
+2. **Halftime feel (kick 1 / snare 3).** The canonical tempo-octave trap —
+   a 140 BPM halftime clip whose truth is 140, not 70 — does not exist.
+   Both estimators (RhythmExtractor2013 + Percival) plus the ratio-based
+   `apply_bpm_correction` (`analyze_core.py:39`, which *always* sides with
+   Percival on a ~2×/1.5× ratio) are unexercised on this case.
+3. **Triplet / shuffle swing and 16th-note shuffle.** Swing clips stop at
+   66% on the 8th grid. UKG/2-step shuffle lives on the **16th** grid —
+   which `compute_swing_detail` (`analyze_rhythm.py:191`) cannot see: it
+   keeps only IOIs in 0.30–0.70 beats and hardwires
+   `gridResolution: "8th"`.
+4. **Sparse / beatless material.** No clip verifies honest abstention —
+   tempo confidence collapsing, `swingDetail: None`, meter falling back to
+   `assumed_four_four`, key ensemble hedging on drones. (The chords clips
+   are beatless but only check key/chords; nothing asserts what the rhythm
+   stack *should not* claim on them.)
+5. **Tempo extremes 85 / 150 / 190.** 85 (halftime/trap felt tempo) and
+   150 (hardstyle) are absent; 174 exists only as a knownGap; **nothing
+   above 174 at all** (gabber territory, and the octave-halving failure
+   is *worse* there).
+
+## Surface 3 — mix-doctor genre profiles (genreProfiles.ts)
+
+The 34 profile ids mirror the backend signature table one-for-one, so every
+signature gap above is also a profile gap: **no trap, footwork, hardstyle,
+gabber, electro, IDM, 2-step, speed garage, halftime, or downtempo
+spectral/loudness targets.**
+
+Consequence chain: a trap track misclassified as `dubstep` receives
+dubstep's `spectralTargets` and LUFS/crest ranges from
+`resolveMixDoctorGenreId` (`mixDoctor.ts:93`); the advice is specific,
+confident, and wrong — the exact failure mode PURPOSE.md invariant #4
+exists to prevent. The genre-agnostic loudness guardrails
+(`loudnessGuardrails.ts`) still hold, but every genre-relative suggestion
+inherits the misclassification.
+
+Profile shape (`genreProfiles.ts:28`): crest/PLR/LUFS ranges + 7 spectral
+band targets. New profiles are additive data entries; the risk is not
+mechanism but **unvalidated targets** — new-genre targets must be sourced
+from reference-track measurements, not invented (same discipline as the
+2026-05-16 audit applied to the existing table).
+
+## Measurement-level implications (what the fix PRs target)
+
+| # | Measurement | Failure mode on priority genres | Evidence |
+|---|---|---|---|
+| 1 | Tempo octave | 174 → 86.9 measured; 190 unmeasured but same mechanism; halftime trap ambiguous by construction | `_KNOWN_GAPS_BY_ID`, `apply_bpm_correction` always prefers Percival on ratio hits |
+| 2 | Meter / downbeats | odd meters read 4/4 (measured); syncopated/broken kicks unmeasured — kick-accent phase heuristic assumes kick-on-downbeat | `analyze_time_signature` 20% margin (`analyze_core.py:1292`), `_compute_downbeat_phase` |
+| 3 | Swing | 16th-grid shuffle invisible; `gridResolution` hardwired `"8th"` | `compute_swing_detail` IOI window 0.30–0.70 beats |
+| 4 | Key on sparse/ambient | ensemble landed (PR-B3, keep EDMA) but no abstention/hedge check exists on beatless or drone material | `_build_key_ensemble` (`analyze_core.py:159`), corpus gap #4 |
+| 5 | Genre classifier | 10 priority families unrepresented or mis-tuned (see Surface 1); family map + TS union must move together | `analyze_detection.py:1031/1078`, `measurement.ts:652`, `backendPhase1Client.ts:1134` |
+
+## Recommended sequence
+
+See `plans/genre-generalization-program.md` for the PR-by-PR program this
+audit feeds. Ordering rationale: corpus first (you can't gate what you
+can't measure), then measurement fixes each behind the new clips'
+pre-registered checks, classifier/profile widening last (it depends on the
+measurements being right for the new material).

plans/genre-generalization-program.md

@@ -0,0 +1,161 @@
+# Genre-generalization program — from "passes on techno" to "passes across electronic"
+
+Successor to the Phase A/B accuracy program (ground-truth harnesses; meter
+evidence, real swing, key ensemble, stem-aware chords — all merged). The
+accuracy gates currently over-index on four-on-the-floor techno/house: the
+synthetic corpus is steady-kick 4/4 at mostly 96–140 BPM, and entire
+electronic families have no signature, no mix-doctor profile, and no corpus
+clip. The measured evidence is in
+`audits/genre-coverage-2026-07-13.md` — read it first; this plan is the
+PR-by-PR execution of its findings.
+
+Priority genres: UK bass / garage / speed garage / 2-step; halftime / trap /
+dubstep / bass; hard dance / hardstyle / gabber; IDM / ambient / electro;
+plus footwork, breakbeat, downtempo.
+
+## Non-negotiable discipline (how every prior PR shipped)
+
+1. **Additive or surfacing-only, behind a pre-registered gate.** Never
+   override an authoritative Phase 1 value (PURPOSE.md invariant #1). A fix
+   that would change a shipped value (e.g. the octave-corrected `bpm`)
+   first ships as surfaced evidence + a frozen decision doc in
+   `incorporations/`, exactly like the key-ensemble gate
+   (`incorporations/key-ensemble-decision-2026-07-04.md`).
+2. **Every new field lands on both sides**: `analyze.py` camelCase →
+   `JSON_SCHEMA.md` + `EXPECTED_TOP_LEVEL_KEYS` (and the full-only list) →
+   `src/types/measurement.ts` → `backendPhase1Client.ts` reconstructor →
+   `phase1FullPayload.ts` fixture → `phase1CitationContract.test.ts` if
+   citable. Then re-baseline the golden deliberately:
+   `UPDATE_PHASE1_GOLDEN=1 ./venv/bin/python -m unittest tests.test_phase1_golden`.
+3. **Green before every push** (from repo root):
+   ```bash
+   cd apps/backend && ./venv/bin/python -m unittest discover -s tests
+   ./venv/bin/python scripts/build_synthetic_corpus.py --check --audio-only \
+     --out-dir tests/fixtures/fundamentals_tracks \
+     --manifest tests/fixtures/fundamentals_eval_manifest.json
+   ./venv/bin/python scripts/evaluate_fundamentals.py --fail-on-skip
+   cd ../ui && npm run verify
+   ```
+4. **One coherent change per PR**, stacked for review — behavior changes
+   are shown before merging. Honest PR bodies: what is validated
+   synthetically vs. what still needs real audio.
+5. New fundamentals clips that fail at baseline enter
+   `_KNOWN_GAPS_BY_ID` (report-but-don't-gate), exactly like the odd-meter
+   and 174 BPM entries — the gap list is the program's burn-down chart.
+
+## PR sequence
+
+### PR-G1 — Coverage audit (docs only) — this PR
+
+`audits/genre-coverage-2026-07-13.md` + this plan. No behavior change.
+
+### PR-G2 — Synthetic corpus expansion + gate promotion
+
+Extend `scripts/build_synthetic_corpus.py` with known-truth clips spanning
+the missing rhythm/tempo surface, and promote them into the active
+fundamentals manifest (`fundamentals_eval_manifest.synthetic.json`):
+
+1. **2-step broken kick** (~132 BPM): kick on 1 and the "and" of 2 (or
+   equivalent classic pattern), snare on 2 and 4 — truth: bpm, 4/4,
+   beatGrid, downbeats. Exercises the kick-accent downbeat phase heuristic
+   off its four-on-the-floor assumption.
+2. **Halftime feel** (kick 1 / snare 3) at 140 and 85 — truth pins the
+   *notated* tempo; this is the deliberate tempo-octave trap.
+3. **Triplet/shuffle swing** (66.7% triplet target) and **16th-note
+   shuffle** clips (UKG-style) — expected initially to land in
+   `knownGaps` because `compute_swing_detail` only reads the 8th grid.
+4. **Breakbeat syncopation**: sampled-breaks-style pattern (syncopated
+   kick, ghost snares) — beat/downbeat truth.
+5. **Sparse/beatless ambient**: slow pad, no percussion — the *abstention*
+   clip. Active checks assert what the pipeline should NOT claim:
+   low/absent tempo confidence, `swingDetail` None or `direction:
+   straight` with low confidence, meter on the assumed-4/4 fallback, key
+   still correct on tonal pads.
+6. **Tempo extremes**: grid clips at 85, 150, 174 (exists), 190. 190 and
+   likely 150+ enter `knownGaps` at baseline (octave halving).
+
+Every new clip kind gets `_EXPECTED_KEYS_BY_KIND` / `_THRESHOLDS_BY_KIND`
+entries and deterministic-render `--check` coverage. Measured baseline
+failures are registered in `_KNOWN_GAPS_BY_ID` with the owning fix PR named
+in a comment. This PR changes no product code.
+
+### PR-G3 — Tempo octave handling at extremes (surfacing first)
+
+Surface octave evidence additively: candidate tempos (measured, ×2, ÷2 with
+their supporting evidence — onset-rate consistency, beat-grid density,
+genre-prior band) as a new full-mode field; never rewrite `bpm`. Gate: the
+PR-G2 halftime + extreme clips flip from `knownGaps` to active passes on
+the *evidence* field (does the true octave appear as a candidate with
+dominant support?). Promotion of any evidence-driven correction into the
+shipped `bpm` requires a frozen `incorporations/` decision doc measured on
+GiantSteps tempo (staged locally, 664 clips) — same protocol as the
+key-ensemble gate.
+
+### PR-G4 — Meter/downbeats on non-4/4 and broken kicks
+
+The measured weak layer (audit baseline 2026-07-03). Improve
+`analyze_time_signature`'s discrimination (the 20% margin conservatism) and
+`_compute_downbeat_phase`'s kick-on-downbeat assumption using the meter
+evidence field (Phase B) plus snare/backbeat evidence. Gate: PR-G2's
+2-step/breakbeat clips + the existing odd-meter knownGaps burn down without
+regressing the 29 existing clips or the golden. beat_this (venv-eval,
+adopt_pending_asa_slice) remains the candidate backend for the bigger jump;
+its gate stays the frozen beat-this doc, not this PR.
+
+### PR-G5 — Swing on the 16th grid (2-step shuffle)
+
+Extend `compute_swing_detail` to detect the dominant swing grid (8th vs
+16th) instead of hardwiring `"8th"` — additive: `gridResolution` already
+exists as a field; new evidence lands beside it. Gate: PR-G2 shuffle/16th
+clips activate; existing 8th-grid swing clips (50–66%) must not move.
+
+### PR-G6 — Key honesty on sparse/ambient material
+
+No algorithm change expected: the ensemble (keep-EDMA) already ships. This
+PR adds the abstention/hedge checks from the PR-G2 ambient clips (key
+confidence + ensemble disagreement behavior on drones/beatless pads) and
+fixes only what those checks prove wrong, surfacing-only.
+
+### PR-G7 — Genre classifier + mix-doctor widening (both sides, one PR)
+
+Add signatures for the missing families — trap, footwork, hardstyle,
+gabber/hardcore, electro, 2-step, speed garage, halftime, downtempo — and
+tune uk-garage's sidechain assumption. IDM is a verified-abstain, not a
+signature. Must move together (tripwire #3):
+
+1. `_GENRE_SIGNATURES` + `_GENRE_FAMILY_MAP` (backend),
+2. the `genreFamily` union in `src/types/measurement.ts:652` + the
+   `validFamilies` allowlist in `backendPhase1Client.ts:1134`,
+3. `genreProfiles.ts` mix-doctor profiles with *sourced* targets
+   (reference-track measurements, not invented numbers),
+4. golden re-baseline if any fixture's classification shifts.
+
+Gate: classification checks on PR-G2 clips where the synthetic render is
+genre-representative; for families a synthetic clip can't honestly
+represent (gabber kick THD, trap 808 character), the PR body says so and
+the check waits for real-audio fixtures.
+
+## Real-audio validation (operator machine, network available)
+
+From `plans/owner-actions-accuracy-program.md`: GiantSteps key/tempo (604 +
+664 clips) and GTZAN + GTZAN-Rhythm are **already staged locally**
+(2026-07-05). Remaining:
+
+1. **Demucs weights fetch** — first `--separate` run on this machine pulls
+   the torchaudio Hybrid Demucs weights, activating the stem-aware chord
+   path (PR-B4) that cloud sessions could never exercise (falls back to
+   full-mix without weights). Then re-run the fundamentals multi clips and
+   record whether the stem path holds its ≥0.45 floor.
+2. **GiantSteps tempo re-run** whenever PR-G3 surfaces octave evidence —
+   the real-audio octave-error rate is the gate metric.
+3. **ASA 18-clip beat slice** stays the owner's hand-annotation task (the
+   one deliberately non-automatable item).
+
+## Definition of done
+
+The fundamentals summary gates on clips spanning broken-kick, halftime,
+shuffle, breakbeat, beatless, and 85–190 BPM material — with
+`_KNOWN_GAPS_BY_ID` empty or every remaining entry tied to a named,
+frozen-gate decision doc — and the genre surfaces (signatures, family map,
+TS union, mix-doctor profiles) cover the priority families on both sides of
+the contract.