Skip to content

feat(eval): genre-generalization synthetic corpus + honesty checks (PR-G2) - #211

Merged
slittycode merged 1 commit into
mainfrom
accuracy/genre-corpus-expansion
Jul 13, 2026
Merged

feat(eval): genre-generalization synthetic corpus + honesty checks (PR-G2)#211
slittycode merged 1 commit into
mainfrom
accuracy/genre-corpus-expansion

Conversation

@slittycode

Copy link
Copy Markdown
Owner

What

PR-G2 of the genre-generalization program (plans/genre-generalization-program.md, stacked on #210): expands the synthetic fundamentals corpus from 29 to 38 clips so the accuracy gates span broken-kick, halftime, shuffle, breakbeat, beatless, and 85–190 BPM material instead of only steady four-on-the-floor 4/4.

New clips (all deterministic, --check-verified)

  1. twostep_132 — classic 2-step broken kick (kick 1 + "and" of 2, backbeat snares), straight 8th hats.
  2. halftime_140 / halftime_174 — kick 1 / snare 3, 8th hats carrying the notated tempo (the tempo-octave trap).
  3. breakbeat_136 — syncopated kick pattern (anticipated second kick + "and" of 3).
  4. shuffle16_130_62 — four-on-floor kicks + 16th-grid shuffled hats at 62% (UKG shuffle).
  5. ambient_beatless_70 — two slow A-minor triads, no percussion: the abstention clip.
  6. grid_4_4_85, grid_4_4_150, grid_4_4_190 — tempo extremes.

New harness capability

fundamentals_evaluation.py gains an additive honesty check family (expected.honesty) asserting what the pipeline must NOT claim on evidence-free material: maxBpmConfidence, swingDetailAbsent, meterSources. Unit-tested in tests/test_fundamentals_evaluation.py.

Measured baseline (calibrated on this machine, 2026-07-13)

Passing at baseline — now permanently gating: 2-step (beatGrid F1 0.954, downbeats 0.875), breakbeat (0.954/0.875), halftime at 140 (139.9 BPM measured — the hat grid carries it), 85 and 150 BPM grids, and the beatless-ambient abstention checks (bpmConfidence 0.114 ≤ 0.4, swingDetail absent, meter on the assumed-4/4 fallback, key still exact).

Failing at baseline — registered as knownGaps (report-but-don't-gate), each with a named owner:

Clip Check Measured Owner
grid_4_4_190 tempo / beatGrid / downbeats 94.9 BPM (octave-halved); grid degrades to 0.625/0.667 — worse than at 174, where the grid survived PR-G3
halftime_174 tempo / beatGrid / downbeats 117.0 BPM — a 2:3 ratio error, not clean halving; grid follows (0.370/0.308) PR-G3
shuffle16_130_62 swing:swingPercent None — 16th-grid shuffle is invisible to the 8th-IOI-windowed swing measurement PR-G5

Full-manifest run: 38/38 evaluated, 94 active checks passed, 0 failed, 15 informational.

What this PR does NOT do

  1. No product-path code changes — analyze.py and friends untouched; no schema/type/golden changes.
  2. CI's default 4-clip manifest is unchanged; the 38-clip synthetic manifest remains the local accuracy gate (same split as the 2026-07-03 baseline).
  3. The knownGaps failures are documented, not fixed — PR-G3 (tempo octave) and PR-G5 (16th-grid swing) own them.

Validation

  1. Backend: ./venv/bin/python -m unittest discover -s tests — 1340 tests, OK (3 skipped).
  2. Synthetic corpus determinism: build_synthetic_corpus.py --check — every clip double-rendered byte-identical.
  3. Full synthetic eval: 38/38, 94 passed / 0 failed / 15 informational.
  4. CI-mirror fundamentals gate (default manifest): 15/15, allPassed.
  5. UI: npm run verify — 51 smoke passed (re-run solo after an initial load-flaked run; zero UI code touched).

Everything validated here is synthetic-corpus truth; real-audio confirmation of the tempo-octave behavior lands with the PR-G3 GiantSteps tempo re-run (corpus already staged locally).

🤖 Generated with Claude Code

@slittycode

Copy link
Copy Markdown
Owner Author

Note on CI: the main CI workflow only triggers on PRs based on main, so this stacked PR shows only claude-review. Full validation ran locally (backend 1340 tests OK, synthetic eval 38/38 with 94/0 active checks, CI-mirror fundamentals gate 15/15, UI verify 51 smoke). After #210 merges, rebasing this branch onto main and pushing will arm the full CI run before merge.

Base automatically changed from accuracy/genre-coverage-audit to main July 13, 2026 05:51
…time, shuffle, beatless, tempo extremes (PR-G2)

Expands the synthetic fundamentals corpus 29 -> 38 clips: 2-step and
breakbeat broken-kick patterns, halftime at 140/174 (the tempo-octave
trap), a 16th-grid shuffle, a beatless-ambient abstention clip, and
85/150/190 BPM grid extremes. Adds an additive honesty check family to
the eval harness (maxBpmConfidence / swingDetailAbsent / meterSources)
so evidence-free material gates on what the pipeline must NOT claim.

Calibrated baseline: 94 active checks pass, 0 fail; three measured
failures registered as knownGaps with owners — 190 BPM octave halving
and halftime-174's 2:3 tempo error (PR-G3), 16th-shuffle invisible to
the 8th-IOI swing window (PR-G5). 2-step, breakbeat, halftime-140, and
ambient abstention all pass at baseline. No product-path changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@slittycode
slittycode force-pushed the accuracy/genre-corpus-expansion branch from fc0fc92 to deaa72a Compare July 13, 2026 05:52
@slittycode
slittycode merged commit 9011f58 into main Jul 13, 2026
4 checks passed
@slittycode
slittycode deleted the accuracy/genre-corpus-expansion branch July 13, 2026 05:59
slittycode added a commit that referenced this pull request Jul 13, 2026
…R-G4

# Conflicts:
#	apps/backend/scripts/build_synthetic_corpus.py
#	apps/backend/tests/fixtures/fundamentals_eval_manifest.synthetic.json
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant