feat(eval): genre-generalization synthetic corpus + honesty checks (PR-G2) - #211
Merged
Conversation
Owner
Author
|
Note on CI: the main CI workflow only triggers on PRs based on |
…time, shuffle, beatless, tempo extremes (PR-G2) Expands the synthetic fundamentals corpus 29 -> 38 clips: 2-step and breakbeat broken-kick patterns, halftime at 140/174 (the tempo-octave trap), a 16th-grid shuffle, a beatless-ambient abstention clip, and 85/150/190 BPM grid extremes. Adds an additive honesty check family to the eval harness (maxBpmConfidence / swingDetailAbsent / meterSources) so evidence-free material gates on what the pipeline must NOT claim. Calibrated baseline: 94 active checks pass, 0 fail; three measured failures registered as knownGaps with owners — 190 BPM octave halving and halftime-174's 2:3 tempo error (PR-G3), 16th-shuffle invisible to the 8th-IOI swing window (PR-G5). 2-step, breakbeat, halftime-140, and ambient abstention all pass at baseline. No product-path changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
slittycode
force-pushed
the
accuracy/genre-corpus-expansion
branch
from
July 13, 2026 05:52
fc0fc92 to
deaa72a
Compare
slittycode
added a commit
that referenced
this pull request
Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
PR-G2 of the genre-generalization program (
plans/genre-generalization-program.md, stacked on #210): expands the synthetic fundamentals corpus from 29 to 38 clips so the accuracy gates span broken-kick, halftime, shuffle, breakbeat, beatless, and 85–190 BPM material instead of only steady four-on-the-floor 4/4.New clips (all deterministic,
--check-verified)twostep_132— classic 2-step broken kick (kick 1 + "and" of 2, backbeat snares), straight 8th hats.halftime_140/halftime_174— kick 1 / snare 3, 8th hats carrying the notated tempo (the tempo-octave trap).breakbeat_136— syncopated kick pattern (anticipated second kick + "and" of 3).shuffle16_130_62— four-on-floor kicks + 16th-grid shuffled hats at 62% (UKG shuffle).ambient_beatless_70— two slow A-minor triads, no percussion: the abstention clip.grid_4_4_85,grid_4_4_150,grid_4_4_190— tempo extremes.New harness capability
fundamentals_evaluation.pygains an additivehonestycheck family (expected.honesty) asserting what the pipeline must NOT claim on evidence-free material:maxBpmConfidence,swingDetailAbsent,meterSources. Unit-tested intests/test_fundamentals_evaluation.py.Measured baseline (calibrated on this machine, 2026-07-13)
Passing at baseline — now permanently gating: 2-step (beatGrid F1 0.954, downbeats 0.875), breakbeat (0.954/0.875), halftime at 140 (139.9 BPM measured — the hat grid carries it), 85 and 150 BPM grids, and the beatless-ambient abstention checks (bpmConfidence 0.114 ≤ 0.4, swingDetail absent, meter on the assumed-4/4 fallback, key still exact).
Failing at baseline — registered as
knownGaps(report-but-don't-gate), each with a named owner:Full-manifest run: 38/38 evaluated, 94 active checks passed, 0 failed, 15 informational.
What this PR does NOT do
analyze.pyand friends untouched; no schema/type/golden changes.knownGapsfailures are documented, not fixed — PR-G3 (tempo octave) and PR-G5 (16th-grid swing) own them.Validation
./venv/bin/python -m unittest discover -s tests— 1340 tests, OK (3 skipped).build_synthetic_corpus.py --check— every clip double-rendered byte-identical.npm run verify— 51 smoke passed (re-run solo after an initial load-flaked run; zero UI code touched).Everything validated here is synthetic-corpus truth; real-audio confirmation of the tempo-octave behavior lands with the PR-G3 GiantSteps tempo re-run (corpus already staged locally).
🤖 Generated with Claude Code