feat(eval): GiantSteps real-audio harness + Phase A operator runbooks - #203
Merged
Merged
Conversation
Completes accuracy-program Phase A (ground truth first):
- giantsteps_evaluation.py + scripts/evaluate_giantsteps.py: score Phase 1
key/tempo against the GiantSteps Key + Tempo datasets (real annotated
Beatport clips). MIREX weighted key score (mir_eval convention:
1.0/0.5 fifth-above/0.3 relative/0.2 parallel), tempo Acc1/Acc2 (±4%,
octave/triple family), enharmonic folding shared with the fundamentals
harness. Runs analyze.py --fast --yes; env passthrough so ASA_*_BACKEND
experiments score identically. Zero evaluable clips exits 1
('underpowered') — never vacuous green. Research-only.
- scripts/fetch_giantsteps.py: operator-run corpus fetcher — clones the
MIT-licensed annotation repos, downloads Beatport previews (JKU mirror
fallback), verifies upstream MD5s; --verify-only re-checks. Audio is
gitignored, never committed (tests/fixtures/giantsteps/README.md).
- build_beat_manifest.py --asa-slice: builds the frozen beat gate's
MIN_CLIPS_ASA electronic slice from a flat asa/{audio,annotations}/
layout; beat_tracks README documents hand-annotated GiantSteps previews
as the contamination-safe source (GiantSteps has no upstream beat
annotations, so it is outside beat_this's training data).
- tests/ground_truth: deleted the 10 fake cache stubs that would have
silently corrupted any calibration report; README records the new
corpus strategy (synthetic + public previews; owner library not needed).
- plans/owner-actions-accuracy-program.md: the four local operator tasks
(GiantSteps fetch, GTZAN + venv-eval, ~1-2h ASA-slice annotation,
later calibration tracks) with exact commands and what each unblocks.
Tests: tests/test_giantsteps_evaluation.py (11 — MIREX table, Acc1/Acc2
boundaries, corpus loading, underpowered semantics, mocked runners).
Backend suite: 1294 OK.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T4wfz87k6kzJqkKLKZE7YL
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the accuracy program's Phase A (ground truth first), following #202's synthetic-corpus foundation. All research-only — nothing on the product path changes.
What's here
GiantSteps evaluation harness (
giantsteps_evaluation.py+scripts/evaluate_giantsteps.py): scores Phase 1 key/tempo against the GiantSteps Key + Tempo datasets — ~600 expert-annotated Beatport preview clips per subset, i.e. real electronic music spanning the EDM genre surface. This is the reality check the synthetic baseline explicitly calls for.analyze.py --fast --yes(fast mode populates bpm/key), inherits the environment soASA_*_BACKENDexperiments are scored by the identical harness.status: underpowered, exit 1 — the same never-vacuous-green rule fix(eval): make the synthetic fundamentals gate real — post-#201 review #202 established.Corpus fetcher (
scripts/fetch_giantsteps.py): operator-run (cloud proxies block the audio hosts — verified). Clones the MIT-licensed annotation repos, downloads previews with a mirror fallback, verifies upstream MD5 checksums,--verify-onlyfor re-checks. Audio is gitignored and never committed; runbook + licence notes intests/fixtures/giantsteps/README.md.ASA-slice tooling (
build_beat_manifest.py --asa-slice): builds the frozen beat gate'sMIN_CLIPS_ASAelectronic subset from hand-annotated clips. The beat_tracks README now documents GiantSteps previews as the contamination-safe annotation source (GiantSteps carries no upstream beat annotations, so it's outside beat_this's training data) — resolving the gate's ASA-slice requirement without the owner's personal library.Calibration corpus cleanup: deleted the 10 fake
tests/ground_truth/cache/stubs (hand-crafted JSON that did not represent real analyzer output and would have silently corrupted any calibration report); README records the new corpus strategy.Owner runbook (
plans/owner-actions-accuracy-program.md): the four local tasks — GiantSteps fetch, GTZAN + venv-eval, the single ~1–2 h hand-annotation task, later calibration tracks — with exact commands and which gate each unblocks.Tests
tests/test_giantsteps_evaluation.py— 11 tests: MIREX score table (incl. enharmonic exact), Acc1 tolerance boundaries, octave/triple Acc2 family, annotation parsing, missing-audio handling, underpowered semantics, max-clips, all with mocked runners (no network, no audio). Full backend suite: 1294 tests OK.What this unblocks
With the operator actions run locally: real-audio key/tempo accuracy numbers (feeding the PR-B3 key-ensemble gate), and the complete input set for the pre-registered beat_this ship/no-ship decision (PR-C2) — targeting the measured weak layer (meter → downbeats).
🤖 Generated with Claude Code
https://claude.ai/code/session_01T4wfz87k6kzJqkKLKZE7YL
Generated by Claude Code