Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -445,7 +445,7 @@ A quick map from intent to the right place to start:

## Backport Candidates

The full original `sonic-architect-app` backlog is **shipped** — genre profiles, Ableton device mappings (`abletonDevices.ts` — ported but **research-only** since the 2026-06-11 demotion decision: a scoring baseline for the recommendation-proof harness, imported by no product code; see the NEEDS-WIRING decision in `apps/backend/NEEDS.md`), mix doctor, eight detectors, Phase 3 audition-sample generation, and `patchSmith.ts` (Phase 3 Vital preset generation, [`apps/ui/src/services/patchSmith.ts`](apps/ui/src/services/patchSmith.ts)). See [`BACKLOG.md`](BACKLOG.md) for the landed inventory. Consult it before re-implementing genre detection, mix analysis, acid/reverb/vocal/supersaw/bass/kick detection — they're already in [`apps/backend/analyze_detection.py`](apps/backend/analyze_detection.py) and emit fields visible in `EXPECTED_TOP_LEVEL_KEYS`.
The full original `sonic-architect-app` backlog is **shipped** — genre profiles, Ableton device mappings (`abletonDevices.ts` — **product-wired 2026-07-02** as the Phase-2-off deterministic fallback via `src/services/deterministicRecommendations.ts` (cited + hedged, renders only when interpretation is unavailable), reversing the 2026-06-11 research-only demotion; the eval bridge that scores it as the harness baseline is unchanged — see `apps/backend/NEEDS.md`), mix doctor, eight detectors, Phase 3 audition-sample generation, and `patchSmith.ts` (Phase 3 Vital preset generation, [`apps/ui/src/services/patchSmith.ts`](apps/ui/src/services/patchSmith.ts)). See [`BACKLOG.md`](BACKLOG.md) for the landed inventory. Consult it before re-implementing genre detection, mix analysis, acid/reverb/vocal/supersaw/bass/kick detection — they're already in [`apps/backend/analyze_detection.py`](apps/backend/analyze_detection.py) and emit fields visible in `EXPECTED_TOP_LEVEL_KEYS`.

## Companion Agent Docs

Expand Down
22 changes: 21 additions & 1 deletion apps/backend/JSON_SCHEMA.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ Conventions:

Top-level keys:

`phase1Version`, `bpm`, `bpmConfidence`, `bpmPercival`, `bpmAgreement`, `bpmDoubletime`, `bpmSource`, `bpmRawOriginal`, `key`, `keyConfidence`, `keyProfile`, `tuningFrequency`, `tuningCents`, `timeSignature`, `timeSignatureSource`, `timeSignatureConfidence`, `durationSeconds`, `sampleRate`, `lufsIntegrated`, `lufsRange`, `lufsMomentaryMax`, `lufsShortTermMax`, `lufsCurve`, `truePeak`, `crestFactor`, `dynamicSpread`, `monoCompatible`, `plr`, `dynamicCharacter`, `textureCharacter`, `stereoDetail`, `spectralBalance`, `spectralBalanceTimeSeries`, `spectralDetail`, `stemAnalysis`, `transientDensityDetail`, `saturationDetail`, `snareDetail`, `hihatDetail`, `rhythmDetail`, `melodyDetail`, `transcriptionDetail`, `pitchDetail`, `grooveDetail`, `beatsLoudness`, `rhythmTimeline`, `sidechainDetail`, `reverbDetail`, `vocalDetail`, `acidDetail`, `supersawDetail`, `bassDetail`, `kickDetail`, `genreDetail`, `effectsDetail`, `synthesisCharacter`, `danceability`, `structure`, `arrangementDetail`, `segmentLoudness`, `segmentSpectral`, `segmentStereo`, `segmentKey`, `chordDetail`, `perceptual`, `essentiaFeatures`.
`phase1Version`, `fundamentalsQuality`, `bpm`, `bpmConfidence`, `bpmPercival`, `bpmAgreement`, `bpmDoubletime`, `bpmSource`, `bpmRawOriginal`, `key`, `keyConfidence`, `keyProfile`, `tuningFrequency`, `tuningCents`, `timeSignature`, `timeSignatureSource`, `timeSignatureConfidence`, `durationSeconds`, `sampleRate`, `lufsIntegrated`, `lufsRange`, `lufsMomentaryMax`, `lufsShortTermMax`, `lufsCurve`, `truePeak`, `crestFactor`, `dynamicSpread`, `monoCompatible`, `plr`, `dynamicCharacter`, `textureCharacter`, `stereoDetail`, `spectralBalance`, `spectralBalanceTimeSeries`, `spectralDetail`, `stemAnalysis`, `transientDensityDetail`, `saturationDetail`, `snareDetail`, `hihatDetail`, `rhythmDetail`, `melodyDetail`, `transcriptionDetail`, `pitchDetail`, `grooveDetail`, `beatsLoudness`, `rhythmTimeline`, `sidechainDetail`, `reverbDetail`, `vocalDetail`, `acidDetail`, `supersawDetail`, `bassDetail`, `kickDetail`, `genreDetail`, `effectsDetail`, `synthesisCharacter`, `danceability`, `structure`, `arrangementDetail`, `segmentLoudness`, `segmentSpectral`, `segmentStereo`, `segmentKey`, `chordDetail`, `perceptual`, `essentiaFeatures`.

**Shared (fast + full) vs full-only.** Fast-mode output is asserted byte-for-byte against the `EXPECTED_TOP_LEVEL_KEYS` set in [`tests/test_analyze.py`](tests/test_analyze.py). Full mode emits those keys *plus* a handful of detail-only fields that are deliberately absent from the shared snapshot: `keyProfile`, `tuningFrequency`, `tuningCents`, `lufsMomentaryMax`, `lufsShortTermMax`, and `pitchDetail`. When changing the schema, update both this list and `EXPECTED_TOP_LEVEL_KEYS`; full-only fields stay out of that set on purpose. See CLAUDE.md tripwire #4. Schema changes are also cross-checked executably against the frontend fixture and parser by `apps/ui/tests/services/phase1ContractParity.test.ts`, driven by the golden snapshot's `topLevelKeys`/`keyTree` — regenerating the golden (`UPDATE_PHASE1_GOLDEN=1`) is what arms the nested half of that gate.

Expand Down Expand Up @@ -150,6 +150,7 @@ Current server behavior that affects schema expectations:
| Field | Type | Description | Units / Scale | LLM interpretation note |
|---|---|---|---|---|
| `phase1Version` | `string` | Phase 1 JSON schema version, e.g. `"phase1.v2"`. | identifier | Lets consumers detect the schema generation. v2 changed `truePeak` (now dBTP) and `bpmConfidence` (now 0-1) units vs v1. |
| `fundamentalsQuality` | `object` | Local-only authority summary for tempo, beat grid, downbeats, meter, key, chords, percussion, and monophonic transcription. | `fundamentals-quality.v1` | Never produced by an LLM. Use its status and plain-English messages to avoid presenting weak measurements as settled facts. |
| `bpm` | `float \| null` | Primary tempo estimate from `RhythmExtractor2013`. | beats per minute | Main tempo anchor for Ableton project tempo and clip warp assumptions. |
| `bpmConfidence` | `float \| null` | Tempo confidence from `RhythmExtractor2013`, normalized to 0-1. | 0-1 (Phase 1 v2: raw Essentia confidence ~0-5.32 divided by 5.0 and clamped) | Higher = stronger rhythmic periodicity. Below 0.4 (raw < 2.0) suggests an ambiguous pulse or half/double-time content; hedge tempo claims there. |
| `key` | `string \| null` | Global key label from `KeyExtractor` (`edma` profile), e.g. `"A Minor"`. | categorical | Starting point for harmonic reconstruction; validate by ear against bass/chord roots. |
Expand All @@ -163,6 +164,25 @@ Current server behavior that affects schema expectations:
| `tuningFrequency` | `float \| null` | Estimated tuning reference frequency from spectral peak analysis. | Hz | Deviation from 440 Hz helps detect detuned material or concert-pitch variants. |
| `tuningCents` | `float \| null` | Tuning offset from A440 in cents. | cents | Positive = sharp of A440, negative = flat. Useful for pitch-correcting reconstructions. |

### `fundamentalsQuality`

`fundamentalsQuality` is a summary of trust, not a separate analyzer. It reads
the local Phase 1 output and labels each musical fundamental as
`authoritative`, `ambiguous`, `failed`, or `not_run`.

| Field | Type | Description |
|---|---|---|
| `fundamentalsQuality.schemaVersion` | `string` | Currently `"fundamentals-quality.v1"`. |
| `fundamentalsQuality.targetProfile` | `string` | Currently `"electronic_ableton_v1"`. |
| `fundamentalsQuality.localOnly` | `bool` | Always `true`; this block is derived from local DSP/model outputs only. |
| `fundamentalsQuality.llmExcluded` | `bool` | Always `true`; Phase 2 must never write or override this block. |
| `fundamentalsQuality.overallStatus` | `string` | Worst-case status across domains. |
| `fundamentalsQuality.domains.<domain>.status` | `string` | Domain status for `tempo`, `beatGrid`, `downbeats`, `meter`, `key`, `chords`, `percussion`, and `transcription`. |
| `fundamentalsQuality.domains.<domain>.plainEnglish` | `string` | Short user-facing explanation of how much to trust that domain. |
| `fundamentalsQuality.domains.<domain>.source` | `string \| null` | Local algorithm/source label, e.g. `rhythm_extractor_confirmed`, `edma`, or `librosa_viterbi`. |
| `fundamentalsQuality.domains.<domain>.confidence` | `float \| null` | Normalized confidence when a domain has a scalar confidence. |
| `fundamentalsQuality.domains.<domain>.evidence` | `object` | Small domain-specific evidence object, such as BPM agreement, chord disagreement, hit counts, or full-mix fallback. |

---

## BPM Cross-Check
Expand Down
11 changes: 11 additions & 0 deletions apps/backend/NEEDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,17 @@ resolved the wire-or-demote choice: **demote**. Standing consequences:
change (it remains a legitimate baseline-fairness tweak if the comparison
ever needs it).

**⤴ REVERSED 2026-07-02 (owner-approved): wired as the Phase-2-off fallback.**
`apps/ui/src/services/deterministicRecommendations.ts` now projects the real
Phase 1 payload into `AudioFeatures` (the render-gated projection flagged
above — it lives there, not in `analyzer.ts`), attaches per-card Phase 1
citations, and hedges/skips cards whose fundamentals are weak;
`DeterministicAdviceSection` renders it **only** when Phase 2 interpretation
is unavailable (off, failed, interrupted, or terminally empty), labeled
"Deterministic Baseline · NO AI". The eval bridge and its uncited-by-design
scoring are unchanged. Standing consequence #2 stands for the Phase-2-on
path: when interpretation runs, its output is the product surface.

---

## Sub-goal 3 — the Gemini verdict
Expand Down
9 changes: 9 additions & 0 deletions apps/backend/analyze.py
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@
TorchcrepeBackend,
analyze_transcription,
)
from fundamentals_quality import build_fundamentals_quality # noqa: E402


def analyze_structure(
Expand Down Expand Up @@ -1531,6 +1532,10 @@ def main():
"perceptual": result.get("perceptual"),
"essentiaFeatures": result.get("essentiaFeatures"),
}
output["fundamentalsQuality"] = build_fundamentals_quality(
output,
analysis_mode="fast",
)
_emit_progress_marker("complete", "Analysis complete.", 1.0)
print("Done.", file=sys.stderr)
print(json.dumps(output, indent=2))
Expand Down Expand Up @@ -1946,6 +1951,10 @@ def main():
"perceptual": result.get("perceptual"),
"essentiaFeatures": result.get("essentiaFeatures"),
}
output["fundamentalsQuality"] = build_fundamentals_quality(
output,
analysis_mode="standard" if run_standard else "full",
)

# Conditional MT3 namespace — *absent* (not null) by default. Only added
# when ASA_ENABLE_MT3=1 AND the transcribe() call succeeded. See the gate
Expand Down
15 changes: 13 additions & 2 deletions apps/backend/analyze_core.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,8 +135,14 @@ def analyze_bpm(
}


def analyze_key(mono: np.ndarray) -> dict:
"""Extract musical key and confidence using KeyExtractor with EDMA profile."""
def analyze_key(mono: np.ndarray, *, include_tuning: bool = True) -> dict:
"""Extract musical key and confidence using KeyExtractor with EDMA profile.

``include_tuning`` gates the per-frame spectral-peak tuning-frequency pass,
which is the dominant cost of key analysis on a full track. Fast mode does
not surface ``tuningFrequency``/``tuningCents``, so it passes ``False`` to
skip that work while keeping the return shape identical.
"""
try:
extractor = es.KeyExtractor(profileType="edma")
key, scale, strength = extractor(mono)
Expand All @@ -147,6 +153,11 @@ def analyze_key(mono: np.ndarray) -> dict:
"keyProfile": "edma",
}

if not include_tuning:
result["tuningFrequency"] = None
result["tuningCents"] = None
return result

try:
frame_size = 2048
hop_size = 1024
Expand Down
86 changes: 15 additions & 71 deletions apps/backend/analyze_fast.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,15 @@

import sys
import numpy as np
import essentia.standard as es

from analyze import apply_bpm_correction
from analyze_core import (
analyze_bpm,
analyze_key,
analyze_loudness,
analyze_time_signature,
analyze_true_peak,
extract_rhythm,
)


def analyze_fast(mono: np.ndarray, sample_rate: int = 44100) -> dict:
Expand All @@ -37,89 +43,27 @@ def analyze_fast(mono: np.ndarray, sample_rate: int = 44100) -> dict:
"""
result = {}

# Run RhythmExtractor2013 for BPM and beat data
try:
rhythm_extractor = es.RhythmExtractor2013(method="multifeature")
bpm, beats, bpm_confidence, _, _ = rhythm_extractor(mono)
result["bpm"] = round(float(bpm), 2) if bpm is not None else None
# Phase 1 v2: normalize RhythmExtractor2013 confidence (~0-5.32) to 0-1.
result["bpmConfidence"] = (
round(min(max(float(bpm_confidence), 0.0) / 5.0, 1.0), 3)
if bpm_confidence is not None
else None
)
except Exception as e:
print(f"[warn] Fast mode BPM analysis failed: {e}", file=sys.stderr)
result["bpm"] = None
result["bpmConfidence"] = None
beats = None

# Percival BPM for cross-check
try:
percival = es.PercivalBpmEstimator()
bpm_percival = percival(mono)
result["bpmPercival"] = round(float(bpm_percival), 2) if bpm_percival is not None else None
except Exception as e:
print(f"[warn] Fast mode Percival BPM failed: {e}", file=sys.stderr)
result["bpmPercival"] = None

# BPM agreement
if result.get("bpm") is not None and result.get("bpmPercival") is not None:
result["bpmAgreement"] = abs(result["bpm"] - result["bpmPercival"]) < 2.0
else:
result["bpmAgreement"] = None

# BPM correction (shared helper)
correction = apply_bpm_correction(result["bpm"], result["bpmPercival"], result["bpmAgreement"])
result.update(correction)

# Key extraction
try:
key_extractor = es.KeyExtractor(profileType="edma")
key, scale, strength = key_extractor(mono)
result["key"] = f"{key} {scale}".title() if key and scale else None
result["keyConfidence"] = round(float(strength), 3) if strength is not None else None
except Exception as e:
print(f"[warn] Fast mode key analysis failed: {e}", file=sys.stderr)
result["key"] = None
result["keyConfidence"] = None

# Time signature (default to 4/4 if we have rhythm data)
if beats is not None and len(beats) > 0:
result["timeSignature"] = "4/4"
result["timeSignatureSource"] = "assumed_four_four"
result["timeSignatureConfidence"] = 0.0
else:
result["timeSignature"] = None
result["timeSignatureSource"] = None
result["timeSignatureConfidence"] = None

# Duration
rhythm_data = extract_rhythm(mono)
result.update(analyze_bpm(rhythm_data, mono, sample_rate))
result.update(analyze_key(mono, include_tuning=False))
result.update(analyze_time_signature(rhythm_data, mono=mono, sample_rate=sample_rate))
result["durationSeconds"] = round(float(len(mono) / sample_rate), 3)
result["sampleRate"] = sample_rate

# Basic loudness (LUFS integrated and range)
try:
# Need stereo for LUFS — Essentia expects shape (N, 2), not (2, N)
stereo = np.stack([mono, mono], axis=-1)
loudness_algo = es.LoudnessEBUR128(sampleRate=sample_rate)
_, _, lufs_integrated, lufs_range = loudness_algo(stereo)
result["lufsIntegrated"] = round(float(lufs_integrated), 2) if lufs_integrated is not None else None
result["lufsRange"] = round(float(lufs_range), 2) if lufs_range is not None else None
result.update(analyze_loudness(stereo, sample_rate=sample_rate))
except Exception as e:
print(f"[warn] Fast mode loudness analysis failed: {e}", file=sys.stderr)
result["lufsIntegrated"] = None
result["lufsRange"] = None

# True peak (from stereo), emitted in dBTP (Phase 1 v2). NOTE: fast mode uses
# a plain sample peak (np.max(np.abs)), NOT the oversampled inter-sample true
# peak the standard path computes via Essentia's TruePeakDetector — so this is
# a sample-peak approximation and may read lower than the standard path on
# material with real inter-sample overs.
# True peak (from stereo), emitted in dBTP (Phase 1 v2).
try:
if stereo is not None:
max_peak = float(np.max(np.abs(stereo)))
result["truePeak"] = round(20.0 * np.log10(max_peak), 1) if max_peak > 0 else None
result.update(analyze_true_peak(stereo))
else:
result["truePeak"] = None
except Exception as e:
Expand Down
Loading
Loading