Merge open branches into main: fix stack (#54+#55+#56+#58), docs refresh (#57), nightly audit doc

CHANGELOG.md

@@ -5,6 +5,14 @@ All notable changes to `asa` are documented here.
 ## Unreleased
 
 ### Added
+- **Phase 3 audition samples** (`POST/GET /api/analysis-runs/{run_id}/samples`): on-demand heuristic WAV/MIDI clips derived from Phase 1 measurements (and Phase 2 context when available), with a citation manifest. PyTheory generates the musical plan, FluidSynth (with sine-additive fallback) renders audio, NumPy synthesizes drum one-shots. Not part of the staged-execution queue. See [`docs/SAMPLE_GENERATION.md`](docs/SAMPLE_GENERATION.md).
+- **URL-mode ingestion** for `POST /api/analysis-runs`: provide a public `http`/`https` `url` form field instead of a multipart `track`. SSRF-guarded against private/loopback/link-local addresses; streams through the shared 100 MiB cap. See [`apps/backend/url_ingest.py`](apps/backend/url_ingest.py).
+- **Operator `X-Admin-Key` DELETE bypass** for `DELETE /api/analysis-runs/{run_id}`: when `SONIC_ANALYZER_ADMIN_KEY` is set, operators can purge any run; otherwise owner-only deletes apply. Admin path is closed by default.
+- **`GET /api/analysis-runs/{run_id}/source-audio`**: owner-only re-serve of the originally ingested audio (no admin bypass). Saves a round-trip vs looking up the source artifact id.
+- **CSV export** of Phase 1 time-series fields: `GET /api/analysis-runs/{run_id}/export/csv/{field_path}`, backed by the registry in [`apps/backend/csv_export.py`](apps/backend/csv_export.py). Schema record: [`docs/adr/0001-phase1-json-schema-v1.md`](docs/adr/0001-phase1-json-schema-v1.md).
+- **Additive `publicStatus` field** on every stage in the run snapshot — collapses the eight internal stage statuses into a five-value client-facing vocabulary (`queued`, `running`, `completed`, `failed`, `interrupted`, plus `null` for `not_requested`). See [`apps/backend/stage_status.py`](apps/backend/stage_status.py).
+- **Reassigned spectrogram** as an opt-in spectral enhancement (`POST /api/analysis-runs/{run_id}/spectral-enhancements/reassigned`): sharper transient/frequency localization via `librosa.reassigned_spectrogram`. Joins existing `cqt`, `hpss`, `onset`, `chroma_interactive`.
+- **Chain-of-custody audit pass**: recommendations-first information architecture, applied-recommendations tracker, and grammar post-process on Phase 2 output. New components: [`CitationBlock.tsx`](apps/ui/src/components/CitationBlock.tsx), [`appliedRecommendations.ts`](apps/ui/src/services/appliedRecommendations.ts).
 - **Stem summary interpretation** (Experiment B): Gemini stem listening via Structured Outputs on separated bass and musical stems, producing bar-aligned musical descriptions with scale degrees, rhythmic patterns, and uncertainty flags
   - Backend: stem files persist as run artifacts during pitch/note work; `stem_summary` profile runs against persisted `stem_bass` and `stem_other` audio; per-stem Gemini outputs combined into one product-facing result
   - Frontend: polling loop auto-queues `stem_summary` after pitch/note translation completes; renders separate "AI stem summary" section alongside Session Musician draft notes
@@ -24,6 +32,9 @@ All notable changes to `asa` are documented here.
 - Dense techno boundary regression test (145 BPM)
 
 ### Changed
+- **Backend monolith split** into domain modules (commit `5c40dd44`): `analyze_core.py`, `analyze_audio_io.py`, `analyze_detection.py`, `analyze_estimate.py`, `analyze_rhythm.py`, `analyze_segments.py`, `analyze_structure.py`, `analyze_transcription.py`, `analyze_fast.py`. Server routes likewise split into `server_phase1.py`, `server_phase2.py`, `server_upload.py`, `server_samples.py`.
+- **`BatchedBandpass` centralization** ([`apps/backend/dsp_bandbank.py`](apps/backend/dsp_bandbank.py)): per-band 4th-order Butterworth filtering with zero-phase `filtfilt` now lives in one place instead of being duplicated across detectors.
+- **Hosted runtime foundation** landed without disturbing local mode: [`runtime_profile.py`](apps/backend/runtime_profile.py), [`worker.py`](apps/backend/worker.py), [`artifact_storage.py`](apps/backend/artifact_storage.py), [`auth_context.py`](apps/backend/auth_context.py). See [`docs/PUBLIC_HOSTING_FOUNDATION.md`](docs/PUBLIC_HOSTING_FOUNDATION.md).
 - Synthesis character labels aligned to phase2 prompt thresholds (three-tier: clean subtractive / FM-acid / wavetable-noise)
 - Removed citation instructions from Phase 2 system prompt (citations added noise, not value)
 - `dynamicCharacter` forwarded through `_build_phase1()` to Gemini Phase 2

CLAUDE.md

@@ -51,7 +51,7 @@ npm run test:smoke -- tests/smoke/upload-phase1.spec.ts
 ```bash
 ./apps/backend/scripts/bootstrap.sh         # Create/recreate venv (Python 3.11.x required)
 ./apps/backend/venv/bin/python apps/backend/server.py  # FastAPI server on 8100
-./apps/backend/venv/bin/python apps/backend/analyze.py <file> [--separate] [--transcribe] [--fast] [--yes]
+./apps/backend/venv/bin/python apps/backend/analyze.py <file> [--separate] [--transcribe] [--fast] [--standard] [--yes] [--pitch-note-only] [--stem-dir DIR] [--stem-output-dir DIR] [--pitch-note-backend BACKEND]
 
 # All backend tests (run from apps/backend/)
 ./venv/bin/python -m unittest discover -s tests
@@ -71,8 +71,32 @@ npm run test:smoke -- tests/smoke/upload-phase1.spec.ts
 TEST_FLAC_PATH=/path/to/track.flac GEMINI_API_KEY=… VITE_ENABLE_PHASE2_GEMINI=true ./scripts/test-e2e.sh  # Full live Gemini run
 ```
 
+### Scripts at a glance
+
+Operational and one-shot scripts live in two places. They are not on the request-path; reach for them when the situation calls for it.
+
+`scripts/` (repo root):
+1. `dev.sh` — full-stack dev launcher (covered above).
+2. `test-e2e.sh` / `test-e2e-integration.sh` — e2e harnesses (covered above).
+3. `calibrate_confidence.py` — threshold-sweep harness for the pitch / chord / sidechain detectors. Research-only.
+
+`apps/backend/scripts/`:
+1. `bootstrap.sh` — recreate the Python 3.11 venv (covered above).
+2. `render_upload_limit_contract.py` — regenerate the operator-facing upload-limit contract text from `upload_limits.py`. Run after changing the canonical limits.
+3. `evaluate_phase1.py`, `evaluate_polyphonic.py`, `evaluate_structure_sweep.py` — offline evaluation harnesses for the Phase 1 detector battery, the research polyphonic transcriber, and structure-segmentation parameter sweeps. Wired to `phase1_evaluation.py` / `polyphonic_evaluation.py`.
+4. `audit_pass1.py`, `genre_check.py`, `replay_catalog_validation.py` — corpus auditing and Live 12 device-catalog validation. `genre_corpus.md` is the corpus manifest.
+
 ## Architecture
 
+### Repo Layout — what's on the product path
+
+`apps/backend/`, `apps/ui/`, and `scripts/` are the product. The remaining top-level directories are off-path and safe to skip unless the task explicitly names them:
+
+1. `advisory/` — auditor notes and one-shot reviews (e.g. `phase1_audit/`). Not imported by either app.
+2. `experiments/` — sandbox scratch space (e.g. `crepe_test/` carries only a venv). Throwaway by intent; do not wire production code here.
+3. `docs/` — long-form rationale (`ARCHITECTURE_STRATEGY.md`, `history/`). Read these *before* structural changes; do not treat them as living API docs.
+4. `tests/ground_truth/` — labeled-corpus fixtures consumed by `scripts/calibrate_confidence.py` (see `tests/ground_truth/README.md`). Not a test suite — each app owns its own (`apps/backend/tests/`, `apps/ui/tests/`).
+
 ### Three-Layer Model
 
 ASA's hybrid architecture splits work into three layers. Read `docs/ARCHITECTURE_STRATEGY.md` before proposing changes to this structure — it records *why* the design is shaped this way.
@@ -112,7 +136,7 @@ The backend supports staged execution via `analysis_runtime.py`, which persists
 
 Run-oriented endpoints (`/api/analysis-runs*`) are the canonical interface for staged execution. Legacy `POST /api/analyze`, `POST /api/analyze/estimate`, and `POST /api/phase2` remain only as temporary compatibility wrappers — do not build new functionality on them.
 
-Phase 3 audition-sample generation (`POST/GET /api/analysis-runs/{run_id}/samples`) is **on-demand**, not a queue stage — the UI explicitly requests it after interpretation completes. See `server_samples.py` and the `sample_*.py` modules.
+Phase 3 audition-sample generation (`POST/GET /api/analysis-runs/{run_id}/samples`) is **on-demand**, not a queue stage — the UI explicitly requests it after interpretation completes. See `server_samples.py` and the `sample_*.py` modules. On the frontend, `apps/ui/src/components/SamplePlayback.tsx` is the entry point — mounted from `AnalysisResults.tsx` and driven by `src/services/sampleGenerationClient.ts` (`generateSamples` posts, `fetchSamples` retrieves).
 
 Frontend polling: `src/services/analysisRunsClient.ts` creates runs and polls stage snapshots. `src/services/analyzer.ts` orchestrates the create-run + poll loop and projects display payloads.
 
@@ -140,7 +164,7 @@ Artifact access goes through `artifact_storage.py` rather than direct disk paths
 9. **`upload_limits.py`**: Canonical 100 MiB raw-audio / 101 MiB request-envelope limits. Regenerate the operator contract via `scripts/render_upload_limit_contract.py` if numbers change.
 10. **`spectral_viz.py`**: Librosa-based spectrogram/time-series artifacts. Called after measurement; failures are non-critical.
 11. **`url_ingest.py`**: SSRF-guarded URL-mode ingestion for `POST /api/analysis-runs` — fetches a public `http`/`https` audio URL and feeds it through the same downstream pipeline as a multipart upload.
-12. **`csv_export.py`**: CSV exporters for Phase 1 time-series fields, backing `GET /api/analysis-runs/{run_id}/export/csv/{field_path}`. Keeps the route handler a thin lookup-and-serve.
+12. **`csv_export.py`**: CSV exporters for Phase 1 time-series fields, backing `GET /api/analysis-runs/{run_id}/export/csv/{field_path}`. Keeps the route handler a thin lookup-and-serve. Field paths are an explicit allowlist (`csv_export.list_supported_fields()` — currently `lufsCurve.shortTerm`, `lufsCurve.momentary`, `rhythmDetail.tempoCurve`, `spectralBalanceTimeSeries`); arbitrary descent into the payload is not supported. Example: `curl http://127.0.0.1:8100/api/analysis-runs/<run_id>/export/csv/rhythmDetail.tempoCurve -o tempo.csv`.
 13. **`stage_status.py`**: Collapses the eight internal stage statuses into the additive client-facing `publicStatus` field on every stage snapshot.
 14. **`server_samples.py` + `sample_generation.py`, `sample_theory.py`, `sample_synthesis.py`, `sample_drums.py`**: Phase 3 audition-sample generation. `sample_theory.py` builds the PyTheory musical plan, `sample_synthesis.py` renders audio (FluidSynth with sine-additive fallback), `sample_drums.py` synthesizes drum one-shots, `sample_generation.py` orchestrates and emits the citation manifest. On-demand only.
 15. **`dsp_bandbank.py` + `dsp_utils.py`**: Shared DSP primitives — `BatchedBandpass` (4th-order Butterworth bandpass bank) and cross-module utility functions.
@@ -156,17 +180,24 @@ The subprocess isolation means `analyze.py` works as a standalone CLI. Check `ap
 
 Single-page React 19 + Vite + TypeScript + Tailwind CSS v4 app with no router. View states managed via React conditionals (upload → estimate → analysis → results). Vitest for unit tests, Playwright for smoke tests.
 
-**Key service files:**
+**Key service files** (canonical transport + chain-of-custody contracts):
 
 1. **`src/services/analysisRunsClient.ts`**: Canonical transport. Creates runs against `/api/analysis-runs`, polls snapshots, fetches pitch/note translations and interpretations.
 2. **`src/services/analyzer.ts`**: Phase orchestration entry point — sequences run creation, polling, and display payload projection.
 3. **`src/services/backendPhase1Client.ts`**: Legacy multipart transport (typed error classes, `AbortController` timeouts, identity probe via `/openapi.json`). Kept for the compatibility wrappers; new flows should go through `analysisRunsClient.ts`.
-4. **`src/services/spectralArtifactsClient.ts`**: Fetches spectrogram/spectral-evolution artifacts via `/api/analysis-runs/{run_id}/artifacts/…`.
-5. **`src/services/mixDoctor.ts`**: Mix advisory logic — client-side scoring and suggestions against measured spectral balance.
-6. **`src/services/phase2Validator.ts`**: Runtime guardrail. Validates Phase 2 consistency against Phase 1 (`validateBPMConsistency`, `validateKeyConsistency`, `validateLUFSConsistency`, `validateGenreDSPConsistency`, `validateNumericBounds`).
-7. **`src/services/midi/`**: MIDI export, preview, and quantization utilities (`midiExport.ts`, `midiPreview.ts`, `quantization.ts`).
-8. **`src/types.ts`** + **`src/types/`**: `types.ts` is a barrel re-export of `./types/{measurement,interpretation,backend}.ts`. `Phase1Result` lives in `types/measurement.ts`; `AnalysisRunSnapshot` in `types/backend.ts`; `Phase2Result` in `types/interpretation.ts`.
-9. **`src/config.ts`**: Runtime resolution of `VITE_API_BASE_URL` and feature flags; falls back to `http://127.0.0.1:8100`. Supports window-level overrides (`window.__VITE_API_BASE_URL_OVERRIDE__`, `window.__VITE_ENABLE_PHASE2_GEMINI_OVERRIDE__`) for hosted deployments that inject config at runtime without a rebuild.
+4. **`src/services/httpClient.ts`**: Shared fetch helpers and request-header injection used by the run/artifact/sample clients.
+5. **`src/services/spectralArtifactsClient.ts`**: Fetches spectrogram/spectral-evolution artifacts via `/api/analysis-runs/{run_id}/artifacts/…`.
+6. **`src/services/sampleGenerationClient.ts`**: Phase 3 audition-sample POST/GET against `/api/analysis-runs/{run_id}/samples`, plus per-clip artifact streaming.
+7. **`src/services/mixDoctor.ts`**: Mix advisory logic — client-side scoring and suggestions against measured spectral balance.
+8. **`src/services/phase2Validator.ts`**: Runtime guardrail. Validates Phase 2 consistency against Phase 1 (`validateBPMConsistency`, `validateKeyConsistency`, `validateLUFSConsistency`, `validateGenreDSPConsistency`, `validateNumericBounds`).
+9. **`src/services/appliedRecommendations.ts`** + **`userLabels.ts`**: Applied-recommendations tracker and persisted-label state used by the audit overhaul.
+10. **`src/services/phase1Picker.ts`** + **`phaseLabels.ts`**: Phase-snapshot projection helpers consumed by the results surface.
+11. **`src/services/audioFile.ts`**: Client-side audio validation, blank-MIME extension fallback, preview-URL lifecycle.
+12. **`src/services/fieldAnalytics.ts`** + **`diagnosticLogs.ts`**: Instrumentation hooks and diagnostic-log capture for the request panel.
+13. **`src/services/midi/`**: MIDI export, preview, and quantization utilities (`midiExport.ts`, `midiPreview.ts`, `quantization.ts`).
+14. **`src/services/sessionMusician/`**: Session Musician helpers — `confidenceBand.ts`, `noteConversion.ts`, `renderState.ts`, `stemListeningNotes.ts`.
+15. **`src/types.ts`** + **`src/types/`**: `types.ts` is a barrel re-export of `./types/{measurement,interpretation,backend}.ts`. `Phase1Result` lives in `types/measurement.ts`; `AnalysisRunSnapshot` in `types/backend.ts`; `Phase2Result` in `types/interpretation.ts`; `./types/samples.ts` is imported directly, not through the barrel.
+16. **`src/config.ts`**: Runtime resolution of `VITE_API_BASE_URL` and feature flags; falls back to `http://127.0.0.1:8100`. Supports window-level overrides (`window.__VITE_API_BASE_URL_OVERRIDE__`, `window.__VITE_ENABLE_PHASE2_GEMINI_OVERRIDE__`) for hosted deployments that inject config at runtime without a rebuild.
 
 `AnalysisResults.tsx` is the large results surface, lazy-loaded via Suspense. Manual vendor chunks in `vite.config.ts` control bundle splitting.
 

apps/backend/AGENTS.md

@@ -7,7 +7,7 @@
 - The repo is a local Python audio-analysis service with two entry points:
   - `analyze.py`: raw CLI analyzer
   - `server.py`: FastAPI wrapper around the CLI
-- There are no repo-local Cursor rules, `.cursorrules`, or Copilot instruction files in this repo as of 2026-05-15.
+- There are no repo-local Cursor rules, `.cursorrules`, or Copilot instruction files in this repo as of 2026-05-17.
 
 ## Working Style For Agents
 
@@ -121,6 +121,7 @@ python3.11 -m venv venv
 - `phase1_evaluation.py` + `phase1_report_html.py`: Offline Phase 1 evaluation harness — deterministic-metric and detector-stability reporting, with a standalone HTML render. Not on the product path; driven by `scripts/evaluate_phase1.py`.
 - `polyphonic_evaluation.py` + `scripts/evaluate_polyphonic.py`: **Research-only.** Offline polyphonic-transcription evaluation harness, not part of the shipped product path.
 - `utils/cleanup.py`: Periodic artifact cleanup helpers used by the server background-task loop.
+- `symbolic_extract.py`: **Orphaned and broken.** Earlier worker-process entry point for pitch/note translation; superseded by `analyze.py --pitch-note-only`. Still imports a removed `BasicPitchBackend` symbol from `analyze.py`, so loading the module would raise `ImportError`. Not referenced from any other module. Slated for removal — do not extend it.
 - `tests/test_server.py`: OpenAPI and envelope contract tests.
 - `tests/test_analyze.py`: generated WAV fixture, `EXPECTED_TOP_LEVEL_KEYS` snapshot, raw payload assertions.
 - `tests/test_csv_export.py`, `tests/test_sample_*.py`, `tests/test_server_samples.py`: Coverage for CSV export and Phase 3 audition samples.

apps/backend/ARCHITECTURE.md

@@ -21,6 +21,7 @@
 | `phase1_evaluation.py` + `phase1_report_html.py` | Offline Phase 1 evaluation harness — deterministic-metric and detector-stability reporting, with a standalone HTML render. Not on the product path; driven by `scripts/evaluate_phase1.py`. |
 | `polyphonic_evaluation.py` + `scripts/evaluate_polyphonic.py` | Research-only offline polyphonic-transcription evaluation harness. Not on the product path. |
 | `utils/cleanup.py` | Periodic artifact cleanup helpers used by the server background-task loop. |
+| `symbolic_extract.py` | **Orphaned.** Earlier worker-process entry point for pitch/note translation; superseded by `analyze.py --pitch-note-only`. Currently imports a removed `BasicPitchBackend` and would fail at module load. Not invoked from anywhere else in the tree; slated for removal. |
 | `tests/test_server.py` | Contract tests for estimate, timeout, and success envelopes. |
 | `tests/test_analyze.py` | Structural snapshot tests for the raw analyzer JSON output. Owns `EXPECTED_TOP_LEVEL_KEYS` — update it whenever you add a root field. |
 | `tests/test_spectral_viz.py` | Unit tests for spectrogram generation, time-series computation, and artifact orchestration. |