diff --git a/README.md b/README.md index c00517b..03ed770 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,16 @@ This repository is the single home for PyAutoLens performance measurement. It ex Results are framed by **astronomy instrument** (HST, Euclid, JWST, …) rather than by raw pixel counts. Pixel counts are recorded too, but the headline numbers a reader sees first are the ones that map onto a real observing programme. +## Latest run-times + +The table below is auto-generated from the latest versioned artifacts under `results/`. Each row is the latest steady-state per-call cost for a likelihood path at a given instrument; numbers refresh whenever the producing scripts are rerun and committed. Hardware tier is **CPU only** today — laptop GPU and HPC GPU columns will land once `results/**` artifacts are tagged with a hardware label. + + +_No data yet — run likelihood scripts to populate. See `likelihood/README.md`._ + + +(Generator: `scripts/build_readme.py`. Run `python scripts/build_readme.py` after producing new artifacts to refresh; `--check` exits non-zero in CI if it would change anything.) + ## JAX gradients — currently out of scope Gradient profiling (`jax.grad` of the likelihood, autodiff-based optimisers) is **not yet** part of this repo. It is tracked in [`PyAutoLabs/autolens_workspace_developer/jax_profiling/gradient/`](https://github.com/PyAutoLabs/autolens_workspace_developer/tree/main/jax_profiling/gradient) and will fold into this repo in a future phase once the gradient story stabilises. @@ -54,18 +64,27 @@ Examples that already exist in the source-of-truth repo: ## Roadmap -This repo is being built in phases: +This repo is being built in phases. Phase numbers correspond to internal sub-prompts under `PyAutoLabs/PyAutoPrompt/z_features/autolens_profiling.md`. | Phase | Title | Status | |-------|-------|--------| -| 0 | Repo bootstrap (this commit) | ✓ shipped | -| 1 | Mirror JIT likelihood profiling scripts + per-section READMEs | not yet started | -| 2 | Mirror simulator profiling scripts + run-time tracking | not yet started | -| 3 | Nautilus profiling, design for sampler expansion | not yet started | -| 4 | Top-level + per-section README dashboard with instrument framing | not yet started | -| 5 | GitHub Actions for lint + profile re-runs + README refresh | not yet started | - -The full multi-phase plan lives in the internal `PyAutoLabs/PyAutoPrompt/z_features/autolens_profiling.md` tracker (not publicly readable). The high-level shape is captured above. +| 0 | Repo bootstrap | ✓ shipped | +| 1 | Mirror JIT likelihood profiling scripts + per-section READMEs | ✓ shipped | +| 2 | Mirror simulator profiling scripts + run-time tracking | ✓ shipped | +| 3 | Nautilus profiling, design for sampler expansion | ✓ shipped | +| 4 | Top-level + per-section README dashboard with instrument framing | ✓ shipped | +| 5 | GitHub Actions for lint + profile re-runs + README refresh | queued | + +### Future enhancements (Phase 4 follow-ups) + +Dashboards can grow in many directions. The list below captures candidate improvements that fit the "profiling and run-times" theme; none of them block the current dashboard from being useful. + +- **Regression-watch indicator** — colour or arrow per cell showing whether the latest cost regressed (>5%) or improved versus the previous PyAutoLens release. Needs the second-latest version per axis kept alongside the latest. Trivial to add to `scripts/build_readme.py`. +- **Per-axis version-history PNGs** — small inline plot of run-time vs PyAutoLens release version, generated from the JSON artifacts (reusing the `_developer/jax_profiling/results/jit/.../*_v.png` generator). Embeds nicely above each section table. +- **Plotly-rendered interactive timeline** — hostable on GitHub Pages once the static dashboard stabilises; lets readers hover/filter across instrument × model × release. +- **Flamegraph captures** — alongside the headline timing numbers, store a flamegraph per instrument × model for the most recent release. +- **Hardware-tier columns** — extend `scripts/build_readme.py` table renderers to show CPU / laptop GPU / HPC GPU as separate columns once result artifacts encode the hardware label (filename suffix or JSON `"hardware"` field). +- **Archive old versions** — once a script has >6 minor releases of artifacts, move the older ones to `results/archive/` so the latest views stay uncluttered. ## Related repos diff --git a/likelihood/datacube/README.md b/likelihood/datacube/README.md index a05c16a..4dc4dea 100644 --- a/likelihood/datacube/README.md +++ b/likelihood/datacube/README.md @@ -37,13 +37,13 @@ That number quantifies how much a future "shared `Lᵀ W̃ L`" optimisation woul For a realistic per-channel-distinct cube, point the loader at the workspace simulator output at `autolens_workspace/dataset/interferometer/datacube/sim_simple/`. The JIT-cost taxonomy doesn't change — it's a function of which arrays are loop-variables in `FitInterferometer`, not the data values themselves. -## Headline run-times (populated by Phase 4) +## Headline run-times (latest per dataset) -| Script | Dataset | N channels | CPU | Laptop GPU | A100 | -|--------|---------|------------|-----|------------|------| -| `delaunay.py` | SMA × 4 | 4 | _populated_ | _populated_ | _populated_ | +Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v.json` artifacts under `results/likelihood/datacube/`. Hardware tier is CPU only today. -Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v.json` artifacts under `results/likelihood/datacube/`. + +_No data yet — run a script under this folder to populate. See section README._ + ## Output diff --git a/likelihood/imaging/README.md b/likelihood/imaging/README.md index 8af3c3c..a74a6e2 100644 --- a/likelihood/imaging/README.md +++ b/likelihood/imaging/README.md @@ -37,15 +37,13 @@ XLA may fuse these differently when compiled as one program vs separate pieces, `dataset/imaging/hst/` — an HST-resolution mock (pixel scale 0.05″, 21×21 PSF) committed to this repo. Other instruments (`euclid`, `jwst`, `ao`) can be regenerated via the source-of-truth scripts at `autolens_workspace_developer/jax_profiling/dataset_setup/imaging.py` and copied into `dataset/imaging//`. -## Headline run-times (populated by Phase 4) +## Headline run-times (latest per script × instrument) -| Script | Instrument | CPU | Laptop GPU | A100 | -|--------|------------|-----|------------|------| -| `mge.py` | HST | _populated_ | _populated_ | _populated_ | -| `pixelization.py` | HST | _populated_ | _populated_ | _populated_ | -| `delaunay.py` | HST | _populated_ | _populated_ | _populated_ | +Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v.json` artifacts under `results/likelihood/imaging/`. Hardware tier is CPU only today. -Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v.json` artifacts under `results/likelihood/imaging/`. + +_No data yet — run a script under this folder to populate. See section README._ + ## Output diff --git a/likelihood/interferometer/README.md b/likelihood/interferometer/README.md index f7d3f4f..a00e097 100644 --- a/likelihood/interferometer/README.md +++ b/likelihood/interferometer/README.md @@ -27,13 +27,11 @@ The interferometer likelihood path is profiled at **full-pipeline JIT** only, no ## Headline run-times (populated by Phase 4) -| Script | Instrument | CPU | Laptop GPU | A100 | -|--------|------------|-----|------------|------| -| `mge.py` | SMA | _populated_ | _populated_ | _populated_ | -| `pixelization.py` | SMA | _populated_ | _populated_ | _populated_ | -| `delaunay.py` | SMA | _populated_ | _populated_ | _populated_ | +Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v.json` artifacts under `results/likelihood/interferometer/`. Hardware tier is CPU only today. -Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v.json` artifacts under `results/likelihood/interferometer/`. + +_No data yet — run a script under this folder to populate. See section README._ + ## Output diff --git a/likelihood/point_source/README.md b/likelihood/point_source/README.md index 63b6a76..b77d487 100644 --- a/likelihood/point_source/README.md +++ b/likelihood/point_source/README.md @@ -36,14 +36,13 @@ For both variants: `dataset/point_source/simple/` — a minimal seeded dataset with `point_dataset_positions_only.json` (4 observed image positions) and the truth `tracer.json`. Both files are committed to this repo. -## Headline run-times (populated by Phase 4) +## Headline run-times (latest per script × dataset) -| Script | Dataset | CPU | Laptop GPU | A100 | -|--------|---------|-----|------------|------| -| `image_plane.py` | simple | _populated_ | _populated_ | _populated_ | -| `source_plane.py` | simple | _populated_ | _populated_ | _populated_ | +Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v.json` artifacts under `results/likelihood/point_source/`. Hardware tier is CPU only today. **Cells may show `—` while [PyAutoLens#514](https://github.com/PyAutoLabs/PyAutoLens/issues/514) is open** — the regression assertion in both scripts is intentionally load-bearing while the upstream drift is triaged, so neither script reaches the JSON-write step in the current PyAutoLens release. -Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v.json` artifacts under `results/likelihood/point_source/`. + +_No data yet — run a script under this folder to populate. See section README._ + ## Output diff --git a/scripts/build_readme.py b/scripts/build_readme.py new file mode 100644 index 0000000..e6bbaff --- /dev/null +++ b/scripts/build_readme.py @@ -0,0 +1,434 @@ +""" +build_readme.py — refresh auto-generated tables in every README from the +latest versioned artifacts under `results/`. + +Run from the repo root: + + python scripts/build_readme.py # rewrite README tables in place + python scripts/build_readme.py --check # exit non-zero if rewriting + # would change any file (CI gate) + +Each table region in a README is delimited by sentinel comments, e.g. + + + | ... | + + +This script: + + 1. Scans `results/**/*_summary_v.json`. + 2. Parses filenames into (section, sub-folder, script, instrument, version). + 3. Picks the latest version per group via PEP 440-ish dotted-version sort. + 4. Generates a markdown table per known region type and replaces the + content inside the matching sentinel block. + +Sections covered today: + + - top-level README.md + ... + - likelihood/README.md (section overview) + - likelihood/imaging/README.md | likelihood-imaging + - likelihood/interferometer/README.md | likelihood-interferometer + - likelihood/point_source/README.md | likelihood-point_source + - likelihood/datacube/README.md | likelihood-datacube + - simulators/README.md | simulators + - searches/nautilus/README.md | searches-nautilus + +Hardware-tier columns (CPU / laptop GPU / HPC GPU) are deferred — every +artifact today is implicitly CPU and the table shows a single "Latest" +column. Once future artifacts encode hardware in the filename or JSON +(`*_summary_v_.json` or `{"hardware": "a100"}`), the +column logic in `_render_*_table` will be extended without touching the +sentinel layout. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable, Optional + +REPO_ROOT = Path(__file__).resolve().parent.parent +RESULTS_ROOT = REPO_ROOT / "results" + +# Sentinel block: keeps surrounding hand-written prose intact, only the +# content between BEGIN and END is rewritten. +SENTINEL_RE = re.compile( + r"()" + r".*?" + r"()", + re.DOTALL, +) + +# Artifact filename: