Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 28 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ This repository is the single home for PyAutoLens performance measurement. It ex

Results are framed by **astronomy instrument** (HST, Euclid, JWST, …) rather than by raw pixel counts. Pixel counts are recorded too, but the headline numbers a reader sees first are the ones that map onto a real observing programme.

## Latest run-times

The table below is auto-generated from the latest versioned artifacts under `results/`. Each row is the latest steady-state per-call cost for a likelihood path at a given instrument; numbers refresh whenever the producing scripts are rerun and committed. Hardware tier is **CPU only** today — laptop GPU and HPC GPU columns will land once `results/**` artifacts are tagged with a hardware label.

<!-- BEGIN auto-table:headline -->
_No data yet — run likelihood scripts to populate. See `likelihood/README.md`._
<!-- END auto-table:headline -->

(Generator: `scripts/build_readme.py`. Run `python scripts/build_readme.py` after producing new artifacts to refresh; `--check` exits non-zero in CI if it would change anything.)

## JAX gradients — currently out of scope

Gradient profiling (`jax.grad` of the likelihood, autodiff-based optimisers) is **not yet** part of this repo. It is tracked in [`PyAutoLabs/autolens_workspace_developer/jax_profiling/gradient/`](https://github.com/PyAutoLabs/autolens_workspace_developer/tree/main/jax_profiling/gradient) and will fold into this repo in a future phase once the gradient story stabilises.
Expand Down Expand Up @@ -54,18 +64,27 @@ Examples that already exist in the source-of-truth repo:

## Roadmap

This repo is being built in phases:
This repo is being built in phases. Phase numbers correspond to internal sub-prompts under `PyAutoLabs/PyAutoPrompt/z_features/autolens_profiling.md`.

| Phase | Title | Status |
|-------|-------|--------|
| 0 | Repo bootstrap (this commit) | ✓ shipped |
| 1 | Mirror JIT likelihood profiling scripts + per-section READMEs | not yet started |
| 2 | Mirror simulator profiling scripts + run-time tracking | not yet started |
| 3 | Nautilus profiling, design for sampler expansion | not yet started |
| 4 | Top-level + per-section README dashboard with instrument framing | not yet started |
| 5 | GitHub Actions for lint + profile re-runs + README refresh | not yet started |

The full multi-phase plan lives in the internal `PyAutoLabs/PyAutoPrompt/z_features/autolens_profiling.md` tracker (not publicly readable). The high-level shape is captured above.
| 0 | Repo bootstrap | ✓ shipped |
| 1 | Mirror JIT likelihood profiling scripts + per-section READMEs | ✓ shipped |
| 2 | Mirror simulator profiling scripts + run-time tracking | ✓ shipped |
| 3 | Nautilus profiling, design for sampler expansion | ✓ shipped |
| 4 | Top-level + per-section README dashboard with instrument framing | ✓ shipped |
| 5 | GitHub Actions for lint + profile re-runs + README refresh | queued |

### Future enhancements (Phase 4 follow-ups)

Dashboards can grow in many directions. The list below captures candidate improvements that fit the "profiling and run-times" theme; none of them block the current dashboard from being useful.

- **Regression-watch indicator** — colour or arrow per cell showing whether the latest cost regressed (>5%) or improved versus the previous PyAutoLens release. Needs the second-latest version per axis kept alongside the latest. Trivial to add to `scripts/build_readme.py`.
- **Per-axis version-history PNGs** — small inline plot of run-time vs PyAutoLens release version, generated from the JSON artifacts (reusing the `_developer/jax_profiling/results/jit/.../*_v<version>.png` generator). Embeds nicely above each section table.
- **Plotly-rendered interactive timeline** — hostable on GitHub Pages once the static dashboard stabilises; lets readers hover/filter across instrument × model × release.
- **Flamegraph captures** — alongside the headline timing numbers, store a flamegraph per instrument × model for the most recent release.
- **Hardware-tier columns** — extend `scripts/build_readme.py` table renderers to show CPU / laptop GPU / HPC GPU as separate columns once result artifacts encode the hardware label (filename suffix or JSON `"hardware"` field).
- **Archive old versions** — once a script has >6 minor releases of artifacts, move the older ones to `results/archive/` so the latest views stay uncluttered.

## Related repos

Expand Down
10 changes: 5 additions & 5 deletions likelihood/datacube/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,13 +37,13 @@ That number quantifies how much a future "shared `Lᵀ W̃ L`" optimisation woul

For a realistic per-channel-distinct cube, point the loader at the workspace simulator output at `autolens_workspace/dataset/interferometer/datacube/sim_simple/`. The JIT-cost taxonomy doesn't change — it's a function of which arrays are loop-variables in `FitInterferometer`, not the data values themselves.

## Headline run-times (populated by Phase 4)
## Headline run-times (latest per dataset)

| Script | Dataset | N channels | CPU | Laptop GPU | A100 |
|--------|---------|------------|-----|------------|------|
| `delaunay.py` | SMA × 4 | 4 | _populated_ | _populated_ | _populated_ |
Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/datacube/`. Hardware tier is CPU only today.

Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/datacube/`.
<!-- BEGIN auto-table:likelihood-datacube -->
_No data yet — run a script under this folder to populate. See section README._
<!-- END auto-table:likelihood-datacube -->

## Output

Expand Down
12 changes: 5 additions & 7 deletions likelihood/imaging/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,15 +37,13 @@ XLA may fuse these differently when compiled as one program vs separate pieces,

`dataset/imaging/hst/` — an HST-resolution mock (pixel scale 0.05″, 21×21 PSF) committed to this repo. Other instruments (`euclid`, `jwst`, `ao`) can be regenerated via the source-of-truth scripts at `autolens_workspace_developer/jax_profiling/dataset_setup/imaging.py` and copied into `dataset/imaging/<instrument>/`.

## Headline run-times (populated by Phase 4)
## Headline run-times (latest per script × instrument)

| Script | Instrument | CPU | Laptop GPU | A100 |
|--------|------------|-----|------------|------|
| `mge.py` | HST | _populated_ | _populated_ | _populated_ |
| `pixelization.py` | HST | _populated_ | _populated_ | _populated_ |
| `delaunay.py` | HST | _populated_ | _populated_ | _populated_ |
Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/imaging/`. Hardware tier is CPU only today.

Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/imaging/`.
<!-- BEGIN auto-table:likelihood-imaging -->
_No data yet — run a script under this folder to populate. See section README._
<!-- END auto-table:likelihood-imaging -->

## Output

Expand Down
10 changes: 4 additions & 6 deletions likelihood/interferometer/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,11 @@ The interferometer likelihood path is profiled at **full-pipeline JIT** only, no

## Headline run-times (populated by Phase 4)

| Script | Instrument | CPU | Laptop GPU | A100 |
|--------|------------|-----|------------|------|
| `mge.py` | SMA | _populated_ | _populated_ | _populated_ |
| `pixelization.py` | SMA | _populated_ | _populated_ | _populated_ |
| `delaunay.py` | SMA | _populated_ | _populated_ | _populated_ |
Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/interferometer/`. Hardware tier is CPU only today.

Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/interferometer/`.
<!-- BEGIN auto-table:likelihood-interferometer -->
_No data yet — run a script under this folder to populate. See section README._
<!-- END auto-table:likelihood-interferometer -->

## Output

Expand Down
11 changes: 5 additions & 6 deletions likelihood/point_source/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,14 +36,13 @@ For both variants:

`dataset/point_source/simple/` — a minimal seeded dataset with `point_dataset_positions_only.json` (4 observed image positions) and the truth `tracer.json`. Both files are committed to this repo.

## Headline run-times (populated by Phase 4)
## Headline run-times (latest per script × dataset)

| Script | Dataset | CPU | Laptop GPU | A100 |
|--------|---------|-----|------------|------|
| `image_plane.py` | simple | _populated_ | _populated_ | _populated_ |
| `source_plane.py` | simple | _populated_ | _populated_ | _populated_ |
Auto-generated by `scripts/build_readme.py` from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/point_source/`. Hardware tier is CPU only today. **Cells may show `—` while [PyAutoLens#514](https://github.com/PyAutoLabs/PyAutoLens/issues/514) is open** — the regression assertion in both scripts is intentionally load-bearing while the upstream drift is triaged, so neither script reaches the JSON-write step in the current PyAutoLens release.

Numbers are the **steady-state per-call cost** (single-JIT, post-warmup), in milliseconds. Phase 4's dashboard auto-fills this from the latest `*_summary_v<version>.json` artifacts under `results/likelihood/point_source/`.
<!-- BEGIN auto-table:likelihood-point_source -->
_No data yet — run a script under this folder to populate. See section README._
<!-- END auto-table:likelihood-point_source -->

## Output

Expand Down
Loading