Rerun the launch scatter benchmark and refresh every published number - #332
Merged
Conversation
The README, docs landing page, and public benchmarks page all quoted the 0.1.0 launch baseline recorded on 2026-07-13. This reruns that benchmark on current main and moves every published figure onto the new measurement. The rerun is a controlled A/B rather than a fresh unrelated run: it uses the 0.1.0 baseline's own uv project with --frozen, whose editable xy source resolves to the worktree it runs from. Plotly 6.9.0, Kaleido 1.3.0, Matplotlib 3.11.0, NumPy 2.5.1 and Python 3.14.5 are therefore identical across both runs on the same M5 Pro, leaving the xy revision and a Chrome patch bump as the only intended variables. Across the 24 successful competitor timing cells the median absolute drift against 0.1.0 is 3.2%, 16 of 24 land within 5% and all 24 within 10%, with the largest deviations on the rows dominated by cold browser startup. Static render time improved 1.26-2.37x, most at 10k-1M where fixed per-call overhead dominates rather than kernel work; at 10M the static CPU PNG is 0.0184 s against 0.0232 s. Interactive TTFR is unchanged within noise because it is bounded by cold browser startup, not by xy. At 1B only xy completes, in 1.129 s static and 1.242 s interactive; Plotly's static row now fails as a Kaleido timeout at 122 s rather than at the memory ceiling. The 0.1.0 baseline stays committed and unchanged as the runbook requires; the refresh lands in a sibling directory and its report carries an explicit 0.1.0-versus-now comparison. Both suites, all sizes, three cold runs per cell, 180 s and 36 GiB guardrails, hardware-WebGL and SwiftShader rows kept separate. The environment was warmed first: a first invocation in a fresh worktree reads 13x the warm value for the xy 10k static row, and the runbook now says so. Every published figure was re-derived from the committed raw JSON and checked against the docs tables, the live docs charts, and the comparison SVG before commit.
|
Warning Review limit reached
Next review available in: 31 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThe PR adds a 2026-07-26 macOS ARM64 launch benchmark baseline, including raw results, environment metadata, configuration, and report data. It updates benchmark documentation, README claims, methodology references, and embedded chart values to reflect the refreshed measurements. ChangesLaunch benchmark refresh
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Merging this PR will not alter performance
Comparing 
Footnotes
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/report.md (1)
7-9: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRecord the RNG seed in the machine-readable baseline too.
The report states seed
20260713(Line 253) but neithermanifest.jsonnorenvironment.jsoncarries it, so a re-run reproducing this baseline has to parse prose to get the one input that determines the data. Consider adding"seed": 20260713tomanifest.json.As per coding guidelines, "Record every decimation and tier decision in the specification; decisions must never be silent."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/report.md` around lines 7 - 9, Update the machine-readable benchmark metadata, preferably manifest.json, to include the RNG seed as a numeric "seed" field with value 20260713. Keep the existing report and benchmark configuration unchanged while ensuring reruns can obtain the seed without parsing report prose.Source: Coding guidelines
docs/overview/benchmarks.md (1)
148-148: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuemarkdownlint MD048: fence style.
This file's configured code-fence style is tilde; the reproduction block opens with backticks. Switch to
~~~bash/~~~to keepdocslint green.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/overview/benchmarks.md` at line 148, Update the reproduction code block in the benchmarks documentation from backtick fences to tilde fences, using the configured ~~~bash opener and matching ~~~ closer so markdownlint MD048 passes.Source: Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/default-results.json`:
- Around line 840-848: Redact absolute home-directory prefixes in the plotly
entry’s stderr_tail before publishing the benchmark artifact, replacing
`/Users/alek` and equivalent local home paths with a non-sensitive placeholder
while preserving the traceback and error details.
In `@benchmarks/README.md`:
- Around line 75-78: Update the benchmark warm-up guidance in
benchmarks/README.md to remove the unsupported “13× the warm value” claim and
any derived cold-run figure, unless a corresponding cold measurement is added to
the committed baseline artifact. Preserve the qualitative instruction to discard
the first invocation because it includes cold-cache, font-cache, and
browser-download costs.
In `@docs/overview/benchmarks.md`:
- Around line 139-143: Add a link to cpu-fallback-results.json in the benchmark
evidence list near the existing Raw default-path results link, using the same
baseline directory and a clear label indicating CPU-fallback raw results.
---
Nitpick comments:
In `@benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/report.md`:
- Around line 7-9: Update the machine-readable benchmark metadata, preferably
manifest.json, to include the RNG seed as a numeric "seed" field with value
20260713. Keep the existing report and benchmark configuration unchanged while
ensuring reruns can obtain the seed without parsing report prose.
In `@docs/overview/benchmarks.md`:
- Line 148: Update the reproduction code block in the benchmarks documentation
from backtick fences to tilde fences, using the configured ~~~bash opener and
matching ~~~ closer so markdownlint MD048 passes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: a2292136-cbc4-4cef-bbf1-4e375e68db22
⛔ Files ignored due to path filters (3)
benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/uv.lockis excluded by!**/*.lockdocs/app/assets/launch-benchmark-comparison.svgis excluded by!**/*.svgspec/assets/launch-benchmark-comparison.svgis excluded by!**/*.svg
📒 Files selected for processing (12)
README.mdbenchmarks/README.mdbenchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/cpu-fallback-results.jsonbenchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/default-results.jsonbenchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/environment.jsonbenchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/manifest.jsonbenchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/pyproject.tomlbenchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/report.mddocs/app/xy_docs/demos/benchmark_charts.pydocs/index.mddocs/overview/benchmarks.mdspec/benchmarks/methodology.md
Comment on lines
+840
to
+848
| { | ||
| "status": "failed(exit=1)", | ||
| "library": "plotly", | ||
| "n": 1000000000, | ||
| "peak_rss_bytes": 28016345088, | ||
| "wall_ms": 122100.63737502787, | ||
| "stdout_tail": "", | ||
| "stderr_tail": "lib/python3.14/asyncio/runners.py\", line 127, in run\n return self._loop.run_until_complete(task)\n ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^\n File \"/Users/alek/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/base_events.py\", line 719, in run_until_complete\n return future.result()\n ~~~~~~~~~~~~~^^\n File \"/Users/alek/Desktop/xy-bench/benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/.venv/lib/python3.14/site-packages/kaleido/__init__.py\", line 104, in calc_fig\n return await k.calc_fig(\n ^^^^^^^^^^^^^^^^^\n ...<3 lines>...\n )\n ^\n File \"/Users/alek/Desktop/xy-bench/benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/.venv/lib/python3.14/site-packages/kaleido/kaleido.py\", line 618, in calc_fig\n return await self.write_fig_from_object( # type: ignore[reportCallIssue]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n ...<4 lines>...\n )\n ^\n File \"/Users/alek/Desktop/xy-bench/benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/.venv/lib/python3.14/site-packages/kaleido/kaleido.py\", line 493, in write_fig_from_object\n res = await asyncio.gather(*tasks, return_exceptions=not cancel_on_error)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/alek/Desktop/xy-bench/benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/.venv/lib/python3.14/site-packages/kaleido/kaleido.py\", line 358, in _render_task\n img_bytes = await asyncio.wait_for(\n ^^^^^^^^^^^^^^^^^^^^^^^\n ...<7 lines>...\n )\n ^\n File \"/Users/alek/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/tasks.py\", line 487, in wait_for\n async with timeouts.timeout(timeout):\n ~~~~~~~~~~~~~~~~^^^^^^^^^\n File \"/Users/alek/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/timeouts.py\", line 114, in __aexit__\n raise TimeoutError from exc_val\nTimeoutError\n" | ||
| } |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== relevant files ==\n'
git ls-files 'benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/default-results.json' \
'benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/manifest.json' \
'benchmarks/launch_baselines/xy-main-2026-07-26/report.md' \
'benchmarks/README.md' \
'benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/default-results.json' \
'benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/manifest.json' \
'benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/*' \
'benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/*'
printf '\n== manifest/report references ==\n'
rg -n --no-heading 'reproduction\.uv_project|xy-main-2026-07-26|xy-0\.1\.0|launch_baselines|venv|Desktop/xy-bench' \
benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/manifest.json \
benchmarks/launch_baselines/xy-main-2026-07-26/report.md \
benchmarks/README.md \
benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/manifest.json \
benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/default-results.json || true
printf '\n== surrounding excerpt from the reviewed file ==\n'
python3 - <<'PY'
from pathlib import Path
p = Path('benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/default-results.json')
text = p.read_text()
needle = '"library": "plotly"'
idx = text.find(needle)
start = max(0, idx - 600)
end = min(len(text), idx + 1800)
print(text[start:end])
PY
printf '\n== compare with 0.1.0 result path patterns ==\n'
python3 - <<'PY'
from pathlib import Path
for rel in [
'benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/default-results.json',
'benchmarks/launch_baselines/xy-0.1.0/macos-arm64-m5-pro/default-results.json',
]:
p = Path(rel)
if p.exists():
s = p.read_text()
import re
paths = sorted(set(re.findall(r'/Users/[^"]+', s)))
print(rel)
for path in paths[:8]:
print(' ', path)
if len(paths) > 8:
print(' ...', len(paths), 'total')
PYRepository: reflex-dev/xy
Length of output: 10141
Redact home-directory paths from stderr_tail
The published JSON leaks /Users/alek/... paths and the local username in stderr_tail; scrub absolute home-directory prefixes before committing this artifact.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/default-results.json`
around lines 840 - 848, Redact absolute home-directory prefixes in the plotly
entry’s stderr_tail before publishing the benchmark artifact, replacing
`/Users/alek` and equivalent local home paths with a non-sensitive placeholder
while preserving the traceback and error details.
…lback artifact Two review fixes. The runbook quoted a "13x the warm value" cold-start multiplier that no committed artifact backs; discarded warm-up runs are deliberately not part of the baseline, and spec/benchmarks/methodology.md bans numbers in prose that don't exist in the artifact. The instruction to run once and discard stands on its own without the figure. The public benchmarks page publishes an interactive CPU-fallback row but only linked the default-path results. cpu-fallback-results.json is the only artifact backing that row and is committed in the same directory, so link it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The README, docs landing page, and public benchmarks page all quoted the 0.1.0 launch baseline recorded on 2026-07-13. This reruns that benchmark on current main (
7604775) and moves every published figure onto the new measurement.Why this is a controlled A/B, not just a fresh run
The rerun uses the 0.1.0 baseline's own uv project with
--frozen, whose editablexysource resolves to the worktree it runs from. So Plotly 6.9.0, Kaleido 1.3.0, Matplotlib 3.11.0, NumPy 2.5.1 and Python 3.14.5 are identical across both runs on the same M5 Pro — the only intended variables are the xy revision and a Chrome patch bump (150.0.7871.115→.186).Across the 24 successful competitor timing cells, median absolute drift against 0.1.0 is 3.2%; 16 of 24 within 5%, all 24 within 10%. The largest deviations sit on the rows dominated by cold browser startup. That's the evidence the machine didn't move, which is what makes the xy-side delta below meaningful.
Results at 10M
Static render time improved 1.26–2.37×, most at 10k–1M where fixed per-call overhead dominates rather than kernel work. Interactive TTFR is unchanged within noise because it is bounded by cold browser startup, not by xy. At 1B only xy completes — 1.129 s static, 1.242 s interactive — and Plotly's static row now fails as a Kaleido timeout at 122 s / 26.09 GiB rather than at the 36 GiB ceiling.
Baseline handling
The 0.1.0 baseline stays committed and unchanged, as the runbook requires. The refresh lands in a sibling directory
benchmarks/launch_baselines/xy-main-2026-07-26/macos-arm64-m5-pro/with raw JSON for both suites, environment, manifest, the copied lock, and a report that carries an explicit 0.1.0-versus-now comparison table.Both suites, all sizes, three cold runs per successful cell, 180 s and 36 GiB guardrails, hardware-WebGL and SwiftShader rows kept separate. Renderer strings are recorded per sample, so the GPU and SwiftShader tables are verified rather than assumed from flags.
The environment was warmed before measuring: a first invocation in a fresh worktree reads 13× the warm value for the xy 10k static row (cold bytecode, Matplotlib's font cache, Kaleido's browser fetch). The runbook now says so.
Docs touched
README.md,docs/index.md,docs/overview/benchmarks.md,docs/app/xy_docs/demos/benchmark_charts.py(the three live charts on the benchmarks page),benchmarks/README.md,spec/benchmarks/methodology.md, and both byte-identical copies oflaunch-benchmark-comparison.svg.Two caveats surfaced by the new data: the CPU-fallback 10M mean now sits slightly below the 1M mean, and static 1M sits below 100k — both flat within noise, so the docs' existing "small reversals" caveat was moved to point at the static path where it now applies.
Verification
ruff check/ruff format --check/codespellclean at repo root and indocs/app;pre-commit run --all-filespassedto_png()and visually checked; the regenerated SVG was screenshotted and checked — its diff is only bar widths and value labels, layout untouchedOnly the launch-scatter suite was rerun, since that is what the README and benchmark docs quote. The other harnesses (
bench_vs,bench_interaction,bench_dashboard,bench_transport, CodSpeed) feedspec/benchmarks/results.mdand CI and are untouched.🤖 Generated with Claude Code
Summary by CodeRabbit