Skip to content

Keynote 2 benchmark updates & refinements#4997

Open
bradleyshep wants to merge 46 commits into
masterfrom
bradley/keynote-benchmark-updates
Open

Keynote 2 benchmark updates & refinements#4997
bradleyshep wants to merge 46 commits into
masterfrom
bradley/keynote-benchmark-updates

Conversation

@bradleyshep
Copy link
Copy Markdown
Contributor

This PR builds on #4978

Description of Changes

Methodology refresh of templates/keynote-2/.

  • Pool size fix. Removed the poolMax = 1000 override on the pg-flavored RPC servers (postgres, cockroach, supabase). All now use node-pg's default of 10, matching bun-server.ts.
  • Convex retry loop removed. The 32-retry client-side loop in connectors/convex.ts was masking conflict-storms with a death spiral.
  • Convex package bumped from ^1.29 to ^1.37.
  • Hidden Postgres tuning removed from docker-compose.yml. Local pg keeps only the explicit default_transaction_isolation = 'serializable' override.
  • Per-second time-series sampling added to core/runner.ts. Each run emits a timeSeries array for warmup/decay/collapse charts.
  • Bench CLI accepts --alpha 0,1.5 (CSV), --runs N, and --prep-between-alphas, writing one JSON per (connector, alpha, run) tuple.
  • Helper scripts added: start-bench.sh, stop-bench.sh, check-bench.sh, bench-stats.py, plot-bench.py. Paths resolve relative to the script file.
  • README updated with current measurements, two PlanetScale rows (PS-2560 single-node EBS and M-15360 Metal HA), and isolation-level methodology notes.
  • Removed unused Docker files

API and ABI breaking changes

No engine API/ABI changes. Internal shape changes for the keynote template:

  • BenchOptions.alpha: numberalphas: number[]. New fields: runs, prepBetweenAlphas.
  • RunResult gained a required timeSeries field.
  • Output JSON filename is now test-1-<connector>-a<alpha>-<timestamp>.json (glob test-1-*.json still matches).

Expected complexity level and risk

3. Contained to templates/keynote-2/ but cross-cutting within it (runner, CLI, RPC servers, docker-compose, scripts, README).

Testing

Testing

Reviewer smoke test:

pnpm run prep
pnpm run bench --alpha 0,1.5 --connectors postgres_rpc --seconds 30 --runs 1
ls runs/test-1-postgres_rpc-a*.json   # expect 2 files
  • pnpm build passes
  • pnpm run bench --help lists the new flags
  • Reviewer: smoke sweep above produces 2 JSONs (one per alpha)
  • Reviewer: confirm --prep-between-alphas actually re-preps between alphas
  • Reviewer: confirm pnpm run demo still works

@clockwork-labs-bot clockwork-labs-bot added this pull request to the merge queue May 12, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 12, 2026
@bfops bfops added this pull request to the merge queue May 12, 2026
@joshua-spacetime joshua-spacetime removed this pull request from the merge queue due to a manual request May 12, 2026
@bfops bfops added this pull request to the merge queue May 12, 2026
Copy link
Copy Markdown
Contributor

@joshua-spacetime joshua-spacetime left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to update the numbers

@joshua-spacetime joshua-spacetime removed this pull request from the merge queue due to a manual request May 12, 2026
@bfops bfops enabled auto-merge May 12, 2026 16:21
@bfops bfops disabled auto-merge May 12, 2026 16:54
@bradleyshep
Copy link
Copy Markdown
Contributor Author

bradleyshep commented May 13, 2026

Need to update the numbers

@joshua-spacetime do you know what configuration we're using for these README #'s? CC @cloutiertyler

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants