Characterise scale on all four engines - #18
Merged
Conversation
SqliteVolumeTests measured the bundled default at scale; the bring-your-own-server
engines were assumed to behave the same. CrossEngineVolumeTests measures them - it is
fixture-driven, so one test profiles whichever engine DISPATCH_TEST_ENGINE selects,
and it exercises BOTH read paths: the keyset cursor (QueryAsync) and the dashboard
dedup-plus-total list (PageAsync, the path whose grouping was just made sargable).
Measured at 100k rows (docs/database.md has the table). The findings that matter:
* Keyset pagination is flat on every engine - page 20 costs the same as page 1
(~0-1 ms), an index seek not a scan. This is the property the Message Log relies
on and it holds on SQLite, PostgreSQL, MariaDB and SQL Server alike.
* The dashboard list (PageAsync) is the heaviest read on EVERY engine, 436 ms to
1.7 s at 100k and growing with total rows, because it deduplicates and returns an
exact total over the whole filtered set - running its grouping twice per page. So
the deferred double-execution optimisation is a real, measured, cross-engine cost,
not a SQLite quirk. This is the number that says when it starts to matter.
* Purge time is dominated by its deliberate 100 ms inter-batch pause on every
engine, not the delete - by design, so ingest can take the write lock between
batches.
Opt-in (DISPATCH_VOLUME_TEST), so normal CI skips it. Assertions catch a collapse (a
scan where a seek was intended), never a slow agent. The printed numbers are the
deliverable; the docs table records a baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested follow-up: measure the bring-your-own-server engines at volume rather than assume they scale like the bundled SQLite default.
CrossEngineVolumeTestsis fixture-driven — one test profiles whichever engineDISPATCH_TEST_ENGINEselects — and exercises both the keyset cursor and the dashboard dedup-plus-total list (PageAsync, whose grouping was just made sargable in #15).Measured at 100k rows (full table in
docs/database.md):¹ under x86 emulation on arm64 — its write/purge times are inflated by that, not the engine.
Findings:
Opt-in (
DISPATCH_VOLUME_TEST); normal CI skips it.🤖 Generated with Claude Code