chore(omnigraph): run the 0.10.0 edge build to measure its effect on the write ceiling - #231
Merged
Merged
Conversation
…the write ceiling
An experiment, deliberately scoped as one: the write path is the thing standing
between this deployment and a team, and upstream main has landed insert-path
work since v0.9.0 that nobody has measured against our load.
WHY THIS IS CHEAP, WHICH IS THE WHOLE REASON TO TRY IT. The v0.8.1 → v0.9.0
upgrade was expensive because the on-disk format broke: ~16 graphs per
environment rebuilt via export → init → load, no downgrade, no canary.
None of that applies here. Verified against the edge binary before touching
anything:
omnigraph 0.10.0 reads storage format 6, as declared.
Same format we already migrated CI to, so this is a binary swap against the
same store, and rollback is repointing the image. The binary-contract suite —
the one that caught 0.9.0's unannounced DateTime export change — passes 20/20,
and the full suites pass against the edge binary: witan-core 466,
witan-council 770, witan-code 516.
★ TWO PINS NOW, BECAUSE THE TAG AND THE VERSION GENUINELY DIFFER. `edge` is a
moving tag that upstream force-updates on every push to main, and the binary
behind it reports 0.10.0 — a version that has no release to download. Folding
both into one string breaks either the URL or the "already installed, skipping"
check, which compares against what `omnigraph --version` actually prints. So
_OMNIGRAPH_RELEASE_TAG says where to fetch from and _OMNIGRAPH_VERSION says what
should come back.
`just check-omnigraph-pins` now enforces agreement on BOTH across all three
files. Adding a second pin without extending that check would have recreated
precisely the gap its own comment describes — the one that "went a full release
cycle claiming the manager covered it when it did not".
── KNOWN LIMITATIONS OF USING A MOVING TAG ──
* The skip check weakens: two different `edge` builds both report 0.10.0, so a
machine that installed yesterday's will not fetch today's. Delete the binary
when you need certainty. Documented at the constant.
* Renovate manages the VERSION line only, and a bump is not meaningful while
the tag is `edge`. Pin a real v<version> before treating updates here as
authoritative.
* `edge` is unreleased. Fine for CI and for measurement; not for Production.
── WHAT THIS IS AND IS NOT EXPECTED TO FIX ──
IN 0.10.0: graph-commit conditional writes (CAS, upstream #470) and exact write
receipts (#479), plus real insert-path work — strict inserts staged without
merge joins, the Lance 10 migration, branch-merge join/index-build cliffs
removed, filter pushdown.
NOT IN IT: cancellation shielding (#490) and the async job API with request-key
idempotency (#415) are still open PRs upstream. Those two are what actually make
a write's outcome knowable to the client, so this build is not expected to fix
the indeterminacy — only, possibly, to move the knee.
The baseline to compare against, measured today at 3e8e3ff: 8 concurrent
writers PASS clean; 24 writers FAIL with 23 of 24 committed-but-reported-failed;
server-side handler durations 3s → 73s on one graph.
Refs tk-upstream-omnigraph-a-single-row-insert-costs-a-f-eeeae3,
tk-concurrent-writes-to-the-deployed-witan-are-canc-02abbd,
tk-omnigraph-conditional-write-cas-precondition-on--94155f
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYb9sMjetD9Nxjf24Aw3m5
Contributor
There was a problem hiding this comment.
Pull request overview
Pins Omnigraph to the rolling edge build for write-performance testing.
Changes:
- Separates release tag and reported version pins.
- Updates both container images to download from
edge. - Extends pin-consistency validation to include release tags.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
packages/witan-core/witan_core/omnigraph_install.py |
Adds separate version and release-tag handling. |
justfile |
Validates tag and version agreement. |
docker/witan.Dockerfile |
Fetches the edge CLI binary. |
docker/omnigraph-server.Dockerfile |
Fetches edge server and CLI binaries. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…erge Review feedback on #231, and the reviewer is right that the previous commit did not deliver what the PR claims. `edge` is force-updated on every push to upstream main, so equal version and tag strings prove nothing: the installer and the two image builds can each resolve the same tag to a DIFFERENT commit while every check passes. For a change whose entire output is a load-test measurement, that makes the result unattributable — which defeats the point of running it. Downloading the `.sha256` published beside the tarball does not fix it either. That file moves with the release, so it only ever attests to whichever build was current at download time. So the digest is now pinned IN THIS REPO, for all three assets, and checked before anything is extracted. `just check-omnigraph-pins` compares the linux/x86_64 digest across all three tiers alongside the version and the tag — verified to fail on a deliberately drifted value, not just asserted to. ★ AND THE INSTALLER VERIFIED NOTHING AT ALL BEFORE THIS. It put whatever the URL returned onto a developer's PATH. That is now a hard refusal on mismatch, and a hard refusal when an asset has no pinned digest — a platform added to _OMNIGRAPH_ASSETS and forgotten in the digest map must fail closed, since the inverse is how verification quietly stops applying. The existing install tests built fake tarballs and were correctly refused by the new check. They now register the fake's own digest through a `_serve` helper, so they go THROUGH verification rather than around it, and two new tests cover the refusal paths. Verified by building the image, both ways: the pinned digest builds and reports `omnigraph 0.10.0`; a deliberately wrong `--build-arg` fails the build at `sha256sum -c`. Also corrected the installer docstring, which told the reader to "pass a real release tag" — there is no such argument, and the tag is deliberately a property of the repo rather than of a run precisely because all three tiers must agree. It now says what to actually do: delete the binary and re-run `witan setup`, or edit the constant and refresh the digests in the same commit. ★ ONE CONSEQUENCE WORTH KNOWING: when upstream pushes to main, the next fetch FAILS LOUDLY on the digest instead of silently installing a different binary. Refreshing the pins is then a deliberate act — which is the property that makes a measurement taken against this build mean something. Refs tk-upstream-omnigraph-a-single-row-insert-costs-a-f-eeeae3 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JYb9sMjetD9Nxjf24Aw3m5
blarghmatey
added a commit
that referenced
this pull request
Aug 14, 2026
Two things that belong in one commit because the revert has to be in the released version: the predictive write gate ships, and the 0.10.0 edge experiment ends. ── THE REVERT: 0.10.0 HALVED THE WRITE CEILING ── #231 pinned the upstream `edge` build to measure its effect on the write ceiling. That is the measurement, and the sign is wrong. Same probe, same cluster, same actor, same evening: 0.9.0 8 writers PASS — 8 acked, 0 indeterminate, readers clean 0.10.0 8 writers FAIL x6 consecutive runs, 2 to 8 of 8 indeterminate, every reader degraded in every run 0.10.0 4 writers PASS x2 — clean ★ THE CLEAN POINT MOVED FROM 8 TO 4. Higher loads are the same story, worse: 12w → 10 of 12 indeterminate, 16w → 10 of 16, 24w → 19 of 24 with the pod log showing all 24 handlers `"outcome": "ok"` — every one of those writes committed while its caller was told it failed. 0.10.0 also produces a shape 0.9.0 never did: a few genuine server-side errors, ~7ms, on `task_get` and `memory_store` at the same instant. Small in number and not the main effect, but new. Mutual exclusion is unaffected — probe A passed at every load on both builds, exactly one `claimed: true` and every loser a structured refusal. What changed is the write path's capacity, not its correctness. ★ NOTHING IN 0.10.0 WAS AIMED AT OUR PROBLEM ANYWAY. The two upstream changes that address the indeterminate write — cancellation shielding and the async job API — are not in it; the first was closed 2026-08-13 as superseded by RFC-035 (#498), which with RFC-034 (#496) and RFC-036 (#499) is an open design document. So this build was carrying a measured regression in exchange for nothing, on an unreleased binary off upstream main. Reverting is as cheap as the upgrade was, for the same reason: storage format 6 is unchanged in both directions, so this is a binary swap against the same store with no migration. Verified by building the fetch stage — the pinned v0.9.0 digests produce `omnigraph 0.9.0`. WEAKNESS TO CARRY: the 0.9.0 side is ONE run at 8 writers against six on 0.10.0. Six consecutive failures where 0.9.0 was clean is strong, but the clean baseline is not itself replicated. Re-run 8w a few times on 0.9.0 before concluding anything upstream. ── THE RELEASE: witan-core 0.18.0 ── The predictive write gate (#232). The gate previously bounded concurrency and the wait for a slot, and neither predicts how long the work takes once admitted — so it let through exactly the writes that strand. It now measures what an admitted write costs per graph and refuses when that no longer fits the call's remaining budget. This does not raise the ceiling; it converts the writes that cannot succeed into a clean pre-send refusal instead of an indeterminate outcome. Refs tk-the-write-gate-is-sized-against-a-3-45s-solo-wri-73fc2b Memory: les-omnigraph-0-10-0-edge-looks-worse-than-0-9-0-on--ef8f5d Claude-Session: https://claude.ai/code/session_01JYb9sMjetD9Nxjf24Aw3m5 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What are the relevant tickets?
tk-upstream-omnigraph-a-single-row-insert-costs-a-f-eeeae3,tk-concurrent-writes-to-the-deployed-witan-are-canc-02abbd,tk-omnigraph-conditional-write-cas-precondition-on--94155f, underwp-witan-multi-user-service-deployment-dcf6ee.Description (What does it do?)
Pins all three omnigraph declarations to the upstream
edgebuild (reports 0.10.0, 73 commits past v0.9.0), as a scoped experiment: the write path is what stands between this deployment and a team, and upstream main has landed insert-path work nobody has measured against our load.Why this is cheap — which is the entire reason to try it
The v0.8.1 → v0.9.0 upgrade was expensive because the on-disk format broke: ~16 graphs per environment rebuilt via export → init → load, no downgrade, no canary. None of that applies. Verified before touching anything:
Same format CI already runs. So this is a binary swap against the same store, and rollback is repointing the image.
Upstream publishes prebuilt binaries for every push to main (
release-edge.ymlforce-updates theedgetag), so there is no source build involved.Two pins, because the tag and the version genuinely differ
edgeis a moving tag; the binary behind it reports0.10.0, a version with no release to download. Folding both into one string breaks either the URL or the "already installed, skipping" check, which compares against whatomnigraph --versionprints. So:_OMNIGRAPH_RELEASE_TAG— where to fetch from (edge, orv<version>for a real release)_OMNIGRAPH_VERSION— what should come back (0.10.0)just check-omnigraph-pinsnow enforces agreement on both, across all three files. Adding a second pin without extending that check would have recreated exactly the gap its own comment describes — the one that "went a full release cycle claiming the manager covered it when it did not".edgebuilds both report0.10.0, so a machine that installed yesterday's will not fetch today's. Delete the binary when you need certainty. Documented at the constant.edge. Pin a realv<version>before treating updates here as authoritative.edgeis unreleased. Fine for CI and for measurement; not for Production.What this is, and is not, expected to fix
In 0.10.0: graph-commit conditional writes (CAS, upstream #470), exact write receipts (#479), and real insert-path work — strict inserts staged without merge joins, the Lance 10 migration, branch-merge join/index-build cliffs removed, filter pushdown to the Lance scanner.
Not in it: cancellation shielding (upstream #490) and the async job API with request-key idempotency (#415) are still open PRs. Those two are what actually make a write's outcome knowable to the client — so this build is not expected to fix the indeterminacy, only possibly to move the knee.
If the answer turns out to be "no change", that is still worth having: it isolates the ceiling to the per-write cost and points squarely at #490/#415 landing upstream.
How has this been tested?
Everything below ran with the edge binary on
PATH:witan-core466 passedwitan-council770 passedwitan-code516 passedjust check-omnigraph-pins→omnigraph pins agree: 0.10.0 (tag edge)just check-omnigraph-format→omnigraph 0.10.0 reads storage format 6, as declared.prekcleanOne test needed a note:
test_pre_upgrade_candidates_exclude_the_current_binaryfails when two omnigraph binaries are onPATH, because its premise is that the scan finds nothing but the current one. That is my local setup (edge + the installed 0.9.0), not a 0.10.0 incompatibility — re-run with a single binary onPATH, it passes. CI installs exactly one.The baseline to compare against, measured today at
3e8e3ff: 8 concurrent writers PASS clean; 24 writers FAIL with 23 of 24 committed-but-reported-failed; server-side handler durations 3s → 73s on one graph. The comparison run should use the same probe invocation so the numbers are directly comparable.Deploying to CI restarts
omnigraph-server(replicas=1,Recreate), the same window #5396 needs.Does this change require a change to documentation?
No.