Skip to content

chore(omnigraph): run the 0.10.0 edge build to measure its effect on the write ceiling - #231

Merged
blarghmatey merged 2 commits into
mainfrom
worktree-omnigraph-0100-edge
Aug 13, 2026
Merged

chore(omnigraph): run the 0.10.0 edge build to measure its effect on the write ceiling#231
blarghmatey merged 2 commits into
mainfrom
worktree-omnigraph-0100-edge

Conversation

@blarghmatey

Copy link
Copy Markdown
Member

What are the relevant tickets?

tk-upstream-omnigraph-a-single-row-insert-costs-a-f-eeeae3, tk-concurrent-writes-to-the-deployed-witan-are-canc-02abbd, tk-omnigraph-conditional-write-cas-precondition-on--94155f, under wp-witan-multi-user-service-deployment-dcf6ee.

Description (What does it do?)

Pins all three omnigraph declarations to the upstream edge build (reports 0.10.0, 73 commits past v0.9.0), as a scoped experiment: the write path is what stands between this deployment and a team, and upstream main has landed insert-path work nobody has measured against our load.

Why this is cheap — which is the entire reason to try it

The v0.8.1 → v0.9.0 upgrade was expensive because the on-disk format broke: ~16 graphs per environment rebuilt via export → init → load, no downgrade, no canary. None of that applies. Verified before touching anything:

omnigraph 0.10.0 reads storage format 6, as declared.

Same format CI already runs. So this is a binary swap against the same store, and rollback is repointing the image.

Upstream publishes prebuilt binaries for every push to main (release-edge.yml force-updates the edge tag), so there is no source build involved.

Two pins, because the tag and the version genuinely differ

edge is a moving tag; the binary behind it reports 0.10.0, a version with no release to download. Folding both into one string breaks either the URL or the "already installed, skipping" check, which compares against what omnigraph --version prints. So:

  • _OMNIGRAPH_RELEASE_TAG — where to fetch from (edge, or v<version> for a real release)
  • _OMNIGRAPH_VERSION — what should come back (0.10.0)

just check-omnigraph-pins now enforces agreement on both, across all three files. Adding a second pin without extending that check would have recreated exactly the gap its own comment describes — the one that "went a full release cycle claiming the manager covered it when it did not".

⚠️ Known limitations of a moving tag

  • The skip check weakens. Two different edge builds both report 0.10.0, so a machine that installed yesterday's will not fetch today's. Delete the binary when you need certainty. Documented at the constant.
  • Renovate manages the VERSION line only, and a bump is not meaningful while the tag is edge. Pin a real v<version> before treating updates here as authoritative.
  • edge is unreleased. Fine for CI and for measurement; not for Production.

What this is, and is not, expected to fix

In 0.10.0: graph-commit conditional writes (CAS, upstream #470), exact write receipts (#479), and real insert-path work — strict inserts staged without merge joins, the Lance 10 migration, branch-merge join/index-build cliffs removed, filter pushdown to the Lance scanner.

Not in it: cancellation shielding (upstream #490) and the async job API with request-key idempotency (#415) are still open PRs. Those two are what actually make a write's outcome knowable to the client — so this build is not expected to fix the indeterminacy, only possibly to move the knee.

If the answer turns out to be "no change", that is still worth having: it isolates the ceiling to the per-write cost and points squarely at #490/#415 landing upstream.

How has this been tested?

Everything below ran with the edge binary on PATH:

  • binary-contract suite: 20/20 — the suite that caught 0.9.0's unannounced DateTime export wire change
  • witan-core 466 passed
  • witan-council 770 passed
  • witan-code 516 passed
  • just check-omnigraph-pinsomnigraph pins agree: 0.10.0 (tag edge)
  • just check-omnigraph-formatomnigraph 0.10.0 reads storage format 6, as declared.
  • prek clean

One test needed a note: test_pre_upgrade_candidates_exclude_the_current_binary fails when two omnigraph binaries are on PATH, because its premise is that the scan finds nothing but the current one. That is my local setup (edge + the installed 0.9.0), not a 0.10.0 incompatibility — re-run with a single binary on PATH, it passes. CI installs exactly one.

⚠️ Deploy note

The baseline to compare against, measured today at 3e8e3ff: 8 concurrent writers PASS clean; 24 writers FAIL with 23 of 24 committed-but-reported-failed; server-side handler durations 3s → 73s on one graph. The comparison run should use the same probe invocation so the numbers are directly comparable.

Deploying to CI restarts omnigraph-server (replicas=1, Recreate), the same window #5396 needs.

Does this change require a change to documentation?

No.

…the write ceiling

An experiment, deliberately scoped as one: the write path is the thing standing
between this deployment and a team, and upstream main has landed insert-path
work since v0.9.0 that nobody has measured against our load.

WHY THIS IS CHEAP, WHICH IS THE WHOLE REASON TO TRY IT. The v0.8.1 → v0.9.0
upgrade was expensive because the on-disk format broke: ~16 graphs per
environment rebuilt via export → init → load, no downgrade, no canary.
None of that applies here. Verified against the edge binary before touching
anything:

    omnigraph 0.10.0 reads storage format 6, as declared.

Same format we already migrated CI to, so this is a binary swap against the
same store, and rollback is repointing the image. The binary-contract suite —
the one that caught 0.9.0's unannounced DateTime export change — passes 20/20,
and the full suites pass against the edge binary: witan-core 466,
witan-council 770, witan-code 516.

★ TWO PINS NOW, BECAUSE THE TAG AND THE VERSION GENUINELY DIFFER. `edge` is a
moving tag that upstream force-updates on every push to main, and the binary
behind it reports 0.10.0 — a version that has no release to download. Folding
both into one string breaks either the URL or the "already installed, skipping"
check, which compares against what `omnigraph --version` actually prints. So
_OMNIGRAPH_RELEASE_TAG says where to fetch from and _OMNIGRAPH_VERSION says what
should come back.

`just check-omnigraph-pins` now enforces agreement on BOTH across all three
files. Adding a second pin without extending that check would have recreated
precisely the gap its own comment describes — the one that "went a full release
cycle claiming the manager covered it when it did not".

── KNOWN LIMITATIONS OF USING A MOVING TAG ──
* The skip check weakens: two different `edge` builds both report 0.10.0, so a
  machine that installed yesterday's will not fetch today's. Delete the binary
  when you need certainty. Documented at the constant.
* Renovate manages the VERSION line only, and a bump is not meaningful while
  the tag is `edge`. Pin a real v<version> before treating updates here as
  authoritative.
* `edge` is unreleased. Fine for CI and for measurement; not for Production.

── WHAT THIS IS AND IS NOT EXPECTED TO FIX ──
IN 0.10.0: graph-commit conditional writes (CAS, upstream #470) and exact write
receipts (#479), plus real insert-path work — strict inserts staged without
merge joins, the Lance 10 migration, branch-merge join/index-build cliffs
removed, filter pushdown.
NOT IN IT: cancellation shielding (#490) and the async job API with request-key
idempotency (#415) are still open PRs upstream. Those two are what actually make
a write's outcome knowable to the client, so this build is not expected to fix
the indeterminacy — only, possibly, to move the knee.

The baseline to compare against, measured today at 3e8e3ff: 8 concurrent
writers PASS clean; 24 writers FAIL with 23 of 24 committed-but-reported-failed;
server-side handler durations 3s → 73s on one graph.

Refs tk-upstream-omnigraph-a-single-row-insert-costs-a-f-eeeae3,
tk-concurrent-writes-to-the-deployed-witan-are-canc-02abbd,
tk-omnigraph-conditional-write-cas-precondition-on--94155f

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYb9sMjetD9Nxjf24Aw3m5
Copilot AI balanced review requested due to automatic review settings August 13, 2026 17:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Pins Omnigraph to the rolling edge build for write-performance testing.

Changes:

  • Separates release tag and reported version pins.
  • Updates both container images to download from edge.
  • Extends pin-consistency validation to include release tags.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
packages/witan-core/witan_core/omnigraph_install.py Adds separate version and release-tag handling.
justfile Validates tag and version agreement.
docker/witan.Dockerfile Fetches the edge CLI binary.
docker/omnigraph-server.Dockerfile Fetches edge server and CLI binaries.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread justfile
Comment thread packages/witan-core/witan_core/omnigraph_install.py Outdated
…erge

Review feedback on #231, and the reviewer is right that the previous commit did
not deliver what the PR claims. `edge` is force-updated on every push to
upstream main, so equal version and tag strings prove nothing: the installer
and the two image builds can each resolve the same tag to a DIFFERENT commit
while every check passes. For a change whose entire output is a load-test
measurement, that makes the result unattributable — which defeats the point of
running it.

Downloading the `.sha256` published beside the tarball does not fix it either.
That file moves with the release, so it only ever attests to whichever build
was current at download time.

So the digest is now pinned IN THIS REPO, for all three assets, and checked
before anything is extracted. `just check-omnigraph-pins` compares the
linux/x86_64 digest across all three tiers alongside the version and the tag —
verified to fail on a deliberately drifted value, not just asserted to.

★ AND THE INSTALLER VERIFIED NOTHING AT ALL BEFORE THIS. It put whatever the URL
returned onto a developer's PATH. That is now a hard refusal on mismatch, and a
hard refusal when an asset has no pinned digest — a platform added to
_OMNIGRAPH_ASSETS and forgotten in the digest map must fail closed, since the
inverse is how verification quietly stops applying.

The existing install tests built fake tarballs and were correctly refused by the
new check. They now register the fake's own digest through a `_serve` helper, so
they go THROUGH verification rather than around it, and two new tests cover the
refusal paths.

Verified by building the image, both ways: the pinned digest builds and reports
`omnigraph 0.10.0`; a deliberately wrong `--build-arg` fails the build at
`sha256sum -c`.

Also corrected the installer docstring, which told the reader to "pass a real
release tag" — there is no such argument, and the tag is deliberately a property
of the repo rather than of a run precisely because all three tiers must agree.
It now says what to actually do: delete the binary and re-run `witan setup`, or
edit the constant and refresh the digests in the same commit.

★ ONE CONSEQUENCE WORTH KNOWING: when upstream pushes to main, the next fetch
FAILS LOUDLY on the digest instead of silently installing a different binary.
Refreshing the pins is then a deliberate act — which is the property that makes
a measurement taken against this build mean something.

Refs tk-upstream-omnigraph-a-single-row-insert-costs-a-f-eeeae3

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JYb9sMjetD9Nxjf24Aw3m5
@blarghmatey
blarghmatey merged commit 8973144 into main Aug 13, 2026
13 checks passed
@blarghmatey
blarghmatey deleted the worktree-omnigraph-0100-edge branch August 13, 2026 18:17
blarghmatey added a commit that referenced this pull request Aug 14, 2026
Two things that belong in one commit because the revert has to be in the
released version: the predictive write gate ships, and the 0.10.0 edge
experiment ends.

── THE REVERT: 0.10.0 HALVED THE WRITE CEILING ──
#231 pinned the upstream `edge` build to measure its effect on the write
ceiling. That is the measurement, and the sign is wrong. Same probe, same
cluster, same actor, same evening:

    0.9.0    8 writers  PASS — 8 acked, 0 indeterminate, readers clean
    0.10.0   8 writers  FAIL x6 consecutive runs, 2 to 8 of 8 indeterminate,
                        every reader degraded in every run
    0.10.0   4 writers  PASS x2 — clean

★ THE CLEAN POINT MOVED FROM 8 TO 4. Higher loads are the same story, worse:
12w → 10 of 12 indeterminate, 16w → 10 of 16, 24w → 19 of 24 with the pod log
showing all 24 handlers `"outcome": "ok"` — every one of those writes committed
while its caller was told it failed.

0.10.0 also produces a shape 0.9.0 never did: a few genuine server-side errors,
~7ms, on `task_get` and `memory_store` at the same instant. Small in number and
not the main effect, but new.

Mutual exclusion is unaffected — probe A passed at every load on both builds,
exactly one `claimed: true` and every loser a structured refusal. What changed
is the write path's capacity, not its correctness.

★ NOTHING IN 0.10.0 WAS AIMED AT OUR PROBLEM ANYWAY. The two upstream changes
that address the indeterminate write — cancellation shielding and the async job
API — are not in it; the first was closed 2026-08-13 as superseded by RFC-035
(#498), which with RFC-034 (#496) and RFC-036 (#499) is an open design
document. So this build was carrying a measured regression in exchange for
nothing, on an unreleased binary off upstream main.

Reverting is as cheap as the upgrade was, for the same reason: storage format 6
is unchanged in both directions, so this is a binary swap against the same
store with no migration. Verified by building the fetch stage — the pinned
v0.9.0 digests produce `omnigraph 0.9.0`.

WEAKNESS TO CARRY: the 0.9.0 side is ONE run at 8 writers against six on
0.10.0. Six consecutive failures where 0.9.0 was clean is strong, but the clean
baseline is not itself replicated. Re-run 8w a few times on 0.9.0 before
concluding anything upstream.

── THE RELEASE: witan-core 0.18.0 ──
The predictive write gate (#232). The gate previously bounded concurrency and
the wait for a slot, and neither predicts how long the work takes once
admitted — so it let through exactly the writes that strand. It now measures
what an admitted write costs per graph and refuses when that no longer fits the
call's remaining budget.

This does not raise the ceiling; it converts the writes that cannot succeed
into a clean pre-send refusal instead of an indeterminate outcome.

Refs tk-the-write-gate-is-sized-against-a-3-45s-solo-wri-73fc2b
Memory: les-omnigraph-0-10-0-edge-looks-worse-than-0-9-0-on--ef8f5d


Claude-Session: https://claude.ai/code/session_01JYb9sMjetD9Nxjf24Aw3m5

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants