fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload) - #1506
Closed
kriszyp wants to merge 3 commits into
Closed
fix(encoder): heal a diverged shared-structures dictionary on decode (durable-wins reload)#1506kriszyp wants to merge 3 commits into
kriszyp wants to merge 3 commits into
Conversation
…(durable-wins reload)
A v5 worker can strand an in-memory classic shared-structures dictionary whose
id->shape order diverges from the canonical durable order - most often via the
replication-apply / full-copy re-encode path, which mints structures off the
saveStructures CAS. Because msgpackr only auto-reloads on a *missing* id (and its
decode-time merge is in-memory-wins), a "present-but-wrong" decode ("Unexpected
end of MessagePack data" / "Data read, but end of buffer not reached") never
self-heals: every read of the affected ids returns null and the node serves wrong
counts / empty reads until restart. No durable data loss (restart recovers), but
an outage-class per-node wedge under v4->v5 upgrade write load.
Reproduced on a 4-node 4.7.34->5.1.14 in-place rolling upgrade at 100 w/s: the
mid-rolling node accumulates 100k+ such decode errors and fails to converge.
- decode(): on a present-but-wrong (isStructureMismatchError) decode, reload the
durable dictionary AUTHORITATIVELY (durable-wins, one-arg _mergeStructures) and
retry once, guarded against recursion. Heals the diverged worker transparently.
- saveStructures(): read the latest committed structures fresh
(this.rootStore.getBinarySync) instead of the txn start-snapshot, matching
rocksdb-js's own saveStructures, so the compatibility check sees concurrent peers.
- saveStructures(): enforce the append-only invariant - reject a same-length but
reordered dictionary (isFaithfulExtension) that the length-only isCompatible
would otherwise admit, clobbering a peer's committed order.
Validated: the in-place upgrade repro goes green (exact ledger parity on all four
nodes, 100% value-readable, recovery fires and heals, no wedge).
The off-CAS divergence source (replication-apply re-encode not reconciling through
the structures CAS, and #1455's canonical seed persisted in a format the typed-
table runtime discards) is tracked as a follow-up; this change makes the condition
self-healing regardless.
Refs #1453, #1445
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to detect and recover from 'present-but-wrong' structure mismatch errors during decoding in RecordEncoder.ts by reloading the canonical durable structures and retrying. It also enforces an append-only invariant on structures to prevent silent corruption. The feedback recommends hardening namedStructures to guarantee it always returns an array to prevent bypassing the append-only check, and improving error handling and logging within the recovery block of decode() to prevent silent failures if _mergeStructures is unavailable or throws an error.
Contributor
|
Reviewed; no blockers found. |
kriszyp
marked this pull request as draft
June 26, 2026 21:20
This was referenced Jun 26, 2026
Member
Author
|
Repositioned as defense-in-depth and holding in draft: the root cause is fixed at the source by #1520 (the migration now persists the canonical seed; |
Address gemini-code-assist review threads on PR #1506: - namedStructures() now validates the extracted `named` field is actually an array before returning it, so a malformed dict (non-array `named`) can't silently bypass isFaithfulExtension's append-only check. - The present-but-wrong recovery catch in decode() now logs the swallowed error instead of discarding it, so a failed _mergeStructures/getStructures reload is diagnosable instead of masked by the generic mismatch error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…is.rootStore The 3eb50bb fresh-read change moved saveStructures()'s compatibility-check read from txn.getBinarySync() to this.rootStore.getBinarySync() to avoid a stale txn-snapshot admitting a divergent write. But reading via rootStore bypasses RocksDB's optimistic-transaction conflict tracking for the shared- structures key: two concurrent CAS transactions can each read a compatible "existing" state and both commit, with neither detecting the other, silently overwriting one side's newly-minted structure ids. Records already written against the clobbered ids then permanently fail to decode ("Data read, but end of buffer not reached") or — when the collision lands on a same-length shape — silently decode with values attributed to the wrong field. Reproduced via integrationTests/database/concurrentPatchMerge.test.ts and patch-disjoint-field-merge.test.ts (PR #1506 CI, run 29377186730): concurrent disjoint-field PATCH on a typed table mints a new shared structure per field, racing exactly this path. Confirmed diff-tied (fails on the PR merge commit, 0/6 clean across two test files pre-fix; passes cleanly on origin/main and 17/17 post-fix) and bisected to this specific read. Reading through the transaction restores RocksDB's write-conflict detection: a concurrent committer of the same key now fails THIS transaction's commit, and retryOnBusy re-runs the callback with a fresh txn/read — achieving the same "see concurrent peers" goal the fresh read was going for, without sacrificing the CAS guarantee. isFaithfulExtension's append-only check is unchanged and still guards the same-length-reorder case. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Member
Author
|
Closing per review — the shared-structures divergence this heals on-decode is addressed by the merged root-cause fix (#1520); the durable-wins reload path here isn't needed as a separate change. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Status: defense-in-depth (held) — root cause now fixed by #1520
The v4→v5 structure-dictionary fork's root cause is the migration not persisting the canonical seed:
copyDb'sshapeForStructurestubbed theRecordObjects the migration reads (value.constructor === Objectgate), so the observer minted no structure and the seed was skipped — every v5 worker then minted the dictionary from an empty durable and forked it. That's fixed at the source by #1520. With the seed persisted, workers adopt one canonical dictionary and don't diverge.This PR is therefore no longer the primary fix — it's defense-in-depth:
isFaithfulExtensionsaveStructuresguard here is not load-bearing — the optimistic-transaction retry + durable-wins merge already converge concurrent appends (thanks @kzyp for pushing on this). It's kept only as cheap belt-and-suspenders; if this PR is promoted it should likely be trimmed to just the decode-heal.Holding in draft pending evidence it's needed beyond #1520. If residual divergence shows up in the field after #1520 ships — or we want no-restart recovery as insurance — promote (and trim) this; otherwise close it.
Superseded as the primary fix by #1520. Refs #1508, #1453.
— repositioned by KrAIs (Claude Opus 4.8)