feat(minidb): ClusterDb sharding, incremental reader catch-up, and engine hardening - #1816
Merged
Merged
Conversation
Cluster layer: - add ClusterDb: hash-routed keys over N MiniDb shard directories with a per-shard lock pool (lease renewal, lockHoldMs yield, takeover on dead PID), merged ordered scans, cross-shard index registry, live cross-process read visibility, and crash-recovery handoff - add cluster bench suite and multi-process test suites Engine hardening (stress-driven fixes): - compaction: pre-copy now gives up when the tail copy is not converging and rotation seals the old WAL (retryable), so compaction always terminates under sustained write storms and no committed write slips through rotation - recovery: re-sync WAL size bookkeeping after torn-tail truncation; read-only opens create/modify no files and never compact under a live writer - lockfile: stale-lock takeover via atomic bid-rename + settle; release only unlinks its own pid - query: streaming candidates with skip/limit applied before materialization, plus Store.rawKeys for value-free key scans - eviction: O(1) LRU victim picking via insertion-ordered access set - TTL: adaptive expire budget drains simultaneous-expiry storms in seconds - store: O(1) size fast path when no TTL is set; has() no longer materializes disk-backed values - openOrRebuild preserves data when only a sidecar definition file is corrupt; sidecar definitions written atomically; stale compaction temps cleaned on open - RESP server serializes replies per connection; over-64KiB tokens can no longer poison the full-text index
- cluster readers: track a per-shard WAL watermark (dev/ino/offset) and catch up from appended frames instead of fully reopening; fall back to a full reopen on rotation, truncation, or index-definition changes - MiniDb.catchUpFromWal applies WAL tail frames to a live instance (store plus secondary/dt/compound/text indexes), sharing recover()'s frame interpretation; RecoveryInfo exposes walScanEnd/dev/ino as the safe anchor - lock-pool: incremental refresh with stats (incrementalCatchups, catchupFramesApplied); bench/reader-catchup shows p50 read latency drop from 130ms to 0.4ms at 10k keys and from 532ms to 0.4ms at 50k keys while a neighbor process writes - compaction: keep the db writable on rotation failure (fresh WAL swap, remap once); count stats.compactions only on full success including the onCompacted hook - restoreKey: seq guard so a failed op never wipes a concurrently committed value; eviction DELs retry on WAL seal - text index: atomic build (stage then swap), createTextIndex registers only after a successful build, open-time cleanup of db.text-*.postings.tmp - RESP server: swallow per-connection socket errors, reset the parser buffer after an oversized request, isolate per-command errors while preserving reply order
- compaction rotation: on Windows, renaming over an open destination is EPERM, so replace renames with a retrying renameReplace helper and let go of the db's own ValueReader handles for the renames (reopened right after the pointer remap) - value reader: hold snapshot/WAL handles only in valueMode 'disk'; in memory mode the handles were idle and, on Windows, blocked rotation - lock takeover: retry the takeover bid's rename on Windows, but re-check the corpse before every attempt — a blind retry loop could land our bid late and overwrite an already-verified winner, double-holding the lock - lock takeover: raise the co-bidder settle window to 50ms so loaded CI machines with tens-of-milliseconds descheduling still elect one winner - test hardening for shared CI runners: a 30s minidb-wide timeout floor, batched prefills instead of sequential setup loops, explicit timeouts for process-spawning and heavy e2e tests, and Vitest 4 test() signature normalization Verified green across macOS (arm64), Windows Server 2022 (x64, 2-core), and Ubuntu 22.04 (launchpad aarch64): 321 passed, 1 skipped in each.
🦋 Changeset detectedLatest commit: 9c6cc4e The changes in this PR will be included in the next version bump. This PR includes changesets to release 3 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2bfe4cf6c8
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+354
to
+359
| db.listIndexes().some((i) => i.name === name) ? db.findRange(name, opts) : [], | ||
| ); | ||
| out.push(...rows); | ||
| } | ||
| out.sort((a, b) => a.field - b.field || compareEntries({ key: a.key, value: a.value }, { key: b.key, value: b.value })); | ||
| return out; |
There was a problem hiding this comment.
When callers pass range options with global semantics, this returns the wrong rows: count/offset are applied independently inside each shard and then all shard results are returned, while reverse is discarded by the unconditional ascending sort here. A clustered findRange(..., { reverse: true, count: 10 }) can therefore return low values and/or more than 10 rows; merge/sort all shard candidates (or top candidates per shard) and apply reverse, offset, and count once globally.
Useful? React with 👍 / 👎.
Comment on lines
+271
to
+273
| const tmp = `${file}.tmp-${process.pid}`; | ||
| await fs.writeFile(tmp, JSON.stringify(reg, null, 2)); | ||
| await fs.rename(tmp, file); |
There was a problem hiding this comment.
When two ClusterDb processes create or drop different indexes concurrently, each can load the same old cluster.indexes.json, mutate it, and this save blindly replaces the whole file; the last rename loses the other process's definition even though its shard sidecars were already updated. Because the registry is the source of truth for listIndexes, requireIndex, and future shard opens, this leaves indexes invisible or unapplied after normal multi-process administration; protect the load-modify-save with a registry lock/CAS merge.
Useful? React with 👍 / 👎.
Comment on lines
+300
to
+304
| await this.forEachShardWriter(async (db) => { | ||
| if (!db.listIndexes().some((i) => i.name === name)) await db.createIndex(name, def); | ||
| }); | ||
| reg.indexes.push({ name, def }); | ||
| await ClusterDb.saveRegistry(this.indexPath, reg); |
There was a problem hiding this comment.
If db.createIndex throws after earlier shards succeeded (for example, creating a unique index where a later shard contains duplicates, or an I/O failure while writing a later sidecar), this method exits before saving the registry and leaves the successful shards with a local db.indexes.json. Those hidden indexes are not reported by ClusterDb.listIndexes/findEq, but writes to those shards still maintain/enforce them, so a failed create can permanently change behavior until manually cleaned up.
Useful? React with 👍 / 👎.
- takeover now registers a liveness watch file before touching the lock, so every contender is visible to every other for the whole attempt; the settle/verify loop abstains while any live foreign watch exists. Settle-only heuristics could not survive a bidder descheduled before its bid write on shard-parallel CI runners (observed double-holds on ubuntu-latest and on a 2-core Windows box). - adaptive settle scales with the attempt's own wall clock (floored, capped), replacing the fixed window. - keep the Windows EPERM tolerance in the bid rename, re-inspecting the corpse before every attempt on ALL platforms, not just win32. - lint: fix restrict-template-expressions in an e2e RESP helper. - test: widen the cluster wait-read budget for slow CI spawns.
- wrap the compaction-storm error interpolation as String() so the type-aware restrict-template-expressions lint passes - agent-core-v2's minidb query-store corruption test now expects the intended semantics: a corrupt index-definition sidecar is dropped while the data survives, and the definition can be re-registered
Address three multi-process administration races in the cluster layer: - findRange now merges candidates from all shards first and only then applies reverse/offset/count globally, instead of clipping per shard and discarding reverse - cluster.indexes.json mutations go through a compare-and-swap loop (reload, re-apply idempotently, publish, verify) with an in-process mutex, so concurrent create/drop from two processes loses neither registry entries nor shard sidecars - a failed createIndex/createTextIndex fan-out now rolls back exactly the shards it already created on, so the registry and every shard agree whether an index exists Also make LockFile sidecars (tmp/bid/watch) unique per acquire attempt: two lock users in the same process (independent shard pools) must never share a path, or one user's cleanup would delete the other's in-flight file.
Merged
7723qqq
pushed a commit
to 7723qqq/kimi-code
that referenced
this pull request
Jul 17, 2026
…gine hardening (MoonshotAI#1816) * feat(minidb): add ClusterDb sharding and harden engine under stress Cluster layer: - add ClusterDb: hash-routed keys over N MiniDb shard directories with a per-shard lock pool (lease renewal, lockHoldMs yield, takeover on dead PID), merged ordered scans, cross-shard index registry, live cross-process read visibility, and crash-recovery handoff - add cluster bench suite and multi-process test suites Engine hardening (stress-driven fixes): - compaction: pre-copy now gives up when the tail copy is not converging and rotation seals the old WAL (retryable), so compaction always terminates under sustained write storms and no committed write slips through rotation - recovery: re-sync WAL size bookkeeping after torn-tail truncation; read-only opens create/modify no files and never compact under a live writer - lockfile: stale-lock takeover via atomic bid-rename + settle; release only unlinks its own pid - query: streaming candidates with skip/limit applied before materialization, plus Store.rawKeys for value-free key scans - eviction: O(1) LRU victim picking via insertion-ordered access set - TTL: adaptive expire budget drains simultaneous-expiry storms in seconds - store: O(1) size fast path when no TTL is set; has() no longer materializes disk-backed values - openOrRebuild preserves data when only a sidecar definition file is corrupt; sidecar definitions written atomically; stale compaction temps cleaned on open - RESP server serializes replies per connection; over-64KiB tokens can no longer poison the full-text index * feat(minidb): incremental WAL catch-up for cluster readers - cluster readers: track a per-shard WAL watermark (dev/ino/offset) and catch up from appended frames instead of fully reopening; fall back to a full reopen on rotation, truncation, or index-definition changes - MiniDb.catchUpFromWal applies WAL tail frames to a live instance (store plus secondary/dt/compound/text indexes), sharing recover()'s frame interpretation; RecoveryInfo exposes walScanEnd/dev/ino as the safe anchor - lock-pool: incremental refresh with stats (incrementalCatchups, catchupFramesApplied); bench/reader-catchup shows p50 read latency drop from 130ms to 0.4ms at 10k keys and from 532ms to 0.4ms at 50k keys while a neighbor process writes - compaction: keep the db writable on rotation failure (fresh WAL swap, remap once); count stats.compactions only on full success including the onCompacted hook - restoreKey: seq guard so a failed op never wipes a concurrently committed value; eviction DELs retry on WAL seal - text index: atomic build (stage then swap), createTextIndex registers only after a successful build, open-time cleanup of db.text-*.postings.tmp - RESP server: swallow per-connection socket errors, reset the parser buffer after an oversized request, isolate per-command errors while preserving reply order * chore(minidb): add changesets for reader catch-up and review hardening * fix(minidb): support Windows in WAL rotation and lock takeover - compaction rotation: on Windows, renaming over an open destination is EPERM, so replace renames with a retrying renameReplace helper and let go of the db's own ValueReader handles for the renames (reopened right after the pointer remap) - value reader: hold snapshot/WAL handles only in valueMode 'disk'; in memory mode the handles were idle and, on Windows, blocked rotation - lock takeover: retry the takeover bid's rename on Windows, but re-check the corpse before every attempt — a blind retry loop could land our bid late and overwrite an already-verified winner, double-holding the lock - lock takeover: raise the co-bidder settle window to 50ms so loaded CI machines with tens-of-milliseconds descheduling still elect one winner - test hardening for shared CI runners: a 30s minidb-wide timeout floor, batched prefills instead of sequential setup loops, explicit timeouts for process-spawning and heavy e2e tests, and Vitest 4 test() signature normalization Verified green across macOS (arm64), Windows Server 2022 (x64, 2-core), and Ubuntu 22.04 (launchpad aarch64): 321 passed, 1 skipped in each. * fix(minidb): make stale-lock takeover exactly-one under CI load - takeover now registers a liveness watch file before touching the lock, so every contender is visible to every other for the whole attempt; the settle/verify loop abstains while any live foreign watch exists. Settle-only heuristics could not survive a bidder descheduled before its bid write on shard-parallel CI runners (observed double-holds on ubuntu-latest and on a 2-core Windows box). - adaptive settle scales with the attempt's own wall clock (floored, capped), replacing the fixed window. - keep the Windows EPERM tolerance in the bid rename, re-inspecting the corpse before every attempt on ALL platforms, not just win32. - lint: fix restrict-template-expressions in an e2e RESP helper. - test: widen the cluster wait-read budget for slow CI spawns. * fix(minidb): clean remaining CI lint and consumer-test failures - wrap the compaction-storm error interpolation as String() so the type-aware restrict-template-expressions lint passes - agent-core-v2's minidb query-store corruption test now expects the intended semantics: a corrupt index-definition sidecar is dropped while the data survives, and the definition can be re-registered * fix(minidb): harden ClusterDb index administration across processes Address three multi-process administration races in the cluster layer: - findRange now merges candidates from all shards first and only then applies reverse/offset/count globally, instead of clipping per shard and discarding reverse - cluster.indexes.json mutations go through a compare-and-swap loop (reload, re-apply idempotently, publish, verify) with an in-process mutex, so concurrent create/drop from two processes loses neither registry entries nor shard sidecars - a failed createIndex/createTextIndex fan-out now rolls back exactly the shards it already created on, so the registry and every shard agree whether an index exists Also make LockFile sidecars (tmp/bid/watch) unique per acquire attempt: two lock users in the same process (independent shard pools) must never share a path, or one user's cleanup would delete the other's in-flight file.
ywh114
pushed a commit
to ywh114/kimi-code
that referenced
this pull request
Jul 19, 2026
…gine hardening (MoonshotAI#1816) * feat(minidb): add ClusterDb sharding and harden engine under stress Cluster layer: - add ClusterDb: hash-routed keys over N MiniDb shard directories with a per-shard lock pool (lease renewal, lockHoldMs yield, takeover on dead PID), merged ordered scans, cross-shard index registry, live cross-process read visibility, and crash-recovery handoff - add cluster bench suite and multi-process test suites Engine hardening (stress-driven fixes): - compaction: pre-copy now gives up when the tail copy is not converging and rotation seals the old WAL (retryable), so compaction always terminates under sustained write storms and no committed write slips through rotation - recovery: re-sync WAL size bookkeeping after torn-tail truncation; read-only opens create/modify no files and never compact under a live writer - lockfile: stale-lock takeover via atomic bid-rename + settle; release only unlinks its own pid - query: streaming candidates with skip/limit applied before materialization, plus Store.rawKeys for value-free key scans - eviction: O(1) LRU victim picking via insertion-ordered access set - TTL: adaptive expire budget drains simultaneous-expiry storms in seconds - store: O(1) size fast path when no TTL is set; has() no longer materializes disk-backed values - openOrRebuild preserves data when only a sidecar definition file is corrupt; sidecar definitions written atomically; stale compaction temps cleaned on open - RESP server serializes replies per connection; over-64KiB tokens can no longer poison the full-text index * feat(minidb): incremental WAL catch-up for cluster readers - cluster readers: track a per-shard WAL watermark (dev/ino/offset) and catch up from appended frames instead of fully reopening; fall back to a full reopen on rotation, truncation, or index-definition changes - MiniDb.catchUpFromWal applies WAL tail frames to a live instance (store plus secondary/dt/compound/text indexes), sharing recover()'s frame interpretation; RecoveryInfo exposes walScanEnd/dev/ino as the safe anchor - lock-pool: incremental refresh with stats (incrementalCatchups, catchupFramesApplied); bench/reader-catchup shows p50 read latency drop from 130ms to 0.4ms at 10k keys and from 532ms to 0.4ms at 50k keys while a neighbor process writes - compaction: keep the db writable on rotation failure (fresh WAL swap, remap once); count stats.compactions only on full success including the onCompacted hook - restoreKey: seq guard so a failed op never wipes a concurrently committed value; eviction DELs retry on WAL seal - text index: atomic build (stage then swap), createTextIndex registers only after a successful build, open-time cleanup of db.text-*.postings.tmp - RESP server: swallow per-connection socket errors, reset the parser buffer after an oversized request, isolate per-command errors while preserving reply order * chore(minidb): add changesets for reader catch-up and review hardening * fix(minidb): support Windows in WAL rotation and lock takeover - compaction rotation: on Windows, renaming over an open destination is EPERM, so replace renames with a retrying renameReplace helper and let go of the db's own ValueReader handles for the renames (reopened right after the pointer remap) - value reader: hold snapshot/WAL handles only in valueMode 'disk'; in memory mode the handles were idle and, on Windows, blocked rotation - lock takeover: retry the takeover bid's rename on Windows, but re-check the corpse before every attempt — a blind retry loop could land our bid late and overwrite an already-verified winner, double-holding the lock - lock takeover: raise the co-bidder settle window to 50ms so loaded CI machines with tens-of-milliseconds descheduling still elect one winner - test hardening for shared CI runners: a 30s minidb-wide timeout floor, batched prefills instead of sequential setup loops, explicit timeouts for process-spawning and heavy e2e tests, and Vitest 4 test() signature normalization Verified green across macOS (arm64), Windows Server 2022 (x64, 2-core), and Ubuntu 22.04 (launchpad aarch64): 321 passed, 1 skipped in each. * fix(minidb): make stale-lock takeover exactly-one under CI load - takeover now registers a liveness watch file before touching the lock, so every contender is visible to every other for the whole attempt; the settle/verify loop abstains while any live foreign watch exists. Settle-only heuristics could not survive a bidder descheduled before its bid write on shard-parallel CI runners (observed double-holds on ubuntu-latest and on a 2-core Windows box). - adaptive settle scales with the attempt's own wall clock (floored, capped), replacing the fixed window. - keep the Windows EPERM tolerance in the bid rename, re-inspecting the corpse before every attempt on ALL platforms, not just win32. - lint: fix restrict-template-expressions in an e2e RESP helper. - test: widen the cluster wait-read budget for slow CI spawns. * fix(minidb): clean remaining CI lint and consumer-test failures - wrap the compaction-storm error interpolation as String() so the type-aware restrict-template-expressions lint passes - agent-core-v2's minidb query-store corruption test now expects the intended semantics: a corrupt index-definition sidecar is dropped while the data survives, and the definition can be re-registered * fix(minidb): harden ClusterDb index administration across processes Address three multi-process administration races in the cluster layer: - findRange now merges candidates from all shards first and only then applies reverse/offset/count globally, instead of clipping per shard and discarding reverse - cluster.indexes.json mutations go through a compare-and-swap loop (reload, re-apply idempotently, publish, verify) with an in-process mutex, so concurrent create/drop from two processes loses neither registry entries nor shard sidecars - a failed createIndex/createTextIndex fan-out now rolls back exactly the shards it already created on, so the registry and every shard agree whether an index exists Also make LockFile sidecars (tmp/bid/watch) unique per acquire attempt: two lock users in the same process (independent shard pools) must never share a path, or one user's cleanup would delete the other's in-flight file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
No linked issue — the problem is explained in the next section.
Problem
@moonshot-ai/minidb(embedded WAL+snapshot engine) needed three reinforcements to be usable by concurrent backend services:What changed
1. ClusterDb sharding
Problem: single-writer directories blocked multi-process use.
What was done: hash-routed keys over N MiniDb shard directories with a per-shard lock pool (lease renewal, lockHoldMs yield, takeover on dead PID), merged ordered scans, a cluster-wide index registry, live cross-process read visibility, crash-recovery handoff, plus a cluster bench suite and multi-process test suites.
2. Incremental WAL catch-up for cluster readers
Problem: every cross-process invalidation triggered a full shard reopen (O(shard size)).
What was done: readers now track a per-shard WAL watermark (dev/ino/offset) and catch up from appended frames via
MiniDb.catchUpFromWal— sharingrecover()'s exact frame interpretation (LWW, batches, TTL, all derived index types) — and fall back to a full reopen on rotation/truncation/definition changes.bench/reader-catchup.ts(committed, with before/after numbers) shows p50 read latency from 130.5ms → 0.4ms @10k keys and 532.2ms → 0.4ms @50k keys while a neighbor process writes.3. Engine hardening under stress
Problem: a stress-test round (see
test/e2e/stress.test.ts) exposed durability/availability bugs: acknowledged writes could slip through a WAL rotation under write storms, torn-tail recovery could desync later disk-mode pointers, read-only opens could compact (and destroy) a live writer's files, the stale process lock could be taken over by several processes at once, and one oversized token could poison the full-text index.What was done: rotation sealing with retryable write commits so no committed write slips through; WAL size bookkeeping re-synced after torn-tail truncation; read-only opens create/modify no files and never compact; stale-lock takeover via atomic bid-rename + settle; token-size guard in the text tokenizer; plus perf items (streaming query candidates, O(1) LRU, adaptive TTL drain, O(1) size) and sidecar/definition-file resilience.
4. Windows compatibility + CI stability
Problem: 47/322 tests failed on Windows Server 2022 — all rooted in rename-over-open-destination EPERM — and several tests used deprecated/removed Vitest signatures and tight timeouts that flake on CI.
What was done:
rotateReplace/sharedrenameReplacehelper rides out transient sharers on Windows; the db's own ValueReader only pins files invalueMode: 'disk'and is closed around rotation renames; the lock takeover bid re-checks the corpse before every rename retry (a blind retry lets a late bid overwrite an already-verified winner — reproducibly double-holding on a 2-core box); takeover settle raised to 50ms. Tests: minidb-wide 30s timeout floor, batched prefills, explicit timeouts for process-spawning/heavy suites, Vitest 4test()signature updates.Verification
Checklist
gen-changesetsskill, or this PR needs no changeset. (4 changesets included; the Windows-compat commit intentionally carries none.)gen-docsskill, or this PR needs no doc update. (packages/minidb/README.mdandDESIGN_NOTES.mdupdated; the change is internal to the minidb package — no CLI user-doc impact.)