feat(minidb): switch agent-core-v2 query-store to ClusterDb with 16 shards - #1907
Conversation
…hards - ClusterDb: add query() (per-shard fan-out with skip=0 and limit=skip+limit, global re-sort, then skip/limit) and compound index management (create/drop/list through the cluster registry, fanned out to every shard and caught up on shard open) - MiniDbQueryStore: replace the single MiniDb with a 16-shard ClusterDb, so multiple kimi processes share the read model instead of failing with storage.locked; per-shard LockError now propagates as a transient error rather than permanently disabling the read model - corruption: lift openOrRebuild's predicate (SyntaxError / CorruptFrameError) to one process-lifetime wipe-and-reopen rebuild - lower lockAcquireTimeoutMs to 1s for the cache read model - tests: cluster query merge and compound fan-out; store coexistence with a peer instance, corrupt-registry rebuild, 16-shard topology; sessionIndex locked-fallback test now drives a stub IQueryStore
|
commit: |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: df6c020179
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| const needed = skip + limit; | ||
| const all: ScanEntry<V>[] = []; | ||
| for (const id of this.router.shardIds()) { | ||
| const rows = await this.reader(id, (db) => db.query({ ...q, skip: 0, limit: needed })); |
There was a problem hiding this comment.
When a caller combines sort with a project that omits the sort field, this forwards project into every shard query. MiniDb.query projects after its local sort, so the rows returned to the cluster no longer contain the field needed by the global merge comparator; query({ sort: { age: -1 }, project: ['name'], limit: 10 }) can therefore return shard-order results instead of the highest ages when matches span shards. Fan out without project, then apply projection after the global sort and slice.
Useful? React with 👍 / 👎.
| const av = getPath(a.value, p) as number | string; | ||
| const bv = getPath(b.value, p) as number | string; |
There was a problem hiding this comment.
For MiniDb.query, sorting on the same column supplied in q.dt is supported by the dt-ordered fast path, and the column is carried on ScanEntry.dt rather than inside value. This merge comparator only reads a.value, so ClusterDb.query({ dt: { ts: ... }, sort: { ts: -1 }, limit: ... }) compares every row as equal when documents do not duplicate ts in their JSON value, then slices in shard order. Use entry.dt?.[p] for dt-backed sort keys or merge the per-shard dt order explicitly.
Useful? React with 👍 / 👎.
| } else { | ||
| all.sort(compareEntries); |
There was a problem hiding this comment.
When q.text is set without an explicit sort, MiniDb.query returns each shard's matches in text-rank order, but this no-sort branch reorders all merged matches by key bytes before the final limit. For ClusterDb.query({ text: ..., limit: n }), lexicographically early but less relevant keys can displace more relevant hits from other shards, so the public text-query semantics diverge from MiniDb even before considering approximate cross-shard IDF. Carry/merge text ranks, or only fall back to key-byte ordering when there is no text ranking.
Useful? React with 👍 / 👎.
Upstream changes: - feat(cli): replace kimi server command tree with kimi web (MoonshotAI#1826) - Removed daemon/server lifecycle (svc/ layer for launchd/systemd/schtasks) - Multi-server instance registry replaces single-writer lock - /web TUI command uses server picker instead of daemon logic - feat(minidb): switch agent-core-v2 query-store to ClusterDb with 16 shards (MoonshotAI#1907) - Multiple kimi processes can share the read model concurrently - feat: scope thinking effort to current session (MoonshotAI#1933) - feat: add thinking_effort to turn telemetry events (MoonshotAI#1916) - fix(agent-core-v2): host fs stat follows symlinks, add explicit lstat (MoonshotAI#1843) - feat(web): add cache invalidation note to model switcher (MoonshotAI#1940) - ci: release packages (MoonshotAI#1868) Conflicts resolved: - 9 modify/delete conflicts: accepted upstream deletion (server/*, svc/*) - 6 content conflicts: merged while preserving fork's i18n t() calls and fork-specific test suites (config edge cases)
1 participant
Related Issue
No linked issue — the problem is explained in the next section.
Problem
The minidb-backed derived read model (
IQueryStore, behind the experimentalpersistence_minidb_readmodelflag) currently runs on a singleMiniDbwith a database-wide single-writer lock. Any second kimi process sharing the same home fails withstorage.locked, and the read model is then permanently disabled for that process (falling back to legacy N+1 disk reads).minidb already shipped
ClusterDb— hash-routed shards over ordinary MiniDb directories with a per-shard lock pool, built for exactly this multi-process scenario — but its v1 cut left out the two pieces the query-store backend needs: the unifiedquery()and compound index management. This PR completes those pieces and moves the query-store onto a 16-shardClusterDb.What changed
1. minidb: complete the
ClusterDbAPI needed by the query-store (pure additions, no behavior change)Problem:
ClusterDbhad noquery()and no compound index management, so the backend could not migrate.What was done:
query(): every shard runs the fullMiniDb.querylocally withskip=0andlimit=skip+limit(index-assisted candidate pruning included — the global top-(skip+limit) is contained in each shard's local top-(skip+limit)); merged results are re-sorted globally (explicit sort, else key bytes to matchscan) andskip/limitapplies at the end. Text scoring stays per-shard (approximate global ranking), the same caveatsearch()already documents.createCompoundIndex/dropCompoundIndex/listCompoundIndexes, fanned out to every shard via the cluster registry with rollback on partial fan-out and catch-up on shard open.cluster.indexes.jsongains acompoundIndexesfield, normalized with?? []on both read paths so old and new files load either way.2. agent-core-v2:
MiniDbQueryStorenow runs on a 16-shardClusterDbProblem: the single-
MiniDbbackend locked out every concurrent process viastorage.locked.What was done:
ClusterDb(shardCount: 16) at the same<cacheDir>/query-storelocation with the same codec/durability settings;lockAcquireTimeoutMslowered to 1s (a cache read must not hang behind a contended shard).LockErrorpropagates as a transient error instead of permanently disabling the read model throughstorage.locked(theFileSessionIndexfallback wiring stays as defense in depth).openOrRebuild's predicate (SyntaxError/CorruptFrameError) to the cluster: one process-lifetime wipe-and-reopen rebuild with a single retry of the triggering op; consumers reproject from checkpoints.batchis atomic per shard and best-effort across shards (theClusterDbdefault).sessionIndexlocked-fallback test now drives a stubIQueryStoresince the backend no longer producesstorage.locked.The feature remains gated behind the experimental
persistence_minidb_readmodelflag (default off). No changeset: internal packages only (@moonshot-ai/minidbis private,agent-core-v2is not published), per maintainer request.Checklist
gen-changesetsskill, or this PR needs no changeset.gen-docsskill, or this PR needs no doc update.