MooncakeStoreConnector: the KV store half is linkable native C++ and gateable over TCP on one box

[localai-bot]

Summary

vLLM ships two Mooncake connectors, and our records treat them as one thing.
.agents/engine-matrix.md (row KV-CONNECTORS) dispositions both as NOT
SCHEDULED, on the grounds that each "needs an external RDMA or store dependency
absent from our boxes and ungateable on GB10". Reading the source at the pin
(555967922), that disposition is correct for one of them and too strong for the
other.

	MooncakeConnector	MooncakeStoreConnector
Shape	P2P prefiller→decoder push over the Mooncake Transfer Engine	Shared KV object store pool
Our analogue	NIXL	LMCache
Upstream	v1/mooncake/mooncake_connector.py (2179 lines)	v1/mooncake/store/ (3554 lines)
Needs to run	2 nodes + bootstrap FastAPI server + disagg proxy + RDMA	one mooncake_master; single node; protocol: "tcp" is a supported fallback

This issue scopes only MooncakeStoreConnector. MooncakeConnector (P/D
disaggregation) stays NOT SCHEDULED and is explicitly out of scope.

Why this is cheaper for us than it is for vLLM

LMCache is a Python library. We could not link it, so KV-EXTERNAL-CACHE W1/W2
reimplemented the lm:// wire protocol from scratch in C++
(remote_protocol.cpp 201 lines — a byte-exact port of a 186-byte
struct.pack header — plus remote_client.cpp 313 lines of socket framing).
That was only tractable because lm:// is trivial.

Mooncake is C++ at the core. mooncake-store/include/client_service.h
exposes mooncake::Client with Create() / Get / BatchGet / Put /
BatchPut / IsExist / Query / Remove / RegisterLocalMemory /
MountSegment. The mooncake.store.MooncakeDistributedStore that vLLM imports
is a pybind wrapper (pyclient.h) over exactly that class. The Transfer Engine
additionally ships a pure C ABI (transfer_engine_c.h).

So we would link the same native code vLLM calls, minus pybind, minus the
GIL, minus the PYTHONHASHSEED=0 requirement their docs impose (our block hashes
are deterministic by construction since KV-OFFLOAD W1). This is one of very few
upstream features where being C++ is a structural advantage rather than a tax.

Seam cost: zero

include/vllm/v1/kv_offload/kv_connector.h (landed KV-CONNECTORS W5) already
carries everything the store connector uses: the std::nullopt third state that
get_num_new_matched_tokens returns while an async lookup is in flight, the
load_async flag, build_connector_meta, request_finished delay-free
ownership, SupportsHMA multi-group finish, REGISTER_KV_CONNECTOR_WITH_WORKER
device predicates, and --kv-transfer-config JSON selection. The header's own
note — that LMCache W3 was "implement the abstract KVConnector with the landed
client, no further seam change" — applies verbatim here.

Real work is a kv_offload/mooncake/ subtree mirroring store/: key builder
(store/data.py:137-163), scheduler half (store/scheduler.py), worker half
(store/worker.py — buffer registration + stride-based layout detection + batch
get/put). Roughly 1.5–2k lines of C++, almost none of it transport.

What is and is not gateable here

Gateable on one box, no RDMA: mooncake_master --port 50051 plus a JSON
config with "protocol": "tcp" and "mode": "embedded" gives a single-node
store. That is the same gate shape KV-EXTERNAL-CACHE already uses for LMCache
(spawn server → store a prefix → restarted connector shortcuts prefill → loaded
bytes byte-identical → output-invariance vs connector-OFF).

Not gateable anywhere we have hardware: the actual payoff. register_kv_caches
registers the GPU KV tensors themselves and transfers run zero-copy
(GPUDirect) RDMA straight into the paged blocks. No box we have (dgx.casa,
Thor, Orin) sits on an RDMA fabric, so the speed claim cannot be established.

GB10 constraint: global_segment_size reserves host RAM, which on GB10 is
the same pool as GPU memory. The 80GB in the upstream doc would OOM-reboot a
Spark; the gate config must use a small segment.

Proposed disposition

Land the store connector against the TCP/single-node correctness gate, and keep
the RDMA/zero-copy speed axis explicitly open — recorded as an unmeasured
axis pending a fabric, never as a ceiling.

Spec: .agents/specs/mooncake-store-connector.md
Row: KV-MOONCAKE-STORE (engine-matrix), portfolio block ROAD-V1-D4

Upstream anchors (pin 555967922)

• vllm/distributed/kv_transfer/kv_connector/factory.py:222-227 — both registrations
• vllm/distributed/kv_transfer/kv_connector/v1/mooncake/store/connector.py
• .../store/scheduler.py:71-140 — lookup + nullopt third state
• .../store/data.py:137-163 — PoolKey format
• .../store/worker.py:1017-1037 — store.setup(); :1240-1300 — register_buffer + layout detection
• .../store/worker.py:698,911 — batch_put_from_multi_buffers / batch_get_into_multi_buffers
• docs/features/mooncake_store_connector_usage.md
• Mooncake mooncake-store/include/client_service.h, mooncake-transfer-engine/include/transfer_engine_c.h