[None][fix] KVv2 C++ backend: nanobind None/uint64 argument handling, constraint-floor parity, backend-aware OutOfPagesError

[lancelly]

What

Three fixes for issues hit while running this branch's C++ backend end-to-end on a DeepSeek-V4 4p1d disaggregated deployment (4×TP8 ctx + 1×TP16 gen, GB300, conc=1000, agentic multi-turn workload), plus an A/B validation of the backend itself. All three are interface-layer issues — the core design ran flawlessly once they were fixed (≈106K requests, zero crashes).

1. set_base_page_index_buf rejects None → every ctx worker dies (kvCacheManagerV2.cpp)

release_index_slot passes buf=None to clear the DSA index buffer. The lambda body already handles is_none(), but the nb::arg("buf") annotation lacks .none(), so nanobind rejects the call at the argument-casting layer:

TypeError: set_base_page_index_buf(): incompatible function arguments ...
Invoked with types: ..._KVCache, int, int, NoneType

On the C++ backend every ctx worker crashes at the first index-slot release. One-line fix: nb::arg("buf").none().

2. int64 bindings reject valid ids/salts → gen workers die in CUDA-graph padding (kvCacheManagerV2.cpp)

Two producers exceed int64 range:

• CUDA_GRAPH_DUMMY_REQUEST_ID = 2**64 - 1 (cuda_graph_runner.py) — create_kv_cache(id=...) raises the same incompatible function arguments TypeError inside CUDA-graph padding, killing all gen ranks (AssertionError: Sampling failed). Only gen uses graphs, and the Python backend's unbounded ints are unaffected — so this is cpp-backend-gen-specific and 100% reproducible.
• ReuseScope.salt is derived via int.from_bytes(sha256(...)[:8]) — a full 64-bit value, i.e. >2**63 with ~50% probability per salted request. Same crash, latent until a cache_salt-carrying request arrives.

Fix: accept uint64 at every id/salt boundary (create_kv_cache, ReuseScope ctor/accessors, the 5 stats APIs, get_dirty_stats_kv_cache_ids) and reinterpret via two's complement in both directions. This is bijective (small ids unchanged), and ReuseScope::toBytes serializes the same 8 bytes either way, so reuse keys stay identical to the Python backend's.

3. except OutOfPagesError never matches on the C++ backend (py_executor.py)

The nanobind module registers its own OutOfPagesError exception class, while py_executor imports the pure-Python class from kv_cache_manager_v2._exceptions — so the guarded allocation paths crash instead of backing off. The package __init__ already re-exports the backend-appropriate class; import that instead (one line).

Also included in the first commit: the C++ StorageManager::computeMinSlotsFromConstraints port of the Python-side fix from NVIDIA#16269 — explicit initialPoolRatio disabled constraint floors entirely and floors weren't scaled by 1/maxUtilForResume (DSv4 warmup starvation when pool_ratio is set). The Python backend has this fix; the C++ translation predates it. More generally, a sweep for Python-side KVv2 fixes that postdate the translation snapshot might be worthwhile before merge.

Validation

With these fixes, TLLM_KV_CACHE_MANAGER_V2_BACKEND=cpp vs python A/B on the setup above (~100K requests per arm, arms differ only in the env var):

metric	python	cpp	Δ
req/s	36.74	38.85	+5.7%
ITL p50	14.2 ms	11.5 ms	−19%
TTFT p90	38.8 s	41.9 s	+8% (closed-loop backpressure onto the ctx-bound stage)

Gen-side nsys (steady state, iters 25000–25100, 16 ranks/arm) attributes the win to the expected places: _schedule p90 11.0→1.0 ms, update_resources p90 1.9→0.3 ms, _handle_responses p90 4.6→1.7 ms, per-event GIL-wait p90 −57%.