POOL-DEVICE-KEY: one scratch pool per DEVICE, rebased off a stale base, with the dgx pair (#516, #486, #614)

.agents/porting-inventory.md

@@ -1482,12 +1482,17 @@ Examples: `examples/cli` ✅ (C-API client), `examples/server` ✅ (OpenAI serve
       CPU-backend device forward AFTER a bf16 CUDA one; at f32 the two arms land
       in different size classes and never trade blocks. The pool already carries
       exactly this invariant for STREAMS — `AuxPool()` exists because "two streams
-      sharing one pool BREAKS" its reuse ordering — and the fix used here is that
-      same sanctioned seam: the CPU arm runs under an `ActivePoolScope` with its
-      own pool. **The DEVICE half of the invariant is still unstated at the pool
-      itself, and a size-keyed device-blind free list in a multi-device process is
-      a trap for the next caller. Repairing it is a shared-hot-path change and is
-      owed as its own row, not this one.**
+      sharing one pool BREAKS" its reuse ordering — and the first fix used that
+      same sanctioned seam: the CPU arm ran under an `ActivePoolScope` with its
+      own pool. **That workaround is GONE, and so is the fault it worked around.**
+      `POOL-DEVICE-KEY` ([#516](https://github.com/mudler/vllm.cpp/issues/516),
+      [`specs/pool-device-key.md`](specs/pool-device-key.md)) states the DEVICE
+      half at the pool itself: a `DevicePool` is bound to one backend, `Pool(b)`
+      is the only spelling and there is no device-less one, every operation
+      throws on a foreign backend, and the per-caller scope in
+      `test_ltx2_device.cpp` was DELETED in the same change — because a list of
+      remembered callers is what this fault was, and leaving one behind would
+      have disarmed the only test that exposes the silent-NaN direction.
     * **OWED, and precisely:** (a) the prompt-K/V cache on the device path, which
       is REFUSED by name rather than ignored; (b) an FP4-RESIDENT arm — the
       `LinearDev` seam is one parameter away from the shared Marlin W4A16

.agents/roadmap_v1.md

@@ -117,6 +117,7 @@ issue is not yet placed. Keyed record: update in place, never append.
 | [#501](https://github.com/mudler/vllm.cpp/issues/501) | `PERF-27B-LMHEAD-FP4` | `AlphaVecBf16TakesTwoLaunch` bounded a COUNT of ulp mismatches instead of their MAGNITUDE, and was RED on its first CUDA run at ~26% — the double-rounding population the bf16-D lever produces by construction. Replaced by a max-ulp bound (`<= 1`, and `<= 0` at a pow2 alpha), measured 0/1-ulp only over 2.17M words on GB10, spec [`perf-fp8-alpha-fold.md`](specs/perf-fp8-alpha-fold.md) §The bf16-vs-f32 divergence is DOUBLE ROUNDING | bug |
 | [#521](https://github.com/mudler/vllm.cpp/issues/521) | `PERF-27B-LMHEAD-FP4` | [`perf-fp8-alpha-fold.md`](specs/perf-fp8-alpha-fold.md) `:19`/`:211` claim the bf16-D lever "also applies to 35B-A3B" — it is INERT there: `GdnOutDType(dense_model=false)` is F32 on a MoE, contradicting the code's own comment at `qwen3_5.cpp:3617-3619` | bug |
 | [#391](https://github.com/mudler/vllm.cpp/issues/391) | `PERF-CPU-DECODE-BARRIER` | CPU backend: batch-1 decode is barrier-bound (47% sync), and paged attention branches per element | perf |
+| [#516](https://github.com/mudler/vllm.cpp/issues/516) | `POOL-DEVICE-KEY` | `vllm::Pool()`'s free list is keyed by size class with NO DEVICE in the key, so a `cudaMalloc` block reaches a CPU `DBuf` (SIGSEGV) and a host block reaches a CUDA forward (uniform `0x7fff0000` NaN); spec [`pool-device-key.md`](specs/pool-device-key.md), lands through `row/MODEL-DIFFUSION-LTX25` | bug |
 | [#299](https://github.com/mudler/vllm.cpp/issues/299) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE`: 5 dense SwiGLU models bypass the MUST-route merged-GEMM seam with no stated blocker (spec [`fusion-dense-migrate.md`](specs/fusion-dense-migrate.md)) | bug |
 | [#314](https://github.com/mudler/vllm.cpp/issues/314) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE` glue half: `glm4`/`phi3` still hand-call add+RMSNorm instead of `vt::FusedChain` (split out of #299, which closed the merged-GEMM half only) | bug |
 | [#337](https://github.com/mudler/vllm.cpp/issues/337) | `ROAD-V1-C1` | `FUSION-DENSE-MIGRATE`: the five dgx SACRED paged-engine gates are OWED after the merged-GEMM fold (`test_{commandr,glm4,minicpm,minicpm3,phi3}_paged_engine` SKIP on a CPU box) | bug |

docs/FEATURES.md

@@ -58,6 +58,7 @@ are our reading of their documented behavior, not measurements.
 | KV events (block create / evict publish) | ◐ no transport | ✅ | ☐ | ☐ |
 | Prefix-cache matching unit | ◐ resolver only | ✅ | ☐ | ☐ |
 | Compute directly on quantized blocks | ✅ | ☐ | ☐ | ✅ |
+| Scratch allocator keyed by device (two backends, one process) | ✅ since [#516](https://github.com/mudler/vllm.cpp/issues/516); a pool is bound to one backend and refuses any other, and a backend with no registered platform is refused rather than given another's residency cap | ✅ device is field 0 of the allocation handle | ✅ | ✅ |
 | Automatic memory sizing (no hand-tuned budget) | ☐ hand-typed block count | ☐ percent, hand-tuned | ☐ | ◐ |
 | Memory cap with a pre-flight error instead of an OOM | ☐ | ◐ KV pool only | ◐ | ☐ |
 

docs/STATUS.md

@@ -1414,6 +1414,17 @@ platform missing from `CurrentPlatform()`'s hardcoded walk registers and answers
 correctly but is NEVER selected, with no compiler diagnostic. `test_platform`
 now gates that every `DeviceType` is in the walk and CPU is last.
 
+**The device-scratch pool is now ONE POOL PER DEVICE (`POOL-DEVICE-KEY`, #516).**
+It was a process-wide free list keyed by byte size class with no device in the
+key, so in a mixed-backend process a block allocated through one backend was
+handed to the next caller of that class on another: a `cudaMalloc` block reaching
+a CPU forward SIGSEGVs host-side with `compute-sanitizer` clean, and a host block
+reaching a CUDA forward returned a uniform `0x7fff0000` quiet NaN. A pool is now
+bound to a backend, `Pool(b)` is the only spelling, every operation refuses a
+foreign backend, and the two per-caller workarounds are deleted. `test_device_pool`
+gates it without a GPU; `VT_POOL_BYPASS`/`VT_POOL_EXACT` keep their meanings and
+the suite is green under both.
+
 **CUDA architectures.** The runtime-gated production arch is GB10 `sm_121a`
 (every gate model, every benchmark). A build-supported cross-family fan-out
 (`sm_80/86/87/89`, `sm_90a`, `sm_100a/103a`, `sm_110`) compiles single-arch,

docs/USAGE.md

@@ -128,6 +128,32 @@ context is never torn down, so the pointers stayed mapped — it simply produced
 corrupted or zeroed output tokens, intermittently
 ([#237](https://github.com/mudler/vllm.cpp/issues/237)).
 
+More than one **backend** in one process is likewise supported — a CPU forward
+running beside a CUDA one, which is what a diffusion pipeline with a host-side
+stage does. Until
+[#516](https://github.com/mudler/vllm.cpp/issues/516) it was not: the shared
+device-scratch pool was a single process-wide free list keyed by byte size class
+with no device in the key, so a block allocated through one backend was handed
+to the next caller of that size class on another. It has two symptoms and the
+direction picks which: a `cudaMalloc` block reaching a CPU forward segfaults in
+the host `memcpy`, and a host block reaching a CUDA forward produces output that
+is uniformly NaN rather than wrong. Neither can happen now — a scratch pool is
+bound to one backend and refuses any other with a `std::logic_error` naming both
+— and no user-facing flag or env var selects the behaviour: it is unconditional.
+
+One consequence is worth knowing before you add a backend. The scratch pool's
+residency cap now comes from *that device's* platform rather than from whichever
+device resolved first, so constructing a buffer on a backend whose platform was
+never registered raises instead of silently inheriting another platform's cap. A
+cap read off the wrong platform is a wrong number, not a default, and every
+backend the tree ships registers one.
+
+`VT_POOL_BYPASS=1` and `VT_POOL_EXACT=1` keep exactly the meanings
+[ENVIRONMENT.md](ENVIRONMENT.md) records for them. They are debugging lanes, not
+timing configurations, and the pool's own test suite is green under both, so
+either one stays usable as a discriminator when something else is under
+suspicion.
+
 ## Starting an agent-assisted contribution
 
 Run `scripts/agent-start.py` first. It reports an inherited worktree role or,

include/vllm/model_executor/models/dense_device_glue.h

@@ -17,6 +17,8 @@
 //   DBuf                    — move-only pooled device allocation + tensor view.
 #pragma once
 
+#include <array>
+#include <atomic>
 #include <cstdint>
 #include <memory>
 #include <vector>
@@ -25,6 +27,8 @@
 #include "vllm/model_executor/models/qwen3_5_weights.h"   // OwnedTensor
 #include "vllm/platforms/interface.h"
 #include "vt/backend.h"
+#include "vt/device.h"  // kNumDeviceTypes
+#include "vt/dtype.h"   // VT_CHECK
 #include "vt/ops.h"
 
 namespace vllm {
@@ -62,19 +66,40 @@ inline Tensor Reshape(const Tensor& src, const std::vector<int64_t>& shape) {
 
 // The device-scratch residency policy (BACKEND-PLATFORM item 2), resolved from
 // the running device's platform. The DevicePool soft cap is platform data (0 ==
-// uncapped, GB10 today ⇒ pool behavior byte-for-byte unchanged). Memoized in a
-// function-local static: DBuf is a per-op hot path and the process runs on ONE
-// device, so the virtual dispatch is paid exactly once. Mirrors qwen3_5.cpp.
+// uncapped, GB10 today ⇒ pool behavior byte-for-byte unchanged). Mirrors
+// qwen3_5.cpp.
+//
+// Memoized PER DEVICE TYPE, not once per process. The previous single
+// function-local static cached whichever device asked FIRST and then applied its
+// cap to every later device — the same ambient-device assumption #516 fixed one
+// layer down, and a mixed-backend process would have run a CUDA DBuf under the
+// CPU platform's policy. DBuf is a per-op hot path, so the virtual dispatch is
+// still paid at most once per device type.
+//
+// A backend whose platform was never REGISTERED now throws out of
+// `platforms::GetPlatform` instead of inheriting whichever device asked first.
+// That is the point: a residency cap read off another platform is a wrong
+// number wearing a default's clothes. Gated by
+// tests/vllm/models/test_device_pool.cpp.
 struct DevicePoolPolicy {
   size_t cap_bytes = 0;  // residency_policy().device_pool_cap_bytes (0 == uncapped)
 };
 inline DevicePoolPolicy ResolveDevicePoolPolicy(const Dev& d) {
-  static const DevicePoolPolicy p = [&] {
-    const auto rp =
-        vllm::platforms::GetPlatform(d.q.device.type).residency_policy();
-    return DevicePoolPolicy{rp.device_pool_cap_bytes};
-  }();
-  return p;
+  // Stored as cap+1 so that 0 means "not resolved yet" and a genuine cap of 0
+  // (every platform today) still caches. Racing threads resolve the same device
+  // type to the same value, so the benign double-resolve needs no lock.
+  static std::array<std::atomic<size_t>, vt::kNumDeviceTypes> cached{};
+  // Same bound, same place, as platforms::Index() (src/vllm/platforms/
+  // platform.cpp) applies to this identical value before indexing ITS registry.
+  // An out-of-range DeviceType is only reachable by a cast, and the two lookups
+  // must not disagree about whether that is a throw or a stray write.
+  const size_t idx = static_cast<size_t>(d.q.device.type);
+  VT_CHECK(idx < vt::kNumDeviceTypes, "invalid device type");
+  const size_t seen = cached[idx].load(std::memory_order_relaxed);
+  if (seen != 0) return DevicePoolPolicy{seen - 1};
+  const auto rp = vllm::platforms::GetPlatform(d.q.device.type).residency_policy();
+  cached[idx].store(rp.device_pool_cap_bytes + 1, std::memory_order_relaxed);
+  return DevicePoolPolicy{rp.device_pool_cap_bytes};
 }
 
 // Owned device allocation + tensor view, routed through the SHARED DevicePool so
@@ -91,7 +116,11 @@ class DBuf {
     bytes_ = static_cast<size_t>(numel) * vt::SizeOf(dt);
     alloc_bytes_ = bytes_ == 0 ? 1 : bytes_;
     cap_ = ResolveDevicePoolPolicy(d).cap_bytes;
-    pool_ = ActivePool();
+    // THIS DEVICE's pool, unless an ActivePoolScope overrides it (the aux
+    // stream). Remembered so the block returns to the pool it came from even if
+    // this DBuf outlives the scope. See device_pool.h: there is no
+    // device-less pool to fall back on.
+    pool_ = &ActivePool(*b_);
     p_ = pool_->Get(*b_, alloc_bytes_);
     t_ = MakeTensor(p_, dt, d.q.device, shape);
     if (host != nullptr && bytes_ > 0) b_->Copy(d.q, p_, host, bytes_);
@@ -130,16 +159,44 @@ class DBuf {
     b_->Synchronize(d.q);
   }
   // Relinquish the pool block WITHOUT returning it (dtor becomes a no-op); the
-  // caller takes over the Pool().Put obligation for alloc_bytes().
+  // caller takes over the Put obligation for alloc_bytes(). Prefer
+  // ReleaseShared() below, which discharges that obligation correctly by
+  // construction.
   void* Release() {
     void* p = p_;
     p_ = nullptr;
     return p;
   }
 
+  // Move the block into a shared_ptr that returns it to THIS buffer's own pool
+  // and backend when the last owner drops it — the carrier every cross-step
+  // hand-off (device logits, MTP hidden states, MoE scratch) wants.
+  //
+  // It replaces ~28 copies of a hand-written deleter that closed over the byte
+  // count ALONE and called `Pool().Put(alloc, q)`. That idiom named neither the
+  // device nor the pool, so it returned every such block to the one global pool
+  // — a block from another device (#516), which was LIVE, and a block drawn
+  // from the aux-stream pool, which was not: none of the nine `Release()` sites
+  // sat inside or under any of the four `ActivePoolScope` regions, so the old
+  // deleter and the buffer's own `pool_` always agreed in practice. It was a
+  // hazard one new call site away from being real, and it is gone either way,
+  // because the carrier now captures the pool it came from rather than
+  // re-deriving it.
+  std::shared_ptr<void> ReleaseShared() {
+    DevicePool* const pool = pool_;
+    Backend* const b = b_;
+    const size_t alloc = alloc_bytes_;
+    void* const p = Release();
+    // A moved-from or already-released buffer owns nothing; a shared_ptr built
+    // over a null pointer with a custom deleter would still RUN that deleter and
+    // push null into the free list.
+    if (p == nullptr) return {};
+    return std::shared_ptr<void>(p, [pool, b, alloc](void* q) { pool->Put(*b, alloc, q); });
+  }
+
  private:
   Backend* b_;
-  DevicePool* pool_ = &Pool();
+  DevicePool* pool_ = nullptr;
   void* p_ = nullptr;
   size_t bytes_ = 0;
   size_t alloc_bytes_ = 0;