vllm::Pool() free list is keyed by size class with no DEVICE in the key: a cudaMalloc'd block can be handed to a CPU DBuf

[localai-bot]

Found while repairing LTX-2.5 phases L7/L8 (#435, PR #437). Not an LTX defect — it is a latent fault in a shared hot path that LTX merely became the first caller to reach.

The defect

vllm::Pool() is a process-wide singleton whose free list is keyed by size class only. The device is not part of the key. So a block allocated with cudaMalloc for a CUDA arm is returned to the pool and then handed straight to a CPU-backend DBuf on the next allocation of the same size class, which copies into it with a host memcpy.

Symptom

test_ltx2_device SIGSEGV on GB10:

__memcpy_sve  <-  UploadStream  <-  PrepareStreamDev

8 cases run, 1 failed, 5 never reached — this project's "changed COUNT = RED" signature.

compute-sanitizer reports 0 device errors, which is the tell: the fault is host-side, dereferencing a device pointer from the CPU. A tool aimed at the GPU cannot see it.

Why it stayed hidden

It requires a bf16 CPU-backend device forward to run after a bf16 CUDA one in the same process. No test had ever done that. At f32 the two arms land in different size classes and never trade blocks, so the f32 gates could not reach it either.

That is spec §7.0(c) again (): a fixture that cannot enter the regime that discriminates. The bug is old; only the ordering is new.

Immediate mitigation (in the LTX repair, not a fix for this issue)

The CPU arm now runs under its own DevicePool via ActivePoolScope. That seam already exists for exactly this shape — AuxPool() is documented as existing because "two streams sharing one pool BREAKS" its reuse ordering. test_ltx2_device is back to 13/498 SUCCESS on the CPU backend.

That scopes one caller away from the hazard. It does not fix the pool.

What this issue wants

Put the device in the free-list key, so a block allocated on one device can never be handed to another. It is a shared hot path, so:

• it needs its own row, spec and red-first test;
• the test must be the ordering that exposes it (same size class, CUDA allocation freed, then a host-backend allocation), because nothing currently in the tree produces it;
• worth auditing whether any other caller mixes backends within a size class today, since the failure mode is a host dereference of a device pointer and would present as an unrelated segfault.

Blast radius, honestly stated

Unknown and worth establishing. Any code path that allocates on CUDA, frees, then allocates the same size class on a host backend in the same process is exposed. It has been latent long enough that nothing in the current suite triggers it, so "no failures today" is not evidence of safety — it is evidence that no test has produced the ordering.

Related: #449 (an instrument that could not see the defect class it existed to catch), .agents/specs/ltx-2-5.md §7.0.

🤖 Generated with Claude Code

[localai-bot]

New evidence: the fault is BIDIRECTIONAL, and the other direction is silent

Diagnosed while closing the LTX-2.5 campaign (#435, PR #437). This issue's mechanism is exactly right; what it got wrong is believing the ActivePoolScope mitigation contained it.

The second direction

direction	consumer	symptom
CUDA cudaMalloc block → CPU-backend forward	any CPU case	SIGSEGV, compute-sanitizer-invisible (this issue's recorded signature)
CPU aligned_alloc block → CUDA forward	the shipped 21B DiT	silent non-finite output

The silent direction is the worse of the two: a completed 21.00B forward, correct output sizes (1024 video / 512 audio), 0.252 s, and non-finite. No crash, no OOM, no refusal — a confident wrong answer.

Attribution, by measurement

arm	change	result
baseline (shared pool)	none	FAILURE 12/13, 4639, test_ltx2_device.cpp:925 REQUIRE(std::isfinite(v)) → false
VT_POOL_BYPASS=1	free list removed	SUCCESS 13/13, 6176
per-case DevicePool	shipped case scoped, nothing else	SUCCESS 13/13, 6176
VT_POOL_EXACT=1	reuse kept, size-class rounding removed	FAILURE, identical to baseline

6176 = 552 + 5624 exactly, so both green arms completed every case, printed the same config source: NONE (manifest shapes only) and the same correct absmax video=0.300781 audio=2.14062 as the solo control.

Size-class rounding is NOT the mechanism — VT_POOL_EXACT=1 stays red. It is the device, as this issue says.

Two alternatives ruled out, not assumed away

Not an order-dependent forward. Under --order-by=name the shipped case runs third instead of twelfth and is byte-identical to solo. And changing only the allocator makes it green with identical inputs, weights and config — an allocator change cannot fix an order-dependent forward.

Not a read-before-write of scratch. Under one ordering the shipped case did receive reused blocks carrying stale content (two CUDA cases had freed reduced-scale scratch into the shared pool first) and was byte-correct. Stale content is harmless; the block's device is fatal. Independently: a read-before-write produces wrong numbers, never a SIGSEGV, and the CUDA→CPU direction segfaults.

Minimal reproducer — 2 seconds, no checkpoint needed

./test_ltx2_device --rand-seed=7
# orders: DEVICE-RESIDENT…on CUDA → bf16 PRODUCTION…on CUDA → bf16 PRODUCTION…(CPU backend)
# third case SIGSEGVs at test_ltx2_device.cpp:533, exit 139, 44 assertions, ~2 s

The 21 GB checkpoint is never touched. The silent direction needs it:

LTX2_SHIPPED_DIT=<ckpt> ./test_ltx2_device \
  -tc='*DEVICE-RESIDENT forward matches upstream (CPU backend)*,*SHIPPED 21.00B*'

Two cases: one CPU-backend forward, then the 21B CUDA forward. Deterministic across repeats.

Where the existing mitigation stands

ActivePoolScope works precisely where it is applied — the F9 case puts nothing into the shared pool. The red comes from its unscoped siblings. Scoping each caller one at a time is not a fix; it is a list of places someone remembered.

Stated uncertainty

The proximate reason a host block yields NaN on GB10 rather than merely running slowly through ATS is not established. Candidates: aligned_alloc's 64-byte alignment against cudaMalloc's 256, or kernels using device-memory-only access paths. This does not move the attribution — three independent arms agree — but it belongs in this row's diagnosis.

A deliberate non-action

The obvious way to make the LTX-2.5 campaign green is to scope the shipped case away. That was declined, because that case is currently the only test in the tree exposing the silent-wrong-answer direction. Making the campaign green by deleting the evidence would be the wrong trade.

🤖 Generated with Claude Code

[localai-bot]

Full diagnosis: every ordering is red, the whole output is NaN, and there is now a 2-second red-first test

Completed under two flock $HOME/gpu.lock holds (waits of 3724 s and 327 s; no holder was ever killed). Diagnosis only, no tracked file changed.

The red-first test this issue said it had no way to write

No checkpoint, no NAS, no 42 GB, two seconds:

./test_ltx2_device --order-by=rand --rand-seed=7
# exit=139  wall=2s
# 4 cases | 3 passed | 1 failed | 9 skipped
# 44 assertions | 44 passed | 0 failed
# :533 FATAL ERROR: test case CRASHED: SIGSEGV

Seed 7 orders DEVICE-RESIDENT…on CUDA -> bf16 PRODUCTION…on CUDA -> bf16 PRODUCTION…(CPU backend); the third crashes. Deterministic: S1 and S2 byte-identical, and identical to a run taken 95 minutes and one reboot earlier.

Note the trap in that output: 44 assertions, 0 failed, beside a SIGSEGV. Grepping assertions: reads it as clean.

The silent direction, characterised

M2 census video: n=1024 nan=1024 inf=0 zero=0  first8: [0]=nan(0x7fff0000) … [7]=nan(0x7fff0000)
M2 census audio: n=512  nan=512  inf=0 zero=0

100% NaN, uniform bit pattern — bf16 0x7fff (canonical quiet NaN) widened by WidenBf16. Not one bad element: the entire output. Uniform quiet-NaN means a NaN was computed and propagated through the downstream GEMMs from one contaminated buffer, not that plausible garbage was read — raw host bytes reinterpreted as bf16 would be mostly finite.

Every ordering is red; the green one is an accident

ordering	symptom
default file order + checkpoint	NaN (12/13, 4639)
--order-by=name	SIGSEGV
--rand-seed=1, --rand-seed=7	SIGSEGV
default, no checkpoint	green (13/13, 552)

The only green configuration is green by declaration order, not by safety.

Attribution (four arms)

arm	change	result
VT_POOL_BYPASS=1	free list removed	SUCCESS 13/13, 6176
per-case DevicePool	victim isolated	SUCCESS 13/13, 6176
VT_POOL_EXACT=1	reuse kept, rounding removed	FAILURE, identical to baseline
bisect	prefix-3 green, prefix-4 red	minimal red prefix = case 4

6176 = 552 + 5624 exactly. Remove the free list -> green; keep it but isolate the victim -> green; keep reuse and drop only size-class rounding -> still red. Cross-device reuse, not over-allocation, not the forward.

Unexplained, and it belongs in this row

Why a host aligned_alloc block yields NaN on GB10 rather than merely running slowly through ATS is NOT established. Unified memory makes host pointers device-addressable, so the naive expectation is correct-but-slow. That is not what happens — the red forward is faster (0.230-0.252 s) than the green solo one (0.594 s), so it is not running at host bandwidth either. Untested candidates: 64-byte aligned_alloc alignment against cudaMalloc's 256 breaking a vectorised or TMA load; or kernels requiring true device memory.

Blast radius is much wider than one suite

20+ test binaries mix a CUDA and a CPU backend in one process, including test_minimax_h3, test_qwen3_forward, test_gemma3_forward, test_deepseek_v2_forward, test_llama_forward, test_kimi_linear_fold_gate, test_capi, test_op_parity. Any of them can trade a block across devices. #486 is plausibly this same bug — "SIGSEGV when two CUDA cases share a process" with compute-sanitizer clean is exactly the signature measured three times here. Not claimed, not measured; flagged for whoever takes this row.

A deliberate non-action, and its cost

A two-line ActivePoolScope around the shipped case turns the LTX-2.5 suite green. It was not applied. That case is the only test in the tree exposing the silent direction — the one returning wrong numbers instead of crashing. Scoping it away buys a green gate by deleting the detector for the failure mode we least want to ship.

ActivePoolScope works exactly where applied: in the name-ordered run the F9 case ran first, contributed nothing to the shared pool, and the shipped case three positions later was correct. The red is its unscoped siblings — which is why per-caller scoping is a list of remembered places, not a fix.

🤖 Generated with Claude Code