test_minimax_h3 is RED on dgx (GB10): cudaFree invalid argument + SIGSEGV when two CUDA cases run in one process

[localai-bot]

Found during LTX-2.5 phase L7 (#435) while running the mandated baselines on dgx.casa. Proven not to be L7's change, but it is a real red gate on a shipped model that the recorded baseline does not match.

Symptom

On dgx (GB10, CUDA build):

[doctest] test cases:  38 |  36 passed | 2 failed | 41 skipped
[doctest] assertions: 42724
[doctest] Status: FAILURE!

• minimax_h3: the WHOLE t2va path composes end to end throws vt cuda: cudaFree: invalid argument
• minimax_h3: an NVFP4 checkpoint loads into a runnable DiT then SIGSEGVs

Each case passes ALONE (873 and 585 assertions, Status: SUCCESS!). Running just those two together crashes the second. So it is cross-test CUDA state inside one process, not a defect in either case.

Why it is not the LTX-2.5 work

An A/B in the same tree: src/vllm/multimodal/ltx2_video.cpp was removed from the library — the only mechanism the LTX change has to reach H3 is its static registrar — then relinked and re-run under the GPU lock. Identical failure: same two cases, same cudaFree: invalid argument, same SIGSEGV, same 38 | 36 passed | 2 failed | 41 skipped, same 42724 assertions. Tree restored byte-for-byte afterwards, md5 verified.

The part that is NOT yet established

The A/B rules out L7. It does not establish that the failure predates the whole LTX-2.5 campaign — nobody has built the branch base or current main on dgx and run this suite. That is the next step, and this issue should not be closed as "pre-existing" until someone does it. Flagged rather than assumed.

Why it matters beyond one suite

test_minimax_h3 is the gate for a shipped model, and its recorded baseline is 79 cases / 57395 assertions. On a CPU box that number reproduces exactly. On dgx the same suite reports 38 cases / 42724 assertions with 41 skipped and two hard failures — so the pinned baseline is a CPU-box number, and the CUDA arm of that gate has been running differently without anyone recording it.

Note the assertion count drops (57395 → 42724) precisely because cases throw and skip. That is the documented doctest trap: a count alone reads like a smaller run rather than a failure. Status: is the signal.

Wants

1. Build the LTX-2.5 branch base (and main) on dgx and run test_minimax_h3 to establish how far back this goes.
2. Diagnose the cross-test CUDA state — a context or allocator torn down by one case and reused by the next is the obvious hypothesis, unverified.
3. Decide what the H3 gate's CUDA baseline actually is, and record it, so "79/57395" stops implying a number that only holds on CPU.

GPU work on that host must go through flock $HOME/gpu.lock (other sessions share the box).

🤖 Generated with Claude Code

[localai-bot]

This is plausibly #516, and there is now a 2-second reproducer to test that with

Raised by the LTX-2.5 GB10 cross-test diagnosis (#435). Not measured against this issue — flagged, not claimed.

Why the signatures line up

This issue records: test_minimax_h3 SIGSEGVs on GB10 when two CUDA cases share a process, each passes alone, and compute-sanitizer is clean.

#516 is a shared, device-blind vllm::Pool() free list keyed by size class with no device in the key. It was measured three separate times during the LTX diagnosis producing exactly that signature: a cudaMalloc block handed to a CPU-backend consumer, which dereferences it on the host. compute-sanitizer reports clean because the fault is host-side — a tool aimed at the GPU structurally cannot see it. That is the same "clean sanitizer + SIGSEGV" pairing recorded here.

The mechanism also explains "each passes alone": a block can only be traded between devices if two cases with different backends run in one process.

Why it plausibly reaches far past both suites

20+ test binaries mix a CUDA and a CPU backend in the same process — test_minimax_h3, test_qwen3_forward, test_gemma3_forward, test_deepseek_v2_forward, test_llama_forward, test_kimi_linear_fold_gate, test_capi, test_op_parity, among others. Any of them can trade a block across devices; whether they do depends on size classes and ordering, which is why this surfaces as sporadic and ordering-sensitive rather than as a clean regression.

The cheap way to test the hypothesis

#516 now has a reproducer needing no checkpoint, no NAS, no large weights, and two seconds:

./test_ltx2_device --order-by=rand --rand-seed=7
# exit=139, ~2s, :533 SIGSEGV

If test_minimax_h3's crash is the same fault, then the same interventions should move it, and they are cheap to try:

1. VT_POOL_BYPASS=1 (removes the free list) — if the H3 crash goes green, that is strong evidence.
2. Reordering (--order-by=name, seeded --order-by=rand) — if which cases share a process changes the outcome, the shared allocator is implicated rather than either case.
3. Check whether the two H3 cases that crash together use different backends.

Also worth noting: #516 has a second, silent direction — CPU block to CUDA consumer produces an all-NaN output rather than a crash (1024/1024 uniform quiet-NaN, measured). If this issue's root cause is #516, then the same binaries could be producing wrong numbers in orderings that do not crash, which would not show up as a red gate at all.

What this does not claim

Nobody has run test_minimax_h3 under VT_POOL_BYPASS=1, and this issue's own open question — whether the failure predates the LTX-2.5 campaign — is still unanswered; that needs the branch base built on dgx. Both remain owed. The point of this comment is that a plausible shared root cause now exists with a two-second test, so the next person here should try that before diagnosing from scratch.

🤖 Generated with Claude Code