CUDA BF16 MoE resident cache reuses stale pointers after engine teardown

[richiejp]

Problem

Repeated LoadedEngine construction and destruction in one CUDA process can reuse a MoeBlockWeights address while MoeBf16ResidentFor retains the previous engine's device-pointer array in a process-lifetime static map. A later engine can therefore execute the BF16 fast MoE path with stale resident pointers.

Reproduction

On current main plus issue #206's exact pretokenized benchmark fixture, run the focused tests in this order:

nix develop .#cuda -c ctest --test-dir build-nix-cuda \
  -R '^(test_engine_core_proc|test_async_llm|test_bench|test_gdn_decode_fused|test_gdn_prefill_conv|test_ops_gdn)$' \
  --output-on-failure

The ordered suite intermittently fails test_bench: later synthetic Qwen3.5 engines emit corrupted or zero output-token IDs. The target case alone passed 50/50. Setting VT_MOE_BF16_FAST=0 made the full test_bench suite pass 50/50, isolating the fault to the fast resident-weight path.

Expected

Resident CUDA MoE state must be owned by, or invalidated with, the engine/weights lifetime. Recreating an engine in the same process must never reuse stale device pointers, and the default fast path must remain enabled.

Scope

• Fix cache ownership/lifetime at the cause.
• Add a deterministic repeated-engine regression test plus a destructive mutation check.
• Preserve single-engine output and performance behavior.
• Discovered while rebasing RTX 5070 Ti: close Qwen3.5-4B TTFT, TPOT, and VRAM gaps vs vLLM #206; its PR should reference this issue.

[localai-bot]

Landed in #276 (3926a3a8). Summarising here because the issue auto-closed on merge, and one part of what you asked for is not verified.

Fixed at the cause, and it was six caches, not one — MoeResidentFor, MoeBf16ResidentFor, MoeMarlinResidentFor, MarlinDenseResidentFor, MarlinDensePairResidentFor (all qwen3_5.cpp) and LagunaMoeMarlinResidentFor (laguna.cpp). All static, all keyed on a raw weights address, and erase/clear appeared zero times in the repo. You found the bf16 one because VT_MOE_BF16_FAST=0 was the A/B lever available; the Marlin ones have no such switch and sit on the default quantized path.

Residency is now a member of the weights it describes (ResidentSlot), so it cannot outlive them and an address cannot be inherited.

Your diagnosis was right about why it corrupts instead of crashing, and it is worth writing down: nothing in src/ calls cudaDeviceReset or destroys the CUDA context, so the leaked resident arrays stay mapped after an engine dies — they just hold d_dev addresses that were freed with the old weights.

What is verified: the ownership invariant, deterministically (test_moe_resident_lifetime forces address reuse with placement-new; a mutation restoring the old design reds exactly that case), plus full CPU ctest 367/367 and no e2e regression on GB10.

What is NOT verified: that your symptom is gone. A same-tree A/B on GB10 with the real Qwen3-Coder-30B-A3B, three engines per arm, gave prefix=PASS and fixed=PASS — the pre-fix arm should have failed. The defect only fires when the allocator hands the second engine an address the first freed, and on that box it never did.

Two dead ends so nobody repeats them: the ordered ctest selection in your report gives a meaningless 10/10 on pre-fix code, because without #206's fixture test_bench builds an 8-prompt toy engine that never reaches MoE — and test_gdn_decode_fused has never existed in this repo (git log -S finds nothing), so that line was always running 5 tests, not 6.

Could you run test_moe_two_engines and your original repro against current main on the 5070 Ti? You have the hardware and the fixture where it actually reproduces. If it still misbehaves, please reopen — the close is a side effect of a Closes #237 line in the PR body, not a claim that your symptom was reproduced and fixed.