fix(moe): the weights own their resident device state, not a static address map (#237)

[localai-bot]

Closes #237 pending reporter confirmation — please read the evidence split before merging.

@richiejp found a second engine in one process emitting corrupted and zeroed output token ids, isolated it with VT_MOE_BF16_FAST=0, and named the cause. Confirmed — and it is six caches, not one:

File:line	Key	Path
qwen3_5.cpp:539	MoeBlockWeights*	fp4 fused MoE
qwen3_5.cpp:562	MoeBlockWeights*	bf16 fast MoE ← reported
qwen3_5.cpp:654	MoeBlockWeights*	Marlin grouped MoE
qwen3_5.cpp:2271	Nvfp4Weight*	Marlin dense
qwen3_5.cpp:2439	Nvfp4Weight*	Marlin dense pair
laguna.cpp:469	LagunaMoeWeights*	Laguna NVFP4 MoE

All static, all keyed on a raw address, and erase/clear appears zero times in the repo. The bf16 one surfaced because VT_MOE_BF16_FAST was the A/B lever available; the Marlin ones have no such switch and sit on the default quantized path.

Why it corrupts rather than crashes: nothing in src/ calls cudaDeviceReset or destroys the CUDA context, so the leaked resident arrays stay mapped after an engine dies. They hold the previous engine's d_dev addresses, which were freed with the weights — so the next engine allocates over them and the fast path reads its own activations as expert pointers.

The fix

Structural, as the issue asks. Residency is now a ResidentSlot member of the weights it describes, so it cannot outlive them and an address cannot be inherited. The maps are gone. The slot is deliberately opaque (shared_ptr<void>) — the resident types are CUDA-path details of the model .cpp files and do not belong in a public header — and it still runs the right destructor. The per-call lock the map accessors held is kept rather than narrowed, so a regression cannot be ambiguous between lifetime and synchronisation.

Not changed: the device allocations those types point at are still leaked for the process, exactly as before. Freeing them is a question about backend teardown ordering, and putting a shutdown hazard on the critical path of a correctness fix is the wrong trade.

Evidence, including what it does not show

Deterministic (CPU): test_moe_resident_lifetime, 6 cases / 19 assertions — including placement-new reconstruction of a block in the exact storage a built-up block just vacated, which is the aliasing the old map could not distinguish. Mutation check: restoring the old ownership reds that case and only that case, 5/6. Full CPU ctest 367/367.

On real hardware (GB10), and here is the honest part — the pre-fix arm PASSES. Same-tree A/B, identical flags, only qwen3_5.cpp/laguna.cpp swapped, both arms tree-checked by grepping the built source:

arm=prefix  address-keyed map PRESENT  PASS  (8 assertions)
arm=fixed   address-keyed map ABSENT   PASS  (8 assertions)

Both loaded the real Qwen3-Coder-30B-A3B, built three engines each in one process, and decoded identically with non-degenerate tokens. So this is a no-regression gate, not a red/green proof.

The defect only fires when the allocator hands the second engine an address the first freed, and on this box it never did. That is inherent — your repro is intermittent, needs a specific ctest ordering plus #206's fixture, and reproduces on a 5070 Ti, not on GB10. A test cannot command address reuse.

Two earlier gating attempts are worth recording so nobody repeats them: the ordered ctest selection from the issue returns a meaningless 10/10 on the pre-fix tree, because without #206's fixture test_bench builds an 8-prompt toy engine that never reaches MoE — and test_gdn_decode_fused has never existed in this repo (git log -S finds nothing), so that line was always running 5 tests, not 6.

What I am asking of you

@richiejp — could you run test_moe_two_engines (and your original repro) against this branch on the 5070 Ti? You have the hardware and the fixture where the corruption actually reproduces; I can show the invariant holds and that nothing regressed, but not that your symptom is gone.