vulkan: disable async transfer queue on amdvlk (mitigate MoE partial-offload crash) - #25196
Conversation
…offload crash) Mitigates the intermittent long-context crash on amdvlk with MoE + partial offload (ggml-org#25195): streamed expert-weight re-uploads via set_tensor_2d_async on the transfer queue race across submits (same-queue cross-submit write-after-write on a reused device buffer). amdvlk corrupts on the race; RADV tolerates it and keeps the async queue. Mitigation, not root fix; follows the existing "disable async transfer queue on AMD GCN" precedent from ggml-org#19976. Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com> Assisted-by: GitHub Copilot
|
I wasn't aware that amdvlk runs on Windows, how does that work? What would be needed to resolve the race properly? |
AMDVLK is cross-platform: in addition to Linux, it also provides a Vulkan driver for Windows; the binary is amdvlk64.dll (not exclusive to Linux). |
To be clear, this PR only sidesteps the hazard: on amdvlk it stops using the separate async transfer queue and falls back to the synchronous compute-queue path, so the underlying cross-submit write-after-write on the reused MoE-weight upload buffer is still latent (RADV just tolerates it). |
I think that dll is just the proprietary driver. |
Yes, on Windows, they are proprietary drivers. PR attempted to add a guard targeting both proprietary and open-source drivers. |
Overview
Disables the async transfer queue on the amdvlk driver family to mitigate an intermittent long-context crash with MoE + partial offload (#25195). Under partial offload, streamed MoE expert-weight re-uploads race across submits on the async transfer queue (
prefers_transfer_queue, added in #19976 for AMD partial-offload perf) — amdvlk corrupts on the race, RADV tolerates it. Disabling the queue drives the validated sync-validation racing-write count to 0.const bool prefers_transfer_queue = device->vendor_id == VK_VENDOR_ID_AMD && device->architecture != AMD_GCN && + device->driver_id != vk::DriverId::eAmdProprietary && + device->driver_id != vk::DriverId::eAmdOpenSource && !device->uma && !allow_graphics_queue;This is a mitigation, not the root fix — the race is latent (tolerated on RADV); a proper submit-level synchronization fix is the Vulkan backend maintainer's domain. Perf on amdvlk is within noise (near-free); RADV keeps the async queue and is unaffected. Full diagnosis in #25195.
Requirements