Skip to content

vulkan: disable async transfer queue on amdvlk (mitigate MoE partial-offload crash) - #25196

Closed
liminfei-amd wants to merge 1 commit into
ggml-org:masterfrom
liminfei-amd:amd-rocm/25195-amdvlk-disable-async-transfer
Closed

vulkan: disable async transfer queue on amdvlk (mitigate MoE partial-offload crash)#25196
liminfei-amd wants to merge 1 commit into
ggml-org:masterfrom
liminfei-amd:amd-rocm/25195-amdvlk-disable-async-transfer

Conversation

@liminfei-amd

@liminfei-amd liminfei-amd commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Overview

Disables the async transfer queue on the amdvlk driver family to mitigate an intermittent long-context crash with MoE + partial offload (#25195). Under partial offload, streamed MoE expert-weight re-uploads race across submits on the async transfer queue (prefers_transfer_queue, added in #19976 for AMD partial-offload perf) — amdvlk corrupts on the race, RADV tolerates it. Disabling the queue drives the validated sync-validation racing-write count to 0.

 const bool prefers_transfer_queue =
     device->vendor_id == VK_VENDOR_ID_AMD &&
     device->architecture != AMD_GCN &&
+    device->driver_id != vk::DriverId::eAmdProprietary &&
+    device->driver_id != vk::DriverId::eAmdOpenSource &&
     !device->uma &&
     !allow_graphics_queue;

This is a mitigation, not the root fix — the race is latent (tolerated on RADV); a proper submit-level synchronization fix is the Vulkan backend maintainer's domain. Perf on amdvlk is within noise (near-free); RADV keeps the async queue and is unaffected. Full diagnosis in #25195.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES — AI tooling assisted the investigation and drafting this description. The one-line driver-ID guard I authored/reviewed and validated on real hardware; I take responsibility for the submission.

…offload crash)

Mitigates the intermittent long-context crash on amdvlk with MoE + partial offload (ggml-org#25195): streamed expert-weight re-uploads via set_tensor_2d_async on the transfer queue race across submits (same-queue cross-submit write-after-write on a reused device buffer). amdvlk corrupts on the race; RADV tolerates it and keeps the async queue. Mitigation, not root fix; follows the existing "disable async transfer queue on AMD GCN" precedent from ggml-org#19976.

Signed-off-by: liminfei-amd <91481003+liminfei-amd@users.noreply.github.com>
Assisted-by: GitHub Copilot
@liminfei-amd
liminfei-amd requested a review from a team as a code owner July 1, 2026 08:05
@github-actions github-actions Bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Jul 1, 2026
@0cc4m

0cc4m commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

I wasn't aware that amdvlk runs on Windows, how does that work?

What would be needed to resolve the race properly?

@liminfei-amd

Copy link
Copy Markdown
Contributor Author

I wasn't aware that amdvlk runs on Windows, how does that work?

AMDVLK is cross-platform: in addition to Linux, it also provides a Vulkan driver for Windows; the binary is amdvlk64.dll (not exclusive to Linux).
The crash dump provided by the user specifically lists amdvlk64.dll (v9.2.10.353)—which is the AMD Vulkan driver on his Windows system.

@liminfei-amd

liminfei-amd commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

What would be needed to resolve the race properly?

To be clear, this PR only sidesteps the hazard: on amdvlk it stops using the separate async transfer queue and falls back to the synchronous compute-queue path, so the underlying cross-submit write-after-write on the reused MoE-weight upload buffer is still latent (RADV just tolerates it).
Do you see a clean root fix — e.g., double-buffering the streamed-weight upload target (a distinct device buffer + command buffer per in-flight pass) so a region isn't overwritten while a prior transfer submit that wrote it is still in flight? Thanks.

@0cc4m

0cc4m commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

I wasn't aware that amdvlk runs on Windows, how does that work?

AMDVLK is cross-platform: in addition to Linux, it also provides a Vulkan driver for Windows; the binary is amdvlk64.dll (not exclusive to Linux). The crash dump provided by the user specifically lists amdvlk64.dll (v9.2.10.353)—which is the AMD Vulkan driver on his Windows system.

I think that dll is just the proprietary driver.

@liminfei-amd

Copy link
Copy Markdown
Contributor Author

I think that dll is just the proprietary driver.

Yes, on Windows, they are proprietary drivers. PR attempted to add a guard targeting both proprietary and open-source drivers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning Vulkan Issues specific to the Vulkan backend

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants