ggml: fix backend split scheduler race condition

[0cc4m]

Overview

Fixes #23321

Splits without input were running concurrently with other splits, while potentially reusing memory the other split is accessing. In this case, for Qwen models with -nkvo, one Vulkan split contains model.input_embed as input, while a following CPU split without inputs reuses the same memory. The Vulkan backend runs fully asynchronously, it just schedules the copies and the graph execution and returns, so the CPU backend was able to run immediately and overwrite the memory area used for the model.input_embed tensor, before the Vulkan backend actually read it.

The solution I chose here is to make sure all splits run sequentially, since currently the allocator assumes it can reuse memory in following splits. Potentially faster may be if the allocator took concurrency into account and didn't reuse memory for these cases, but that would be much more complicated.

I'm not that familiar with the ggml-backend.cpp code, so let me know if this is not the right way to handle it.

This did not affect CUDA because CUDA cpy_tensor_async does not run for CPU->GPU copies, it falls back to a synchronous copy.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, AI was used for debugging and prototyping the code, I manually reviewed and fixed.

[JohannesGaessler]

As of right now ggml backend events are used in unsafe ways. They are being typecast unconditionally so I think this patch will result in segfaults if multiple different backends (e.g. CUDA + Vulkan) are used together.

[0cc4m]

I can't trigger any crash with this on my system with CUDA + Vulkan, but I don't know if there are edge cases. Are there plans to fix the way events are used? We could also just use synchronize here until then.

[ORippler]

@aendk for backend scheduling related matters

[ORippler]

There is a PR open to do alignment on a backend's expected beahvior: #25319 Unfortunately I have not had the time to follow-up on this

[aendk]

For reference, I was able to reproduce this extremly rarely on underclocked NVIDIA GPUs.
It is harder to do so due to the synchronizations for each CPU-GPU copy. The race condition only occurs on the rare combo when 3 successive splits (CPU->CUDA->CPU) all have no inputs. In that case, the second CPU split can overwrite the output of the first split before it has been copied to CUDA; before the CUDA stream guarantees apply.

To fix this, several options spring to mind:

1. force an additional sync if no syncs took place due to no input (this proposal)
2. add another event mechanism so CPU split N+1 awaits successful copy of CPU split N
3. separate output tensors; CPU split N and N+1 do not write to the same output tensor, no race can occur

As a hotfix, I think 1 is ok; I have not tested performance though.
In the longer term, moving away from synchronizations (like in #25319) is the way to go in my eyes.
Also, inside a single µ-batch, I think we should only synchronize what absolutely needs to be synchronized for maximum performance and that the scheduling is already complex as-is.
From this perspective, Option # 3 currently makes the most sense for me.