[SYCL] add dev2dev memcpy by SYCL API - #24476
Merged
Merged
Conversation
github-actions
Bot
added
documentation
arthw
added
the
merge ready
|
PR crashes at startup on dual Intel Arc Pro B70 (Battlemage), Windows, oneAPI 2026. It dies on the first cross-GPU dev2dev copy (the new SYCL path), with Crash: Command: Build: llama.cpp It's not an actual OOM — the build without #24476 loads the same model on the same two GPUs without issue. Update: A Logs attached. |
|
I let loose Opus to see if it was a driver or sycl usage issue and here is what it reported. Dug into why it fails on the B70, and it turns out to be a Battlemage peer-access (P2P) limitation, not a context/usage bug. So the fix needs to be a capability check rather than a platform check. Not a null pointer, not a real OOM. A Error 39 is a misleading UR/Level-Zero code here. Minimal standalone SYCL repro (no llama.cpp), two Arc Pro B70 — cross-device
The decisive line: Implications for this PR:
|
Contributor
Author
|
@grukx Thank you for your feedback! |
Contributor
Author
|
@grukx Thank you! |
Contributor
Author
|
@grukx Thank you! |
ggerganov
removed
the
merge ready
|
Tested your updated branch (1ee0ada, "update the detect method for p2p comm") on the dual Intel Arc Pro B70 (Battlemage), Windows, oneAPI 2026. Both configs now run cleanly, no crash and correct output. Build: PR #24476 head 1ee0ada (base fdc3db9), Intel oneAPI 2026 DPC++/C++, static SYCL. 1) Default settings: 2) GGML_SYCL_ENABLE_LEVEL_ZERO=1 + GGML_SYCL_DEV2DEV_MEMCPY=1: Thanks for fixing this. |
Contributor
Author
|
@grukx I will update to merge as soon. |
arthw
added
the
merge ready
|
I also verified this, and no more crashes. When can we have this checked in? |
Contributor
Author
It's ready to merge. Wait for owner final review and merge. Thank you for your support! |
|
Hello - I have setup with 2 x B50 PRO if you guys would like to test something ... |
Contributor
Author
|
@pankleks Could you test by llama-server or llama-cli with multiple GPUs on latest code? For two cases: Expect: Thank you! |
|
both answers are ok: log from MEMCPY=0 and MEMCPY=1 |
Contributor
Author
|
@pankleks Thank you very much! :) |
|
if you need anything else - let me know |
|
Thanks for this work. I know it's already merged, but I've been testing dual Arc Pro B70 (BMG-G31) on Linux/xe and wanted to share what I'm seeing in case you have more context. Happy to move this somewhere more appropriate if there's a better place for it. With a small kernel patch to add the Arrow Lake-S host bridge to the pci_p2pdma whitelist, the B70s do report peer access: Digging into compute-runtime, Two questions if you have time. Has direct peer copy ever actually been shown working on these (vs. host-staged), and was there any attempt to push this upstream to the Intel driver? I'm trying to figure out whether the unsupported copy is a permanent limit, an unfinished xe/compute-runtime path, or something about my hardware. Happy to share the probes and logs if it's useful. |
|
@ccross vllm-project/vllm#41663 and intel/compute-runtime#935 -- we're all still trying to figure it out. What I'm doing is trying the B70's in a threadripper setup, so at least the cross port stuff is eliminated. |
Contributor
Author
|
@ccross @mjsabby I have no dual B60/B70 PC to test this issue. We don't know when it will be fixed by driver. Hope it be fixed in driver and more linux kernel. |
papamoose
pushed a commit
to papamoose/llama.cpp
that referenced
this pull request
Jun 27, 2026
* add dev2dev memcpy by SYCL API * mv GGML_SYCL_DEV2DEV_MEMCPY to runntime table * update the detect method for p2p comm * fix the erro created during fix confilct --------- Co-authored-by: Neo Zhang <NA>
|
@ccross I just confirmed, you need an AMD host with the two Intel B70 GPUs to have P2P along with Kernel 7.1 and Intel Compute Runtime 26.22.38646.4. Prompt processing does get a pretty big boost, no real imapct on token generation. @Spruill-1's change #24152 really has made it worth while to have two B70's and llama.cpp in an AMD setup. |
adrianhoehne
pushed a commit
to adrianhoehne/llama.cpp
that referenced
this pull request
Jul 5, 2026
* add dev2dev memcpy by SYCL API * mv GGML_SYCL_DEV2DEV_MEMCPY to runntime table * update the detect method for p2p comm * fix the erro created during fix confilct --------- Co-authored-by: Neo Zhang <NA>
|
Possibly relevant for people who built a multi-GPU setup on Intel's consumer platforms: NVIDIA/open-gpu-kernel-modules#1253 This seems to be a hardware/firmware issue. It seems like direct GPU-originating writes may be functioning fine, reads being the problem part. The bandwidth is (possibly?) cut, but latency might be able to be at least somewhat regained. Neither GPUs nor PCIe are my domains of knowledge, but a thought has come to me: Should one reverse the direction to a write then the matter mostly reduces to a notification of completion to the other GPU. If doing that is possible, practical or beneficial is beyond my expertise. |
Contributor
Author
It's a little complex issue in hardware/firmware in fact. Thank you! |
|
@arthw I totally understand, but mentioning it in official intel docs about the caveats will help B70 adoption. I've basically got Claude and Copilot to write gemma and qwen for Intel B70 P2P dual GPU setup and it is faster than llama.cpp and vLLM for single token stream and can fit unquantized weights up 90k on dual GPUs for Gemma 31b. https://github.com/mjsabby/qwen35-intel-serve and https://github.com/mjsabby/gemma-intel-serve For example, I use 1 SYCL context, llama.cpp uses 2 SYCL context and all sorts of issues come because of it. |
Contributor
|
@mjsabby llama.cpp and vLLM must support multiple LLMs and models of GPU. Thank you for your sharing! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There are several issues about abnormal output in multiple GPUs case, like: #23612.
After disable dev2dev memcpy by L0 API, all issues disappear.
This PR is used to fix them.
This PR use SYCL API to implement dev2dev memcpy directly for dGPU+dGPU case. It should be quicker than host forward mode, and has same performance of L0 API as the test result.
For iGPU+dGPU case, still use host forward mode.
Add running time environment variable GGML_SYCL_DEV2DEV_MEMCPY to choose the SYCL or L0 API in dev2dev memory copy.
Value:
* 0: SYCL API (default)
* 1: L0 API -- L0 API is found to lead to abnormal crash in some case. This debug flag is used to check the issue.
The performance of two APIs are not obvious difference on two Arc770 cases: