[GSD-12874] [BMG / Arc Pro B70] Is cross-root-port P2P expected? L0 dev-to-dev copy returns 0x70000003; kernel refuses at P2PDMA (host bridge not whitelisted) — and can the capability be made queryable?

[TSUMUGI-XE]

Pre-submission Checklist

• I have searched existing issues and discussions for similar questions
• I have checked the documentation and FAQ / programmers-guide

Question Category

API / Development

Other Category

No response

Question

English isn't my first language, so I drafted this with AI help — but every result was measured by me on the hardware described. Full writeup +
reproducer: https://github.com/TSUMUGI-XE/b70-dual-tp2

Confound ruled out (control done). The test host normally runs pcie_acs_override=downstream,multifunction (for an unrelated VFIO passthrough),
which can make the kernel refuse peer P2P via pci_p2pdma_distance() and mimic this signature. I re-ran on a control boot with the override
removed: peer copy still returns 0x70000003, host-staged still passes — identical to baseline (control
log). The override is not the cause. On that boot the
kernel logged the reason: xe 0000:07:00.0: cannot be used for peer-to-peer DMA as the client and provider (0000:03:00.0) do not share an upstream bridge or whitelisted host bridge.

Summary

On a workstation with two Intel Arc Pro B70 (BMG), a Level-Zero direct device-to-device copy between the cards (single process, per-device L0
contexts) returns 0x70000003 (ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY) instead of doing the peer copy. Clean error, not a hang. A host-staged
copy of the same buffers (dev0 → host → dev1) succeeds — so allocations/queues are fine, only the cross-device peer path fails.

This may well be expected behavior. I ruled out my own host config (control above); the kernel then refuses because the two cards are on separate
root ports with no shared PCIe switch and the host bridge isn't on pci_p2pdma_whitelist. So I'm not claiming a clear bug — I'm asking: (1) should
BMG + a consumer Intel host bridge route P2P through the root complex (whitelist gap to fix, or topology limit to document)? and (2) regardless, can
the unsupported case be reported as a queryable capability instead of OUT_OF_DEVICE_MEMORY, so frameworks stop probing-by-failure?

Environment

• GPUs: 2× Intel Arc Pro B70 (BMG), separate PCIe root ports
• Kernel: Linux 7.x, xe driver — reproduced on two builds (stock Ubuntu HWE 7.0.0-22 and a self-built 7.1-rc)
• compute-runtime / Level-Zero: 1.15.x (NEO 26.x, IGC 2.34.x)
• Single process, ONEAPI_DEVICE_SELECTOR=level_zero:*, ZES_ENABLE_SYSMAN=1; IOMMU/VT-d on, GPUs in separate IOMMU groups

Reproducer

~80-line single file: repro/b70_p2p_copy_probe.cpp. Build/run:

source /opt/intel/oneapi/setvars.sh                                                                                                                    
icpx -fsycl -O2 b70_p2p_copy_probe.cpp -lze_loader -o b70_p2p_copy_probe                                                                               
ONEAPI_DEVICE_SELECTOR=level_zero:* ZES_ENABLE_SYSMAN=1 ./b70_p2p_copy_probe                                                                           

Output:

--- TEST 1: L0 direct copy dev0 -> dev1 ---                                                                                                            
  zeCommandListAppendMemoryCopy -> 0x70000003 (FAILED)                                                                                                 
--- TEST 2: host-staged copy dev0 -> host -> dev1 ---                                                                                                  
  checksum: PASS                                                                                                                                       
RESULT: l0_p2p=BROKEN(ze-error)  host_staged=WORKS                                                                                                     

Negative controls (what is NOT the cause)

• Host platform swap (different CPU/chipset, separate root ports, Gen4 x8/x8, separate IOMMU groups) → identical failure. PCIe bandwidth / root-port
  layout / IOMMU grouping ruled out. Reproduces across two kernels.
• pcie_acs_override removed control boot → still 0x70000003, kernel names the P2PDMA refusal (above). Override ruled out. Neither GPU exposes a
  p2pdma sysfs node even with ACS off.

Related (not duplicates — different HW or failure mode)

• intel/compute-runtime [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916 — same OUT_OF_DEVICE_MEMORY class on a 2-GPU Level-Zero context, but A770 (DG2) and urUSMDeviceAlloc, not a peer copy.
• intel/compute-runtime [GSD-12696] [Xe2/BMG-G31] multi-rank MPI + Level Zero: resource_info.cpp:15 abort after upgrading CR 26.05 → 26.14 (Ubuntu 26.04, xe driver) #922 — same dual-B70 (BMG-G31) multi-GPU Level-Zero frontier, but a multi-process zeInit abort, not a single-process peer
  copy.
• vllm-project/vllm #41663 — the downstream TP=2 symptom on the same hardware.

Ask

1. Is BMG + a consumer Intel host bridge eligible for pci_p2pdma_whitelist (does the silicon route P2P through the root complex), or is
   cross-root-port P2P genuinely unsupported here? Whitelist it, or document it?
2. Either way, can compute-runtime report the capability (a zeDeviceCanAccessPeer-style query) so frameworks choose host-staging deliberately
   instead of probing-by-failure?
3. Is 0x70000003 (OUT_OF_DEVICE_MEMORY) the intended code for "peer access refused by the kernel P2PDMA layer", or does it mask the more specific
   reason the kernel already names?

Happy to run additional diagnostics or driver builds — I have the two B70 set up and can iterate.

Additional Notes

No response

[mjsabby]

Do we have an answer from intel on this?

[TSUMUGI-XE]

Follow-up — and to @mjsabby's question above: no technical reply from Intel on my side yet (it's tracked internally as [GSD-12874]), but I have a concrete new result that I think answers my own Ask #1, so I'll add it here.

Short version: I forced the consumer host bridge into pci_p2pdma_whitelist to see if it's eligible. The kernel then allows the peer copy, but the root complex silently corrupts it — the command returns SUCCESS and the data is wrong.

Setup. Same 2× Arc Pro B70 (BMG), on separate CPU root ports, consumer Rocket Lake-S host bridge 8086:4c43. I built a kernel with that host bridge added to pci_p2pdma_whitelist[] and booted with intel_iommu=off (headless, GPUs otherwise idle).

Result.

• The kernel P2PDMA gate now passes — the earlier cannot be used for peer-to-peer DMA ... do not share an upstream bridge or whitelisted host bridge refusal is gone, and a direct dev0→dev1 zeCommandListAppendMemoryCopy returns ZE_RESULT_SUCCESS (0x0) instead of 0x70000003.
• But the destination buffer is fully wrong — checksum FAIL on 100% of elements (errs = 4194304 / 4194304). The same buffers copied host-staged (dev0 → host → dev1) verify correct (checksum PASS).
• So with the whitelist forced open, the operation reports success while the data does not actually arrive — silent corruption.
• The hardware itself is healthy: the PCIe link measures ~14 GB/s H2D/D2H (≈13.96 / 14.13), i.e. full Gen4 x8 per card, and host-staged copies are correct. The only thing that fails is the direct cross-root-port peer copy. (To be clear: that 14 GB/s is host↔device link bandwidth — not a peer-path figure; the peer path moved no valid data, so I'm making no claim about peer throughput.)

Interpretation. Forcing the whitelist makes the kernel allow the mapping, but the RKL-S root complex doesn't correctly route the cross-root-port peer write. The L0/xe side does issue the copy (no 45/39, no capability rejection) — the failure is purely in the peer data path through the root complex. This is consistent with pci_p2pdma_whitelist deliberately excluding consumer client host bridges.

For contrast (not measured by me, just to scope it): Intel's own Platform Evaluation Guide for Multi-ARC notes no expected P2P restriction on AMD hosts, and there are community reports of multi-Arc P2P working on AMD EPYC. So this looks specific to consumer Intel single-IIO host bridges, not Arc P2P in general — Xeon/validated platforms are a separate case I'm not commenting on.

Two practical questions:

1. Is cross-root-port P2P simply unsupported on consumer (RKL-S and similar) single-IIO hosts — i.e. a topology limit worth documenting for Arc Pro multi-GPU, with an external PCIe switch (both GPUs under one upstream bridge) being the only path on these platforms?
2. Independent of (1): since forcing the path yields ZE_RESULT_SUCCESS + corrupt data rather than a clean refusal, can the stack detect/refuse this case (a queryable peer-access capability, or a hard error) so a user can't unknowingly run a workload on a path that returns wrong results? Silent corruption is worse than the original OUT_OF_DEVICE_MEMORY.

I can share the one-line whitelist patch, the ~80-line probe, and the sanitized boot log if any of that is useful.

(English isn't my first language, so I drafted this with AI help — but every result above was measured by me on the hardware described.)

[mjsabby]

Thanks @TSUMUGI-XE. I commented on your repo asking a similar question. Is it worth checking out consumer AMD platform and seeing if it works.

You also specifically call out cross root. Have you tried in a single root and it DOES work or also doesn't work? In other words is there any kernel driver work needed here? I will also clone this repo and ask Claude.

I think it will be tough for Intel to say or reverse course on the fact they themselves probably didn't allow list their client skus for p2p dma.

That said it is concerning that your est did not pass when you have server class skus.

[TSUMUGI-XE]

Thanks @mjsabby — good questions.

Single-root: haven't been able to test it yet, and that's the key gap. Both B70s sit on separate CPU root ports, so I can't make them single-root without changing topology. Linux pci_p2pdma already allows P2P unconditionally when two devices share an upstream PCIe switch — no driver work needed there. The refusal is specific to cross-root-port (no shared switch, host bridge not whitelisted). So I'd expect single-root / behind-a-switch to work at the kernel layer; the open question is whether the data then routes. I'm getting a PEX88096 switch board to put both cards under one upstream bridge and will post ze_peer numbers.

"Kernel driver work needed?" For the shared-switch case, no — the path is already there. For cross-root-port on this consumer host, forcing it (whitelisting the host bridge) produces silent corruption, so the kernel's refusal is correct — it's not a missing driver, it's the root complex not routing peer TLPs.

On the SKU point: the GPUs aren't the problem — the B70s issue the copy and host-staged works. The failing piece is the consumer host's root complex (Rocket Lake-S, 8086:4c43), which isn't in pci_p2pdma_whitelist. On a whitelisted/server host (the list has Skylake-E / Xeon Scalable) or behind a switch, the same GPUs should route P2P.

AMD: agreed, worth checking — I'm on an Intel host so can't test it myself, but Intel's own Multi-ARC guide notes no expected P2P restriction on AMD hosts, and there are community reports of multi-Arc P2P on EPYC. So this looks like a consumer-Intel-root-complex thing, not an Arc-wide limit.

Version caveat (in fairness): my data is from kernel 7.0.0-22 — I stay on 7.0 because 7.1 regresses my Wi-Fi — and I haven't tested released 7.1 or the latest UMF myself. I know it's been reported that the dual-B70 TP=2 path needs Linux ≥7.1; worth noting the one clean working setup I've seen is a Threadripper HEDT with multiple root complexes (TP=2 works across different RCs there, but still crashes on the same RC). The 7.1 xe changelog shows no P2P/multi-GPU changes, so I don't expect a kernel bump alone to flip the cross-root-port verdict on a consumer single-root-complex host — but I'll confirm that with the switch rather than assume it.

Separately — I have a host-staged TP=2 workaround (routes the all-reduce through CPU/gloo so it never touches GPU P2P; works regardless of root-complex topology) plus the ze_peer/checksum + pci_p2pdma distance/whitelist probe scripts I used to characterize all this. Perf is modest but real: on a dense 27B it ran ~1.2x single-stream vs a single GPU and scaled with batch; on an MoE it was capacity-only (single-stream was actually slower than one card). Happy to share both if useful — heads up, I'm traveling for a bit, so I'll post the cleaned-up patch + scripts once I'm back.

Will report back once the switch is in.

[mjsabby]

Yes, your repo was very useful to me for investigations, so please continue. The goal is to not through CPU/gloo for me, but yes your experiment proves it is somewhat usable, although I suspect Intel probably wants to get it fixed.

BTW, you don't need EPYC right? All zen chipsets are allow listed.

[mjsabby]

Another quesiton to ponder, if Windows supports it, given Intel controls the drivers there more strongly than having to wait for upstream Kernel.

[TSUMUGI-XE]

Quick follow-up on the AMD question — I went and checked the actual Linux pci_p2pdma whitelist (drivers/pci/p2pdma.c), and it's asymmetric in a way that's directly relevant:

• AMD: blanket-enabled — if (cpu_vendor == AMD && family >= 0x17) ⇒ any Zen-or-newer host is trusted for p2pdma, no per-device list.
• Intel: an explicit device-ID allowlist of Xeon E5/E7 + Skylake-E only. Consumer parts (mine is Rocket Lake-S, 8086:4c43) aren't in it, so the distance check returns -1 and P2P is refused up front.

So on a Zen+ box your two B70s should clear the kernel gate that mine fails — and the kernel only blanket-trusts AMD because the fabric actually routes peer TLPs (vs. my consumer Intel RC, which even when forced open returns the buffer as all-1s = a clean Unsupported-Request refusal, not a transfer). That makes consumer AMD a genuinely worthwhile test. If you try it, I'd love to know whether p2p_access flips to 1 and a real cross-card copy verifies (not just returns success).

I'm taking the other branch: a PEX (PLX) switch on my Intel box to put both cards under one shared upstream bridge and bypass the root complex entirely — but it's ~2-3 weeks out. So let's split it: you on AMD, me on the switch, and compare notes. Whichever lands first tells us a lot.

[mjsabby]

@TSUMUGI-XE yes I'm going to try with AMD host, your repo's code should be a quick minimal repro to test right?

[TSUMUGI-XE]

Quick answers:

Repro: yes — it's a single ~85-line SYCL/Level-Zero file (repro/b70_p2p_copy_probe.cpp), build icpx -fsycl -O2 ... -lze_loader, runs in seconds. On AMD the key check is that the copy verifies (contents match), not just that it returns success — on my consumer Intel box the call "succeeds" but the destination comes back all-1s (a PCIe Unsupported-Request, i.e. the RC didn't actually move the bytes). So the thing to watch: does zeDeviceCanAccessPeer / p2p_access report 1, and does the copied buffer match the source?

Windows: I don't think it escapes the wall on consumer Intel. The all-1s is a hardware Unsupported-Request from the root complex — that's below the OS, so a Windows driver (even with Intel's tighter control there) hits the same silicon that won't route peer TLPs across root ports. Where the RC does route (AMD Zen / Xeon / behind a PCIe switch), Windows P2P should be fine — but Linux works there too. So Windows is a driver-exposure question, not a fix for the consumer-Intel-RC routing limit.

Looking forward to your AMD numbers — that's the cleanest test of the whole thing.