CUDA: Support CUDA Virtual Devices

[anavp-nvidia]

Overview

Adds support for exposing a configurable number of virtual CUDA devices on top of the physical GPUs actually present, controlled by a new GGML_CUDA_DEVICES environment variable. This allows developing and exercising multi-GPU code paths on machines with a single GPU.

When GGML_CUDA_DEVICES is unset, behavior is identical to before.

-sm layer, -sm row, and -sm tensor are all supported under virtual CUDA devices. For -sm tensor, the internal CUDA AllReduce or the meta-backend fallbacks are supported since NCCL seems to require distinct physical GPUs.

Additional information

The core idea is a single indirection: every (virtual) device id is mapped to a backing physical device id, and all raw CUDA calls that take a device ordinal use the physical id, while ggml-level bookkeeping (pools, buffers, device registry) stays per-virtual-device.

Design choices

• ggml_cuda_init() parses GGML_CUDA_DEVICES, builds the round-robin virtual to physical map, and computes how many virtual devices share each GPU. ggml_cuda_set_device() resolves to the physical device, and a new ggml_cuda_get_physical_device() exposes the mapping.
• ggml_cuda_device_info gains physical_device_count and, per device, physical_device and physical_share_count.
• Property/attribute/VMM/peer-copy/PCI/memInfo calls use the physical id. Device registration appends a -v<i> suffix to each virtual device's pci_bus_id so they stay unique.
• op_mul_mat_device (context member, set by ggml_cuda_op_mul_mat dispatcher) is used to identify the main device in mul mat execution instead of ggml_cuda_get_device(), which returns the physical device and was otherwise causing out-of-bounds writes.
• VRAM accounting: a virtual device reports total/physical_share_count for both free and total memory, so per-device VRAM sums to the physical total and model distribution doesn't overcommit a shared GPU.

Testing

Validated on three Blackwell systems, using four models (Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct, Llama-3.3-70B-Instruct) with greedy decoding (temp 0) and a fixed seed, so as to compare virtual-device output against physical-GPU reference.

System (GPUs)	VRAM	Coverage
2× B200 (Linux)	2 × 192 GB	op + e2e + VRAM tests
2× RTX 5090 (Windows)	2 × 32 GB	e2e tests
1× RTX PRO 6000 Blackwell (Windows)	1 × 96 GB	e2e tests + single-GPU emulation

• Tested ops using test-backend-ops under 2 vdev (virtual device) / 1 phys (physical device) configuration. Tested MUL_MAT / MUL_MAT_ID on baseline, 2 vdev / 1 phys and 4 vdev / 2 phys configurations.
• Ran end-to-end tests across configurations including 2 vdev / 2 phys, 2 vdev / 1 phys, 3 vdev / 2 phys, 4 vdev / 2 phys, 4 vdev / 1 phys for -sm layer, -sm row, and -sm tensor (non-NCCL paths):
  • -sm layer / -sm row: output identical to the real-GPU baseline (flash-attn on/off on 1B/8B; on for 32B/70B)
  • -sm tensor: valid coherent output, deterministic across configurations (n vdev / 1 phys where n = 2, 3, 4), and independent of the virtual to physical mapping (n vdev / 1 phys produces the same as n vdev / 2 phys, n = 2, 4). Both all-reduce fallbacks exercised (GGML_CUDA_ALLREDUCE=internal/none).
• VRAM accounting: per-virtual-device memory = physical / N and sums back to the physical total.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, AI was used to understand the code base, for the initial draft of changes, for overall validation and thorough testing.

[gaugarg-nv]

Could you please rebase your changes on the TOT master?

[Kononnable]

This allows developing and exercising multi-GPU code paths on machines with a single GPU.

Can't this already be achieved by using RPC without any additional code changes? Connecting to a ggml-rpc-server running on the same machine as llama-server should result in multiple devices being recognized and should allow for multi-gpu code paths on a machine with a single gpu to be used for testing/development.

edit: note from future self: Virtual device is much lighter abstraction than rpc, which additionally does not support all operations that virtual device do. Similar feature already exists for metal backend.

[gaugarg-nv]

IIUC, these dev_ids are virtual devices, ncclCommInitAll will fail if you pass [0,1] on a single device.

I think it might be better to disable the NCCL path for the virtual device case.

[anavp-nvidia]

I've disabled the NCCL path when virtual devices are being used.

[gaugarg-nv]

Does the internal allReduce implementation work fine with virtual devices?

[anavp-nvidia]

Yes, the internal AllReduce does work with virtual devices because every CUDA-ordinal call in the pipeline goes through ggml_cuda_set_device() which uses the virtual to physical device mapping. Also, each rank's buffers / streams / events are separate allocations, so two ranks backed by the same physical GPU don't collide. Let me know if I've missed anything here.

In my testing, I verified this by forcing each AllReduce path with virtual devices via GGML_CUDA_ALLREDUCE (internal / none) and setting device count via GGML_CUDA_DEVICES.

• internal path: CUDA_VISIBLE_DEVICES=0 GGML_CUDA_DEVICES=2 (2 vdevs, 1 phys) plus explicit GGML_CUDA_ALLREDUCE=internal. Got coherent and deterministic results, and the chunked kernel and copy-engine paths agreed (toggled via GGML_CUDA_AR_COPY_THRESHOLD).
• meta backend (butterfly) fallback: GGML_CUDA_ALLREDUCE=none (n=2), and auto-fallback for n=3,4 was also coherent and deterministic.
• 2 vdevs / 1 phys gave byte-identical output to 2 vdevs / 2 phys.

This testing was done across Linux (2x B200) and Windows (2x RTX 5090).

[ggerganov]

Let's try adding GPUx2 jobs for the CUDA server CI workflow, similar to the existing multi-GPU Metal jobs:

llama.cpp/.github/workflows/server-self-hosted.yml

Lines 83 to 101 in 683f0c7

	- name: Tests (GPUx2)
	id: server_integration_tests_gpu2
	if: ${{ !github.event.pull_request }}
	run: |
	cd tools/server/tests
	source venv/bin/activate
	export GGML_METAL_DEVICES=2
	pytest -v -x -m "not slow"
	- name: Tests (GPUx2, backend-sampling)
	id: server_integration_tests_gpu2_backend_sampling
	if: ${{ !github.event.pull_request }}
	run: |
	cd tools/server/tests
	source venv/bin/activate
	export GGML_METAL_DEVICES=2 LLAMA_ARG_BACKEND_SAMPLING=1
	pytest -v -x -m "not slow"

Btw, I notice that with CUDA, I have to explicitly pass the 2 devices with -dev CUDA0,CUDA1, while on the Mac they are automatically used without passing the -dev CLI argument. Not sure why that is, but it will likely make the new CUDA GPUx2 jobs work with only one GPU if we don't add some extra env vars to the run command:

# this automatically uses 2 virtual Metal devices:
make -j && GGML_METAL_DEVICES=2 ./bin/llama-completion -hf ggml-org/Qwen3-0.6B-GGUF:Q8_0 -p "I believe the meaning of life is" -n 32 --sampling-seq "k" --top-k 1 -no-cnv -lv 4

# in contrast, this uses only one device:
make -j && GGML_CUDA_DEVICES=2 ./bin/llama-completion -hf ggml-org/Qwen3-0.6B-GGUF:Q8_0 -p "I believe the meaning of life is" -n 32 --sampling-seq "k" --top-k 1 -no-cnv -lv 4

# adding the devices explicitly works as expected
make -j && GGML_CUDA_DEVICES=2 ./bin/llama-completion -hf ggml-org/Qwen3-0.6B-GGUF:Q8_0 -p "I believe the meaning of life is" -n 32 --sampling-seq "k" --top-k 1 -no-cnv -lv 4 -sm layer -dev CUDA0,CUDA1

[JohannesGaessler]

In terms of the program logic this is I think a good implementation and would require only minimal changes from my end.

[JohannesGaessler]

Preferable the device description would specify the index of a virtual device per physical device.

[anavp-nvidia]

Btw, I notice that with CUDA, I have to explicitly pass the 2 devices with -dev CUDA0,CUDA1, while on the Mac they are automatically used without passing the -dev CLI argument. Not sure why that is, but it will likely make the new CUDA GPUx2 jobs work with only one GPU if we don't add some extra env vars to the run command:

@ggerganov I haven't been able to reproduce this issue in my tests across two systems. In my tests, GGML_CUDA_DEVICES=N correctly creates and uses the virtual devices without having to use -dev.

Preferable the device description would specify the index of a virtual device per physical device.

@JohannesGaessler the description now looks like the following:

GGML_CUDA_DEVICES=4 ./build/bin/llama-completion --list-devices
Available devices:
  CUDA0: NVIDIA B200 (physical device 0, virtual device 0) (91312 MiB, 90999 MiB free)
  CUDA1: NVIDIA B200 (physical device 1, virtual device 0) (91312 MiB, 90999 MiB free)
  CUDA2: NVIDIA B200 (physical device 0, virtual device 1) (91312 MiB, 90999 MiB free)
  CUDA3: NVIDIA B200 (physical device 1, virtual device 1) (91312 MiB, 90999 MiB free)

[ggerganov]

In my tests, GGML_CUDA_DEVICES=N correctly creates and uses the virtual devices without having to use -dev.

Yes, I think the problem was on my end. It works correctly now.

Running the CI here to see how it goes: https://github.com/ggml-org/llama.cpp/actions/runs/29398656862/job/87297892207. This should run the server E2E tests on a DGX Spark machine using 2 virtual devices.

[ggerganov]

nit: this can be a static function inside ggml-cuda.cu - we don't expect it to be used in the rest of the CUDA backend, so no need to declare it in the common header.

[anavp-nvidia]

Done

[ggerganov]

After resolving the conflict we can merge.

[ggerganov]

In my tests, GGML_CUDA_DEVICES=N correctly creates and uses the virtual devices without having to use -dev.

Yes, I think the problem was on my end. It works correctly now.

@anavp-nvidia To follow-up on this, the problem does occur only on the DGX Spark for some reason. Here is a command to test with:

make -j && GGML_CUDA_DEVICES=2 ./bin/llama-completion -hf ggml-org/Qwen3-0.6B-GGUF:Q8_0 -p "I believe the meaning of life is" -n 32 --sampling-seq "k" --top-k 1 -no-cnv -lv 4

On my RTX 5090 box this correctly uses 2 devices, while on the DGX Spark it uses only one:

# running on the RTX 5090 machine (correct):
0.00.638.987 I common_memory_breakdown_print: | memory breakdown [MiB]                                     | total    free    self   model   context   compute    unaccounted |
0.00.638.990 I common_memory_breakdown_print: |   - CUDA0 (RTX 5090 (physical device 0, virtual device 0)) | 16054 = 10763 + (2831 =   239 +    2400 +     192) +        2459 |
0.00.638.990 I common_memory_breakdown_print: |   - CUDA1 (RTX 5090 (physical device 0, virtual device 1)) | 16054 = 10763 + (2927 =   364 +    2080 +     482) +        2363 |
0.00.638.990 I common_memory_breakdown_print: |   - Host                                                   |                   321 =   157 +       0 +     164                |

# running on the DSX Spark (incorrect):
0.00.712.915 I common_memory_breakdown_print: | memory breakdown [MiB]                                 | total    free    self   model   context   compute    unaccounted |
0.00.712.921 I common_memory_breakdown_print: |   - CUDA0 (GB10 (physical device 0, virtual device 0)) | 61286 = 58654 + (5382 =   604 +    4480 +     298) +       -2751 |
0.00.712.921 I common_memory_breakdown_print: |   - Host                                               |                   201 =   157 +       0 +      44                |

This likely means that the server CI workflow that we added in this PR does not actually exercise the virtual devices because it runs on a DGX Spark machine.

[ggerganov]

@anavp-nvidia To follow-up on this, the problem does occur only on the DGX Spark for some reason.

@ggml-org/ggml-cuda This continues to be a problem - for some reason the DGX Spark systems need to explicitly list the virtual devices in the command line, otherwise they are not being picked-up. This makes most multi-GPU runs of the CI useless because they effectively run on one GPU.

For example, here when the workflow runs on non-DGX Spark machine it correctly reveals an issue: https://github.com/ggml-org/llama.cpp/actions/runs/31569042233/job/94026908737#step:7:96

But the same workflow when running on a DGX Spark does not detect the issue (due to the problem explained above):

https://github.com/ggml-org/llama.cpp/actions/runs/31522118519/job/93881466133