cuda: read memory through NVML if available by 0cc4m · Pull Request #23604 · ggml-org/llama.cpp

0cc4m · 2026-05-24T09:58:57Z

Overview

This avoids initializing a context (which takes up a few hundred MB of memory) and brings CUDA in line with Metal and Vulkan, which both also read memory without initialization. #21231 requires this change, otherwise reading CUDA available memory initializes VRAM in the router process. See also #21231 (comment). It's a soft-dependency on the nvml driver library, if it is not available or initialization fails, it falls back to the standard cudaMemGetInfo path. iGPUs on Linux also skip directly into the existing /proc/meminfo function.

I also looked into using RSMI for ROCm, but when testing I did not see VRAM being allocated from the use of hipMemGetInfo, so I skipped it.

@ServeurpersoCom FYI

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, Claude was used to implement, I reviewed and tested the changes afterwards.

…ontext Co-authored-by: Pascal <admin@serveurperso.com>

JohannesGaessler

Logically this seems correct to me, I only have requests for cosmetic changes.

JohannesGaessler · 2026-05-24T10:53:04Z

+#include <windows.h>
+#else
+#include <dlfcn.h>
+#endif


Suggested change

#endif

#endif // defined(_WIN32)

JohannesGaessler · 2026-05-24T10:54:23Z

+        auto init  = (ggml_nvml_init_fn)   dlsym(lib, "nvmlInit_v2");
+        get_handle = (ggml_nvml_handle_fn) dlsym(lib, "nvmlDeviceGetHandleByPciBusId_v2");
+        get_mem    = (ggml_nvml_mem_fn)    dlsym(lib, "nvmlDeviceGetMemoryInfo");
+#endif


Suggested change

#endif

#endif // defined(_WIN32)

JohannesGaessler · 2026-05-24T10:58:37Z

+        GGML_LOG_DEBUG("%s: %s: using NVML context-free path\n", __func__, ctx->name.c_str());
+        return;
+    }
+#endif


Suggested change

#endif

#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)

JohannesGaessler · 2026-05-24T10:58:55Z


+#if !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)
+    if (ggml_cuda_nvml_get_memory(ctx->pci_bus_id, free, total)) {
+        GGML_LOG_DEBUG("%s: %s: using NVML context-free path\n", __func__, ctx->name.c_str());


Suggested change

GGML_LOG_DEBUG("%s: %s: using NVML context-free path\n", __func__, ctx->name.c_str());

GGML_LOG_DEBUG("%s: %s: using NVML path\n", __func__, ctx->name.c_str());

JohannesGaessler · 2026-05-24T10:59:17Z

+    }
+#endif
+
+    GGML_LOG_DEBUG("%s: %s: using cudaMemGetInfo fallback\n", __func__, ctx->name.c_str());


Suggested change

GGML_LOG_DEBUG("%s: %s: using cudaMemGetInfo fallback\n", __func__, ctx->name.c_str());

GGML_LOG_DEBUG("%s: %s: using cudaMemGetInfo fallback - this results in eager memory allocation\n", __func__, ctx->name.c_str());

JohannesGaessler · 2026-05-24T11:02:18Z

@ORippler @gaugarg-nv since I am not familiar with NVML, if either one of you could also review this PR it would be appreciated.

ServeurpersoCom · 2026-05-24T12:02:44Z

Thanks! I only ran a quick check on Linux so far, so I'll follow up with a small standalone GGML test on Windows: load nvml.dll via LoadLibraryA, then compare the values returned by the NVML path against cudaMemGetInfo to confirm they match on WDDM. Will report back!

ServeurpersoCom · 2026-05-24T13:20:27Z

Made a small cross-platform tool with Claude Opus to sanity check this on Linux and Windows : 3d09638

It reads VRAM two ways and compares them: the NVML path from this PR, and a plain cudaMemGetInfo that forces the primary context. The NVML read matches ggml's device query exactly and stays contextless, while cudaMemGetInfo permanently eats:

RTX PRO 6000 (Blackwell): 555 MiB
RTX 4060 Ti (Ada): 107 MiB

So the cost scales with SM count, and #21231 would pay it in the router just to read margins. The NVML path avoids that entirely. Windows numbers are a bit noisier (WDDM keeps moving VRAM around) but show the same picture.

Linux

(root|~/llama.cpp.pascal) ./build/bin/llama-gpu-meminfo
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97247 MiB):
  Device 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition, compute capability 12.0, VMM: yes, VRAM: 97247 MiB
backend agnostic view: 2 device(s)
ggml_backend_cuda_device_get_memory: CUDA0: using NVML path
  [0] GPU   CUDA0                  free  97244.8 MiB / total  97887.0 MiB  (NVIDIA RTX PRO 6000 Blackwell Workstation Edition)
  [1] CPU   CPU                    free  94190.7 MiB / total  94190.7 MiB  (AMD Ryzen 9 9950X3D 16-Core Processor)

cuda two modes: 1 device(s)
  [0] 0000:01:00.0
       nvml free before :  97244.8 MiB
       cudaMemGetInfo   :  96689.5 MiB free /  97247.7 MiB total
       nvml free after  :  96689.5 MiB
       context cost     :    555.3 MiB
(root|~/llama.cpp.pascal)

Windows:

D:\A\git\llama.cpp>build\bin\Release\llama-gpu-meminfo.exe
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 16379 MiB):
  Device 0: NVIDIA GeForce RTX 4060 Ti, compute capability 8.9, VMM: yes, VRAM: 16379 MiB
backend agnostic view: 2 device(s)
ggml_backend_cuda_device_get_memory: CUDA0: using NVML path
  [0] GPU   CUDA0                  free  15418.9 MiB / total  16380.0 MiB  (NVIDIA GeForce RTX 4060 Ti)
  [1] CPU   CPU                    free  48399.4 MiB / total  65277.3 MiB  (12th Gen Intel(R) Core(TM) i9-12900K)

cuda two modes: 1 device(s)
  [0] 0000:01:00.0
       nvml free before :  15418.9 MiB
       cudaMemGetInfo   :  15233.0 MiB free /  16379.5 MiB total
       nvml free after  :  15312.2 MiB
       context cost     :    106.7 MiB

D:\A\git\llama.cpp>

gaugarg-nv · 2026-05-24T13:25:50Z

 #include <vector>

+#if defined(_WIN32)
+#include <windows.h>


Suggested change

#include <windows.h>

#define WIN32_LEAN_AND_MEAN

#ifndef NOMINMAX

#define NOMINMAX

#endif

#include <windows.h>

The other code in the project defines these two macros before windows.h to avoid compilation errors.

gaugarg-nv · 2026-05-24T13:43:00Z

NVML changes look fine to me.

ORippler

This avoids initializing a context (which takes up a few hundred MB of memory) and brings CUDA in line with Metal and Vulkan, which both also read memory without initialization. #21231 requires this change, otherwise reading CUDA available memory initializes VRAM in the router process.

CUDA contexts do have a sizeable overhead:
https://forums.developer.nvidia.com/t/whats-cuda-contexts-gpumemory-contain-is-it-necessary-and-available-to-minimum-the-cuda-contexts-gpumemorysize/289411

https://forums.developer.nvidia.com/t/does-cuda-automatically-allocate-more-gpu-memory-during-the-initialization-of-the-application/314666

This overhead will be required when actually doing work in the backend, and thus should be accounted for when budgeting/reporting available VRAM.

In this light, skimming #21231 it seems probing in the child, not the router
is the way we should proceed. General guidance is to have one CUDA context per GPU & application open at a time.

An alternative path would be to add backend destructures/resets so we can construct/destruct the CUDA context via cudaDeviceReset inside CUDA Runtime as needed.

ORippler · 2026-05-26T19:43:21Z

Reading up on this a bit more, I'd say the best path would be to add cudaDeviceReset into the destructor of ggml_backend_cuda_context, which is called by cuda backend's .free method. Following the one-cuda-context-per-device-per-application guidance would be up to llama-server to implement (either ensure child is uninitialized and construct/destruct the backend in the router or move probing to the child)

JohannesGaessler · 2026-05-26T21:23:51Z

Maybe I misunderstood the intent of this PR. I thought it was to avoid allocating VRAM in a situation where ggml is compiled both with CUDA and another backend and one wouldn't want to allocate memory unless a backend is actually used.

JohannesGaessler · 2026-05-26T21:37:36Z

How about this: extend ggml_backend_cuda_device_context with a counter that is incremented/decremented when a ggml_backend_cuda_context is created/destroyed. Call cudaDeviceReset once all backends have been destroyed. In ggml_backend_cuda_get_device_memory, call cudaDeviceReset if the counter is currently at 0.

0cc4m · 2026-06-05T17:44:15Z

Superceded by #23935

cuda: read memory through NVML if available to avoid initializing a c…

3c4d2b7

…ontext Co-authored-by: Pascal <admin@serveurperso.com>

0cc4m requested a review from a team as a code owner May 24, 2026 09:58

github-actions Bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels May 24, 2026

JohannesGaessler approved these changes May 24, 2026

View reviewed changes

gaugarg-nv reviewed May 24, 2026

View reviewed changes

address feedback

f3ba33e

ORippler requested changes May 26, 2026

View reviewed changes

ORippler mentioned this pull request May 26, 2026

server: add router device memory margin parameter for dynamic unloading #21231

Open

0cc4m mentioned this pull request May 31, 2026

cuda: reset cuda context after reading memory size #23935

Merged

0cc4m closed this Jun 5, 2026

	#endif
	#endif // !defined(GGML_USE_HIP) && !defined(GGML_USE_MUSA)

	GGML_LOG_DEBUG("%s: %s: using NVML context-free path\n", __func__, ctx->name.c_str());
	GGML_LOG_DEBUG("%s: %s: using NVML path\n", __func__, ctx->name.c_str());

	GGML_LOG_DEBUG("%s: %s: using cudaMemGetInfo fallback\n", __func__, ctx->name.c_str());
	GGML_LOG_DEBUG("%s: %s: using cudaMemGetInfo fallback - this results in eager memory allocation\n", __func__, ctx->name.c_str());

-#include <windows.h>
+#define WIN32_LEAN_AND_MEAN
+#ifndef NOMINMAX
+    #define NOMINMAX
+#endif
+#include <windows.h>

Uh oh!

Conversation

0cc4m commented May 24, 2026

Overview

Requirements

Uh oh!

JohannesGaessler left a comment

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler May 24, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler May 24, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler May 24, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler May 24, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler May 24, 2026

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler commented May 24, 2026

Uh oh!

ServeurpersoCom commented May 24, 2026

Uh oh!

ServeurpersoCom commented May 24, 2026

Linux

Windows:

Uh oh!

gaugarg-nv May 24, 2026

Choose a reason for hiding this comment

Uh oh!

gaugarg-nv commented May 24, 2026

Uh oh!

ORippler left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ORippler commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JohannesGaessler commented May 26, 2026

Uh oh!

JohannesGaessler commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0cc4m commented Jun 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ORippler left a comment •

edited

Loading

ORippler commented May 26, 2026 •

edited

Loading

JohannesGaessler commented May 26, 2026 •

edited

Loading