Feature Request: Configurable threshold for full GPU-offloaded prompt processing during CPU+GPU inference

[DocShotgun]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Llama.cpp is currently one of the best backends to run large MoE models due to the ability to intelligently split the load between the devices, keeping the routed experts on CPU and the always-active-on-every-token params on GPU. Additionally, it can stream the CPU-allocated weights over to the main GPU to completely delegate prompt processing to the GPU, which tends to have much more computing power.

However, depending on the specific model and hardware - most notably the size of the CPU-allocated weights and the PCIe bandwidth - there's a different break-even point, where for small enough prompts, it's not beneficial to fully offload prompt processing to the GPU because combined CPU+GPU prompt processing will finish before you can copy all of the CPU-allocated weights over PCIe. From my understanding, there's a current hardcoded cutoff of 32 tokens.

I believe @jukofyork mentioned having a custom hack to make the threshold configurable via env var. Perhaps it's worth exposing the threshold param officially?

Motivation

To give a concrete example of when this would make a difference:

I have a 4.93bpw Deepseek quant split between CPU and GPU, with over 300gb of CPU-allocated weights and about 94gb of GPU-allocated weights+compute buffers+k/v cache.

If I send a request that requires processing of 100 new tokens of prompt, llama.cpp would still need to copy over 300gb of weights across PCIe to the GPU because 100 exceeds the hardcoded threshold of 32 tokens - which is a minimum of ~5 seconds at max theoretical PCIe 5.0 x16 speeds. Meanwhile, CPU+GPU prompt processing would finish ingesting 100 new tokens in under 5 seconds and lead to better TTFT in this scenario.

Possible Implementation

The simplest solution would be reading an env var or a cli arg instead of the hardcoded value.

It would be interesting to try to determine the break-even point automatically, but this is likely too complex and would probably need to be empirically measured on each model+quant+hardware setup.

[jukofyork]

I believe @jukofyork mentioned having a custom hack to make the threshold configurable via env var. Perhaps it's worth exposing the threshold param officially?

This is what I use to patch the fixed 32 value:

#!/bin/bash

function safe_sed() {
    local file=$1
    local pattern=$2
    local replacement=$3

    # Check if pattern exists
    if ! sed -n "s/${pattern}/${replacement}/p" "$file" | grep -q .; then
        echo "Error: Pattern not found in $file: $pattern"
        return 1
    fi

    # Create backup
    cp "$file" "$file.bak"

    # Perform the replacement
    sed -i "s/${pattern}/${replacement}/g" "$file"

    # Show diff
    echo "Changes in $file:"
    diff "$file.bak" "$file"

    # Clean up
    rm "$file.bak"

    echo "Successfully replaced in $file"
    echo "-------------------"
}

cd
rm -rf llama.cpp

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

safe_sed "ggml/src/ggml-cuda/ggml-cuda.cu" \
        "const int min_batch_size = 32" \
        "const int min_batch_size = getenv(\"GGML_CUDA_MIN_BATCH_SIZE\") ? atoi(getenv(\"GGML_CUDA_MIN_BATCH_SIZE\")) : 32"

cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON 
cmake --build build --config Release -- -j "$(($(nproc)/2))"

(you could just use sed or diff, but I use this for other things and like to know when the patch has failed, etc.)

This then lets you set this value at run-time, like this:

#!/bin/bash

GGML_CUDA_MIN_BATCH_SIZE=2560 ~/llama.cpp/build/bin/llama-server ...

or this:

#!/bin/bash

export GGML_CUDA_MIN_BATCH_SIZE=2560

~/llama.cpp/build/bin/llama-server ...

---

It would be interesting to try to determine the break-even point automatically, but this is likely too complex and would probably need to be empirically measured on each model+quant+hardware setup.

I've found an easy way to tune the value, but it's not fully automatic...

You first have to do a single run of how you expect to use the model - for example for me:

#!/bin/bash

host_address=192.168.1.1
port_number=8080

# Turn off NUMA balancing
echo 0 | sudo tee /proc/sys/kernel/numa_balancing > /dev/null

# Ask for permission to drop caches
read -p "Do you want to drop caches? (y/n) " -n 1 -r
echo    # Move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]
then
    echo "Dropping caches..."
    echo 3 | sudo tee /proc/sys/vm/drop_caches > /dev/null
fi

# Run the main command
~/llama.cpp/build/bin/llama-server \
        --host "$host_address" \
        --port "$port_number" \
        --alias "local/deepseek-r1-0528" \
        --jinja \
        --chat-template-file ~/models/DeepSeek-R1-0528.jinja \
        --model ~/models/gguf/DeepSeek-R1-0528-Q4_X.gguf \
        --n-gpu-layers 99 \
        --numa distribute \
        --threads "$(nproc)" \
        --override-tensor exps=CPU \
        --flash-attn 1 \
        --ctx_size 65536 \
        --batch-size 12288 \
        --ubatch-size 12288

(this is important as lama-bench doesn't seem to warm up MoE models correctly and it will cause weird bias in the next step due to loading the experts throughout the sampling period!)

Then you set GGML_CUDA_MIN_BATCH_SIZE to an initial guess and run this script:

#!/bin/bash

export GGML_CUDA_MIN_BATCH_SIZE=2560
MODEL_PATH="~/models/gguf/DeepSeek-R1-0528-Q4_X.gguf"
BATCH_SIZE=12288

# Assert that GGML_CUDA_MIN_BATCH_SIZE is a multiple of 512
if (( GGML_CUDA_MIN_BATCH_SIZE % 512 != 0 )); then
    echo "GGML_CUDA_MIN_BATCH_SIZE ($GGML_CUDA_MIN_BATCH_SIZE) is NOT a multiple of 512." >&2
    exit 1
fi

~/llama.cpp/build/bin/llama-bench \
	--model "$MODEL_PATH" \
	--batch-size $BATCH_SIZE \
	--ubatch-size $BATCH_SIZE \
	--n-gpu-layers 99 \
	--flash-attn 1 \
	--numa distribute \
	--threads $(nproc) \
	--override-tensor exps=CPU \
	--n-prompt $GGML_CUDA_MIN_BATCH_SIZE \
	--n-gen 0 \
	--no-op-offload 1,0

(NOTE: The requirement that the batch size be a multiple of 512 is a recent regression in llama.cpp and wasn't needed until a couple of weeks ago for me!)

This will print a table with two sets of PP values due to the --no-op-offload 1,0 argument. Let's call these A and B:

A: Is the PP speed without offloading to the GPU.
B: Is the PP speed with offloading to the GPU.

Your goal is to get these two numbers to be about the same (and before the 512-regression I had a script that would use jq to extract the two values and adjust GGML_CUDA_MIN_BATCH_SIZE automatically, but I seem to have lost it now... ☹️).

You can calculate an approximate adjustment factor for your current GGML_CUDA_MIN_BATCH_SIZE by taking the ratio of the two values:

GGML_CUDA_MIN_BATCH_SIZE = GGML_CUDA_MIN_BATCH_SIZE * A/B

it should be clear how this works:

• If A > B we want to increase the threshold where we are offloading to the GPU.
• If A < B we want to decrease the threshold where we are offloading to the GPU.
• If A = B we have the break-even point.

After a couple of iterations of this, you will have your value for this:

#!/bin/bash

GGML_CUDA_MIN_BATCH_SIZE=2560 ~/llama.cpp/build/bin/llama-server ...

It's a bit shit and clunky, but does work.

[Panchovix]

@jukofyork Nice code! Wondering, would that solve the case of #16912? Like skipping having to revert the change on those lanes.

[DocShotgun]

Thanks, I was trying to crawl through the codebase to figure out where that hardcoded 32 was found lol. I think an env var would be fine, but I'm not sure what the maintainers would find the least intrusive, as I'm assuming this is a fairly niche case.

To confirm - for benchmarking you launch llama-server first, then kill it and then launch llama-bench and that retains the warmup from the initial llama-server run? I'm not sure that stuff would even stay cached between those runs or if it would have been automatically evicted to make room.

Also on an unrelated note, I see that you turn off NUMA balancing when launching llama-server. Is this something I should be considering on my setup? I have a dual socket rig with each socket consisting of 2 SNCs, so it's configured as 4 NUMA nodes in total. I'm currently using numactl to bind to NUMA nodes 0,1 which are the socket connected to the GPU.

[jukofyork]

@jukofyork Nice code! Wondering, would that solve the case of #16912? Like skipping having to revert the change on those lanes.

Yeah, you can remove these using a safe_sed() call but might need to call it twice or change the final /p to /g (I forget the exact syntax of sed). I use it for all sorts of hacks like this (where actual git patches get screwed over by things moving around).

[jukofyork]

Thanks, I was trying to crawl through the codebase to figure out where that hardcoded 32 was found lol. I think an env var would be fine, but I'm not sure what the maintainers would find the least intrusive, as I'm assuming this is a fairly niche case.

Yeah, I'm not suggesting we go with this - it's just a nice easy way to hack the fixed 32 value. I think a command line option would make the most sense (something like the current --no-op-offload option, but taking an argument that defaults to 32 maybe?).

To confirm - for benchmarking you launch llama-server first, then kill it and then launch llama-bench and that retains the warmup from the initial llama-server run? I'm not sure that stuff would even stay cached between those runs or if it would have been automatically evicted to make room.

It probably depends on your RAM, but I have have 1.5TB in the machine I use for these large MoE models, so enough to keep the full model in cache. It might not work as well if you can't do this and probably your best bet would be to use --n-prompt "$GGML_CUDA_MIN_BATCH_SIZE,$GGML_CUDA_MIN_BATCH_SIZE" and discard the first 2 sets of results (this still won't load all experts in llama-bench like llama-server will for some unknown reason, but at least it won't be loading them during the second pair of runs).

Also on an unrelated note, I see that you turn off NUMA balancing when launching llama-server. Is this something I should be considering on my setup? I have a dual socket rig with each socket consisting of 2 SNCs, so it's configured as 4 NUMA nodes in total. I'm currently using numactl to bind to NUMA nodes 0,1 which are the socket connected to the GPU.

Yeah, it seems worthwhile for me on this machine, but no idea in general - I have some older dual-xeons with 0.5TB RAM in, and they perform best with numactl --interleave=all (for some unknown reason lol).

[DocShotgun]

Hmmm adding it in as an official CLI arg might complicate things lol... need to think about how it interacts with the other backends besides CUDA as well, and implement it for the relevant backends.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.