llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts by Ghimli · Pull Request #26414 · ggml-org/llama.cpp

Ghimli · 2026-08-01T20:38:21Z

Overview

Add --pin-hot-experts N to dynamically pin the top-N most frequently used MoE experts in RAM via mlock(), preventing OS page cache eviction. Large MoE models loaded via mmap suffer from severe latency spikes when the OS evicts experts from RAM and they need to be paged back from disk on later use. This feature tracks expert usage at runtime through a tensor callback on ffn_moe_topk tensors and on-the-fly replaces cold pinned experts with hotter ones.

Dense (non-MoE-expert) tensors residing in host memory are mlocked automatically before any hot experts, since they are used on every token. They consume the global budget first, leaving the remainder for hot expert pinning.

A new load mode --load-mode mmap+pin is added for mmap without global mlock, designed to pair with --pin-hot-experts.

CLI arguments

Argument	Default	Description
`--pin-hot-experts <n>`	`0` (disabled)	Number of globally-hot experts to pin in RAM
`--pin-hot-experts-budget-mib <n>`	`0` (unlimited)	Maximum pinned memory in MiB
`--pin-hot-experts-stats-interval <n>`	`0` (disabled)	Print expert stats every N tokens
`--load-mode mmap+pin`	—	mmap without global mlock

Key design points

Real-time tracking via tensor callback on ffn_moe_topk tensors
On-the-fly eviction — cold experts are munlock()'d when replaced by hotter ones
Budget cap via --pin-hot-experts-budget-mib
Fail-safe rollback on mlock failure
Thread-safe with mutex-protected std::set ranking
Selective pinning — only hot experts + dense parts, not the whole model

Example

llama-server \
    -m /path/to/moe-model-00001-of-00006.gguf \
    --load-mode mmap+pin \
    --pin-hot-experts 190 \
    --pin-hot-experts-budget-mib 130000 \
    --pin-hot-experts-stats-interval 2000 \
    -lv 4

Additional information

Files changed

include/llama.h — public API fields in llama_context_params, new LLAMA_LOAD_MODE_MMAP_PIN enum
src/llama.cpp — load mode name/parser for mmap+pin
src/llama-model.cpp — use_mlock logic (MMAP_PIN does NOT trigger global mlock)
src/llama-hot-experts.cpp / src/llama-hot-experts.h — core implementation (new files)
src/llama-context.cpp / src/llama-context.h / src/llama-cparams.h — context integration
common/arg.cpp / common/common.cpp / common/common.h — CLI argument handling
src/CMakeLists.txt — build integration

Hardware / testing environment

CPU: Intel Core i9-13900KF (8P + 16E, 32 threads)
GPU: NVIDIA GeForce RTX 4090 (24 GB VRAM)
RAM: 152 GiB DDR5
OS: Ubuntu 24.04, Linux 7.0.0, x86_64

Baseline test (no pinning, plain mmap):

llama-server \
    -m "/home/ghimli/rootmodels/GLM-5.2-UD-Q2_K_XL-00001-of-00007.gguf" \
    --ctx-size 131072 --threads 32 --flash-attn on -np 1 \
    --fit off --no-warmup --cache-type-k q8_0 --cache-type-v q8_0 \
    --host 127.0.0.1 --port 7777 --jinja --reasoning-preserve \
    --n-gpu-layers 999 --cpu-moe -lv 4

============================================
  Benchmarking llama-server on 127.0.0.1:7777
============================================

  Rep 1/5... total=195.422266s prompt=1544tok gen=128tok
  Rep 2/5... total=65.140298s prompt=1544tok gen=128tok
  Rep 3/5... total=53.022765s prompt=1544tok gen=128tok
  Rep 4/5... total=49.189438s prompt=1544tok gen=128tok
  Rep 5/5... total=51.243147s prompt=1544tok gen=128tok

Results (5 repetitions):

  Prompt tok/s:  avg=24.4  stdev=9.7
  Gen tok/s:     avg=2.0  stdev=0.8

  Raw data:
    Rep 1: prompt=7.9 tok/s, gen=0.6 tok/s
    Rep 2: prompt=23.7 tok/s, gen=1.9 tok/s
    Rep 3: prompt=29.1 tok/s, gen=2.4 tok/s
    Rep 4: prompt=31.3 tok/s, gen=2.6 tok/s
    Rep 5: prompt=30.1 tok/s, gen=2.4 tok/s

Hot experts pinned test (`--load-mode mmap+pin --pin-hot-experts 160`):

llama-server \
    -m "/home/ghimli/rootmodels/GLM-5.2-UD-Q2_K_XL-00001-of-00007.gguf" \
    --ctx-size 131072 --threads 32 --flash-attn on -np 1 \
    --fit off --no-warmup --cache-type-k q8_0 --cache-type-v q8_0 \
    --host 127.0.0.1 --port 7777 --jinja --reasoning-preserve \
    --n-gpu-layers 999 --cpu-moe -lv 4 \
    --load-mode mmap+pin \
    --pin-hot-experts-stats-interval 2000 \
    --pin-hot-experts 160

============================================
  Benchmarking llama-server on 127.0.0.1:7777
============================================

  Rep 1/5... total=217.214807s prompt=1544tok gen=128tok
  Rep 2/5... total=46.257479s prompt=1544tok gen=128tok
  Rep 3/5... total=34.168737s prompt=1544tok gen=128tok
  Rep 4/5... total=30.620768s prompt=1544tok gen=128tok
  Rep 5/5... total=29.886910s prompt=1544tok gen=128tok

Results (5 repetitions):

  Prompt tok/s:  avg=37.5  stdev=18.5
  Gen tok/s:     avg=3.0  stdev=1.5

  Raw data:
    Rep 1: prompt=7.1 tok/s, gen=0.5 tok/s
    Rep 2: prompt=33.3 tok/s, gen=2.7 tok/s
    Rep 3: prompt=45.1 tok/s, gen=3.7 tok/s
    Rep 4: prompt=50.4 tok/s, gen=4.1 tok/s
    Rep 5: prompt=51.6 tok/s, gen=4.2 tok/s

I [pin-hot-experts] obs=48300 | locked=139854.38 MiB | moe_layers=75 | pinned=12000/12000 (global, N=160 x layers=75) | distinct (layer,expert) seen=19141 | pinned count range=[53, 657] | per-layer: {L3=218, L4=224, L5=221, L6=211, L7=210, L8=211, L9=218, L10=206, L11=207, L12=179, L13=208, L14=186, L15=158, L16=171, L17=147, L18=97, L19=111, L20=113, L21=128, L22=141, L23=190, L24=189, L25=178, L26=179, L27=182, L28=158, L29=155, L30=154, L31=144, L32=143, L33=144, L34=143, L35=145, L36=136, L37=140, L38=147, L39=155, L40=159, L41=160, L42=157, L43=146, L44=166, L45=147, L46=172, L47=165, L48=163, L49=163, L50=154, L51=143, L52=153, L53=149, L54=140, L55=143, L56=130, L57=145, L58=139, L59=156, L60=133, L61=149, L62=160, L63=151, L64=153, L65=170, L66=145, L67=155, L68=163, L69=169, L70=153, L71=151, L72=150, L73=173, L74=175, L75=164, L76=166, L77=23}

pinned count range=[53, 657] — all hot experts used more than 53 times are mlocked, cold experts used less are just mmapped and read from disk on cache miss. Hottest expert was used 657 times.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES — AI assistance was used during code development. I can explain every line of code in this PR.

Problem: Large MoE models loaded via mmap suffer from OS page cache eviction of infrequently accessed experts, causing severe latency spikes when those experts are needed later. Solution: --pin-hot-experts N dynamically tracks expert usage at runtime and uses mlock() to pin the top-N most frequently used experts in RAM, preventing OS eviction. Cold experts remain paged normally. Dense parts are mlocked automatically: Before any hot experts are pinned, all dense (non-MoE-expert) tensors that reside in host memory are mlocked in place. Dense parts (embeddings, attention/FFN weights, RMSNorm, output projection, router weights, etc.) are used on every single token — they are the hottest data by definition. This happens unconditionally when --pin-hot-experts is enabled and mlock is supported, consuming the global budget first so that hot experts only ever get the leftover budget. Key features: - Real-time tracking via tensor callback on ffn_moe_topk tensors - On-the-fly eviction/replacement of cold experts with hot ones - Budget cap via --pin-hot-experts-budget-mib - Fail-safe rollback on mlock failure - Per-layer stats reporting at --pin-hot-experts-stats-interval N - New load mode --load-mode mmap+pin (mmap without global mlock) CLI arguments: --pin-hot-experts <n> Number of hot experts to pin (0=off) --pin-hot-experts-budget-mib <n> Max pinned memory in MiB (0=unlimited) --pin-hot-experts-stats-interval N Print stats every N tokens --load-mode mmap+pin mmap without global mlock Naming convention: C API: underscores (n_pin_hot_experts) CLI: hyphens (--pin-hot-experts)

ggml-gh-bot · 2026-08-01T20:42:28Z

Hi @Ghimli, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.
AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

Ghimli · 2026-08-01T20:51:25Z

@ggml-gh-bot Resolved. Please mark it with a "feature" label dear Bot :).

Green-Sky · 2026-08-01T20:54:34Z

Quoting myself from the previous pr (why new pr??):

i wonder if it should not be called "hot" instead of "pin", since pin == lock .

So something like mmap+hot or mmap+lockhot

Ghimli · 2026-08-01T20:58:39Z

@Green-Sky Changed the name of underlying and previous PR closed itself. I couldn't reopen.
It is mmap+lockhot as it is memory locking MOE experts used the most preventing its eviction and reloading from disk. I'm using it frequently to speed up my "overnight" large models.

PmNz8 · 2026-08-01T21:39:38Z

Does this change influence experts in CPU RAM only?

I have 24 GB GPU and lets say that dense + full context takes 14 GB so I have 10 GB VRAM left. I could just use -cmoe to have all experts on CPU RAM + SSD (if experts size > RAM) or get manually a few experts into VRAM with -ncmoe.

If there would be a way to have the hot experts first loaded into remaining VRAM then RAM it would be great.

Also, if you could provide explanation how this change would interact wit things like -ngl or -fit options I would appreciate.

Ghimli · 2026-08-01T22:04:11Z

@PmNz8 Yes RAM only. I already tried several times to do that, but each time i only got lower token processing speed. With RAM, weights are already there, just need protective malloc. With VRAM->RAM you need to transfer a lot of data when expert become hot or cold and it it just too expensive to do. Which expert is hot is dynamic and task dependent. As of second it just optimizes experts which are in RAM and you can't mlock them all.

Green-Sky · 2026-08-02T13:11:29Z

It does not seem to play nice with no-warmup

xashr · 2026-08-02T19:34:26Z

At least on my system (Intel 270K + 5090) mmap is quite slow, even if the whole model fits into RAM and is mlocked. So an approach avoiding mmap would be more promising (#25294 ? or https://github.com/Lidenburg/llama.cpp)

Midaychi · 2026-08-04T00:24:50Z

Windows 11, gemma-4-26b-a4b, tried a bunch of different configurations of this, but it seemed to use the same amount of system ram no matter what I did and spammed 'failed to VirtualUnlock buffer: The segment is already unlocked.'
Maybe I'm using it wrong?
I mean I had plenty of ram to load all the experts I was just trying to test if it had an actual effect

Ghimli · 2026-08-04T08:44:20Z

It seems memory locking is failing silently somehow. I'm on vacation right now. I will fix that when I come back. wt., 4 sie 2026, 02:25 użytkownik Midaychi ***@***.***> napisał:

…

*Midaychi* left a comment (ggml-org/llama.cpp#26414) <#26414 (comment)> Windows 11, gemma-4-26b-a4b, tried a bunch of different configurations of this, but it seemed to use the same amount of system ram no matter what I did and spammed 'failed to VirtualUnlock buffer: The segment is already unlocked.' Maybe I'm using it wrong? I mean I had plenty of ram to load all the experts I was just trying to test if it had an actual effect — Reply to this email directly, view it on GitHub <#26414?email_source=notifications&email_token=ABU4MNXNKI6J6QYRAFSGMA35IEUOTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJXGMYTMNJVGUY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5173165551>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU4MNUXTWSHTODSHMFLFYT5IEUOTAVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TKMBTHA4TEMZXGYY2C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ABU4MNSCKTSEB5MEU5APMZT5IEUOTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJXGMYTMNJVGUY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ABU4MNQTFTNRN246QWTQIM35IEUOTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJXGMYTMNJVGUY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

xashr · 2026-08-09T12:22:15Z

I gave it a try today - in combination with #24524 . I ran DeepSeek 0731 Q4_K_XL on 32 GB VRAM + 128 GB RAM on Ubuntu 26.04.

Issues observed:

Running with mlock requires changing the system config /etc/security/limits.conf as the mlock max default is 8 MB. So it won't work out of the box, unfortunately.
There is/was a bug in llama-model-loader.cpp where load_mode == LLAMA_LOAD_MODE_MMAP_PIN is missing in this->use_mmap = load_mode == LLAMA_LOAD_MODE_MMAP || load_mode == LLAMA_LOAD_MODE_MMAP_MLOCK || load_mode == LLAMA_LOAD_MODE_MMAP_PIN; => In my case this resulted in llama.cpp trying to put the whole model into CUDA memory. Had to fix that.
After that it was working, but there was not real performance improvement ( I was hoping to see some improvements for PP)

Ghimli · 2026-08-09T12:36:48Z

You should get some improvements in pp. Sometimes when model is almost fitting just mmap is enough and there are no disk io operation. Observe iotop when using vanilla if no IO or almost none you are as good as full mlock and your limits are RAM speed and cpu matmul operations. The same when getting almost 100 proc CPU usage that means even nvme speed in enough and you are just CPU limited. niedz., 9 sie 2026, 14:22 użytkownik xashr ***@***.***> napisał:

…

*xashr* left a comment (ggml-org/llama.cpp#26414) <#26414 (comment)> I gave it a try today - in combination with #24524 <#24524> . I ran DeepSeek 0731 Q4_K_XL on 32 GB VRAM + 128 GB RAM on Ubuntu 26.04. Issues observed: - Running with mlock requires changing the system config /etc/security/limits.conf as the mlock max default is 8 MB. So it won't work out of the box, unfortunately. - There is/was a bug in llama-model-loader.cpp where load_mode == LLAMA_LOAD_MODE_MMAP_PIN is missing in this->use_mmap = load_mode == LLAMA_LOAD_MODE_MMAP || load_mode == LLAMA_LOAD_MODE_MMAP_MLOCK || load_mode == LLAMA_LOAD_MODE_MMAP_PIN; => In my case this resulted in llama.cpp trying to put the whole model into CUDA memory. Had to fix that. - After that it was working, but there was not real performance improvement ( I was hoping to see some improvements for PP) — Reply to this email directly, view it on GitHub <#26414?email_source=notifications&email_token=ABU4MNTMONHND7Z7Y2WGR6L5JBUI7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRTGE2DSNZQGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5231497020>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABU4MNU33US7XY4DVUX7IGL5JBUI7AVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TKMBTHA4TEMZXGYY2C5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ABU4MNSJH4F47WFFQUCOVCD5JBUI7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRTGE2DSNZQGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ABU4MNXQL6S6ZWAQAB4NFSL5JBUI7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRTGE2DSNZQGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>

Ghimli · 2026-08-09T12:46:57Z

I also noticed typical benchmarks are very poor to measure this kind of improvements. They tend to have almost static heatmap of MOE experts. Even VRAM->RAM hot experts swapping looks good on benchmarks. While real agentic operations are actually slower due to experts heatmaps being more dynamic. niedz., 9 sie 2026, 14:36 użytkownik Damian O ***@***.***> napisał:

…

You should get some improvements in pp. Sometimes when model is almost fitting just mmap is enough and there are no disk io operation. Observe iotop when using vanilla if no IO or almost none you are as good as full mlock and your limits are RAM speed and cpu matmul operations. The same when getting almost 100 proc CPU usage that means even nvme speed in enough and you are just CPU limited. niedz., 9 sie 2026, 14:22 użytkownik xashr ***@***.***> napisał: > *xashr* left a comment (ggml-org/llama.cpp#26414) > <#26414 (comment)> > > I gave it a try today - in combination with #24524 > <#24524> . I ran DeepSeek 0731 > Q4_K_XL on 32 GB VRAM + 128 GB RAM on Ubuntu 26.04. > > Issues observed: > > - Running with mlock requires changing the system config > /etc/security/limits.conf as the mlock max default is 8 MB. So it > won't work out of the box, unfortunately. > - There is/was a bug in llama-model-loader.cpp where load_mode == > LLAMA_LOAD_MODE_MMAP_PIN is missing in this->use_mmap = load_mode == > LLAMA_LOAD_MODE_MMAP || load_mode == LLAMA_LOAD_MODE_MMAP_MLOCK || > load_mode == LLAMA_LOAD_MODE_MMAP_PIN; => In my case this resulted in > llama.cpp trying to put the whole model into CUDA memory. Had to fix that. > - After that it was working, but there was not real performance > improvement ( I was hoping to see some improvements for PP) > > — > Reply to this email directly, view it on GitHub > <#26414?email_source=notifications&email_token=ABU4MNTMONHND7Z7Y2WGR6L5JBUI7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRTGE2DSNZQGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5231497020>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABU4MNU33US7XY4DVUX7IGL5JBUI7AVCNFSNUABFKJSXA33TNF2G64TZHM3DCMRTGU2DOOBUHNEXG43VMU5TKMBTHA4TEMZXGYY2C5QC> > . > Triage notifications, keep track of coding agent tasks and review pull > requests on the go with GitHub Mobile for iOS > <https://github.com/notifications/mobile/ios/ABU4MNSJH4F47WFFQUCOVCD5JBUI7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRTGE2DSNZQGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> > and Android > <https://github.com/notifications/mobile/android/ABU4MNXQL6S6ZWAQAB4NFSL5JBUI7A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMRTGE2DSNZQGIYKM4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. > Download it today! > You are receiving this because you were mentioned.Message ID: > ***@***.***> >

Ghimli · 2026-08-09T14:12:46Z

@xashr And you can ramp up pp alone almost indefinitely by just playing with --batch-size --ubatch-size settings, especially ubatch at the expense of VRAM/RAM, as prefill uses whole batch size of token at each pass, decode almost always just single token. Just in case you did't know that :).

Ghimli requested review from a team and ggerganov as code owners August 1, 2026 20:38

Ghimli mentioned this pull request Aug 1, 2026

llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts #25932

Closed

yalun753 mentioned this pull request Aug 2, 2026

Feature request: run MoE expert weights from host RAM via PCIe DMA (no H2D copy) — 23GB MoE on 1.6GB VRAM, 81GB Qwen3-235B on 8GB cards #26448

Open

Uh oh!

Conversation

Ghimli commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

CLI arguments

Key design points

Example

Additional information

Files changed

Hardware / testing environment

Baseline test (no pinning, plain mmap):

Hot experts pinned test (--load-mode mmap+pin --pin-hot-experts 160):

Related

Requirements

Uh oh!

ggml-gh-bot Bot commented Aug 1, 2026

Uh oh!

Ghimli commented Aug 1, 2026

Uh oh!

Green-Sky commented Aug 1, 2026

Uh oh!

Ghimli commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PmNz8 commented Aug 1, 2026

Uh oh!

Ghimli commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Green-Sky commented Aug 2, 2026

Uh oh!

xashr commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Midaychi commented Aug 4, 2026

Uh oh!

Ghimli commented Aug 4, 2026 via email

Uh oh!

xashr commented Aug 9, 2026

Uh oh!

Ghimli commented Aug 9, 2026 via email

Uh oh!

Ghimli commented Aug 9, 2026 via email

Uh oh!

Ghimli commented Aug 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Ghimli commented Aug 1, 2026 •

edited

Loading

Hot experts pinned test (`--load-mode mmap+pin --pin-hot-experts 160`):

Ghimli commented Aug 1, 2026 •

edited

Loading

Ghimli commented Aug 1, 2026 •

edited

Loading

xashr commented Aug 2, 2026 •

edited

Loading