Backend sampling multi output by gaugarg-nv · Pull Request #25532 · ggml-org/llama.cpp

gaugarg-nv · 2026-07-10T16:35:59Z

Overview

This PR extends backend sampling from one output per sequence to multiple outputs, enabling backend sampling during speculative decoding.

Add n_sampling_outputs_per_seq_max to context parameters, separating the total output budget from the per-sequence backend-sampling limit.
Build one backend-sampling graph row per output rather than per sequence.
Extend graph reservation and node budgeting for replicated sampler graphs.
Add graph probing for both backend operation support and sampler node counting.
Add a backend_reset hook to clear graph-owned tensor references before graph reconstruction.
Make distribution sampling deterministic between CPU and backend paths - use a temporary RNG for multi-output sampling so rejected tokens do not advance rng
Clamp the distribution mask sum before converting it into a sampled index, preventing an out-of-range access. (Copied from sampling : support multiple outputs per sequence #19833)

Additional information

Performance: I see a perf improvement of ~8% on RTX 5090 with the Qwen-3.6-35B Q4_K_M model. Acceptance ratio with both backend and CPU sampling is exactly same.

CPU sampling: llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --spec-type draft-mtp --seed 42

python3 mtp-bench.py
  code_python        pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=379.5
  code_cpp           pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=343.1
  explain_concept    pred= 192 draft= 210 acc= 119 rate=0.567 tok/s=308.9
  summarize          pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=380.3
  qa_factual         pred= 192 draft= 170 acc= 134 rate=0.788 tok/s=382.0
  translation        pred= 192 draft= 176 acc= 132 rate=0.750 tok/s=370.2
  creative_short     pred= 192 draft= 207 acc= 120 rate=0.580 tok/s=315.8
  stepwise_math      pred= 192 draft= 160 acc= 137 rate=0.856 tok/s=401.1
  long_code_review   pred= 192 draft= 168 acc= 134 rate=0.798 tok/s=379.2

Backend sampling: llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --spec-type draft-mtp --seed 42 -bs

python3 mtp-bench.py
  code_python        pred= 192 draft= 168 acc= 135 rate=0.804 tok/s=411.4
  code_cpp           pred= 192 draft= 190 acc= 127 rate=0.668 tok/s=369.2
  explain_concept    pred= 192 draft= 210 acc= 119 rate=0.567 tok/s=330.2
  summarize          pred= 192 draft= 169 acc= 134 rate=0.793 tok/s=412.3
  qa_factual         pred= 192 draft= 170 acc= 134 rate=0.788 tok/s=414.4
  translation        pred= 192 draft= 176 acc= 132 rate=0.750 tok/s=400.4
  creative_short     pred= 192 draft= 207 acc= 120 rate=0.580 tok/s=337.8
  stepwise_math      pred= 192 draft= 160 acc= 137 rate=0.856 tok/s=437.1
  long_code_review   pred= 192 draft= 168 acc= 134 rate=0.798 tok/s=410.2

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, paired with Codex.

gaugarg-nv · 2026-07-17T11:13:47Z

Backend sampling shows ~12% improvement on Windows + RTX 5090:

CPU Sampling: llama-server.exe -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --spec-type draft-mtp --seed 42

python mtp-bench.py
  code_python        pred= 192 draft= 165 acc= 136 rate=0.824 tok/s=327.7
  code_cpp           pred= 192 draft= 196 acc= 124 rate=0.633 tok/s=270.6
  explain_concept    pred= 192 draft= 198 acc= 123 rate=0.621 tok/s=263.9
  summarize          pred= 192 draft= 176 acc= 131 rate=0.744 tok/s=298.4
  qa_factual         pred= 192 draft= 170 acc= 134 rate=0.788 tok/s=315.8
  translation        pred= 192 draft= 176 acc= 131 rate=0.744 tok/s=299.4
  creative_short     pred= 192 draft= 206 acc= 121 rate=0.587 tok/s=261.3
  stepwise_math      pred= 192 draft= 160 acc= 137 rate=0.856 tok/s=332.8
  long_code_review   pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=266.6

Backend sampling: llama-server.exe -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --spec-type draft-mtp --seed 42 -bs

python mtp-bench.py
  code_python        pred= 192 draft= 165 acc= 136 rate=0.824 tok/s=370.2
  code_cpp           pred= 192 draft= 196 acc= 124 rate=0.633 tok/s=301.3
  explain_concept    pred= 192 draft= 195 acc= 124 rate=0.636 tok/s=294.6
  summarize          pred= 192 draft= 173 acc= 132 rate=0.763 tok/s=336.3
  qa_factual         pred= 192 draft= 173 acc= 133 rate=0.769 tok/s=347.3
  translation        pred= 192 draft= 176 acc= 131 rate=0.744 tok/s=332.3
  creative_short     pred= 192 draft= 206 acc= 121 rate=0.587 tok/s=289.2
  stepwise_math      pred= 192 draft= 160 acc= 137 rate=0.856 tok/s=368.9
  long_code_review   pred= 192 draft= 195 acc= 125 rate=0.641 tok/s=293.4

otacon6530 · 2026-08-04T00:23:11Z

Backend sampling shows ~4% improvement on Linux + Tesla P40 (sm_61, Pascal):

CPU Sampling: llama-server -m Qwen3.6-35B-A3B-UD-IQ4_NL.gguf --spec-type draft-mtp --seed 42

python3 mtp-bench.py
  code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=73.1
  code_cpp           pred= 113 draft=  76 acc=  74 rate=0.974 tok/s=75.9
  explain_concept    pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=62.4
  summarize          pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=59.6
  qa_factual         pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=62.4
  translation        pred= 119 draft=  92 acc=  73 rate=0.793 tok/s=67.0
  creative_short     pred= 192 draft= 197 acc=  92 rate=0.467 tok/s=50.7
  stepwise_math      pred= 192 draft= 133 acc= 124 rate=0.932 tok/s=73.6
  long_code_review   pred= 192 draft= 155 acc= 113 rate=0.729 tok/s=63.8

Backend sampling: llama-server -m Qwen3.6-35B-A3B-UD-IQ4_NL.gguf --spec-type draft-mtp --seed 42 -bs

python3 mtp-bench.py
  code_python        pred= 192 draft= 132 acc= 124 rate=0.939 tok/s=76.2
  code_cpp           pred= 113 draft=  76 acc=  74 rate=0.974 tok/s=79.4
  explain_concept    pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=64.6
  summarize          pred= 192 draft= 167 acc= 107 rate=0.641 tok/s=61.6
  qa_factual         pred= 192 draft= 159 acc= 111 rate=0.698 tok/s=64.6
  translation        pred= 119 draft=  92 acc=  73 rate=0.793 tok/s=69.6
  creative_short     pred= 192 draft= 197 acc=  92 rate=0.467 tok/s=52.1
  stepwise_math      pred= 192 draft= 133 acc= 124 rate=0.932 tok/s=76.6
  long_code_review   pred= 192 draft= 155 acc= 113 rate=0.729 tok/s=65.7

Acceptance ratio with both backend and CPU sampling is exactly same. The improvement is smaller than on RTX 5090 (4% vs 12%), which is expected — the P40 is memory-bandwidth-bound (sm_61, 580 GB/s vs RTX 5090's 1,792 GB/s), so the CPU↔GPU logits round-trip is a smaller fraction of total decode time. However, still the largest improvement in tok/s I have seen in a while. (~+2 t/s).

anvme · 2026-08-04T16:19:23Z

Any explanation for docs/ ?

Pento95 · 2026-08-04T20:08:55Z

Trying to reproduce the claimed speedup, finding some different results.

Setup:

GPU: RTX 3090 Ti (PCIe 4.0 x16)
Model: Gemma 4 31B QAT Q4_K_XL + MTP Draft Q4_0
Flags: --spec-draft-n-max 3 --seed 42 -np 1 --no-cache-prompt --cache-reuse 0`
Tool: mtp-bench.py (9 prompts, 192 max_tokens)
llama-server warmed-up before running script.

MTP-only, controlled test (temp=0.01, top-k=64, top-p=0.95).

(Avoided temp=0 since it reduces the entire sampling pipeline to a single argmax, which wouldn't test the backend properly).

Full sampling pipeline, 8/9 prompts produce identical sequences.

	Wall time	tok/s	Accept rate
Baseline	25.34s	68.19	74.00%
Patched `-bs`	25.53s	67.69	74.20%
Delta	+0.19s	-0.50	-

Small overhead (~0.75%) on single-sequence workloads. weird.

MTP-only, default temperature

8/9 prompts produce different token sequences between baseline and patched.

	Wall time	tok/s	Accept rate
Baseline	26.81s	64.45	68.67%
Patched `-bs`	26.00s	66.46	71.08%
Delta	-0.81s	+2.01	+2.41%

Appears faster, but throughput difference is confounded by different speculative paths (different text generated).

MTP + ngram-mod, default temperature

	Wall time	tok/s	Accept rate
Baseline	16.52s	104.60	72.07%
Patched `-bs`	19.45s	88.84	67.02%

Sequences diverge heavily. ngram-mod amplifies any initial divergence since the lookup table depends on previously generated tokens - small differences cascade into completely different speculative paths.

MTP + ngram-mod, controlled test (temp=0.01)

	Wall time	tok/s	Accept rate
Baseline	13.19s	131.01	75.29%
Patched `-bs`	11.10s	155.68	80.29%

Still divergent on 3/9 prompts. Where sequences match, patched is slightly slower (same pattern as MTP-only). The overall delta is driven by divergent prompts hitting different ngram-mod cache paths.

How can I avoid prompt divergence for proper apples-to-apples benchmarking without resorting to temp=0? Since the seed is fixed, I assume the divergence at temp=0.01 is caused by floating-point arithmetic differences (e.g. softmax/cumsum) between CPU and GPU architectures. Is there a recommended way to enforce exact deterministic sampling across both paths?

AmerM137 · 2026-08-04T22:48:13Z

Here is what I see on my Linux (CachyOS) machine + 5090. I could not find the mtp-bench.py script so I had Claude reverse-engineer it.

CPU sampling: llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --spec-type draft-mtp --seed 42

Backend sampling: llama-server.exe -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --spec-type draft-mtp --seed 42 -bs

prompt	rate (cpu/bs)	cpu tok/s	-bs tok/s	delta
code_python	0.862/0.885	407.9	459.8	+12.7%
code_cpp	0.903/0.903	420.7	458.4	+9.0%
explain_concept	0.539/0.539	309.8	331.8	+7.1%
summarize	0.688/0.688	356.3	384.3	+7.9%
qa_factual	0.668/0.668	350.2	376.4	+7.5%
translation	0.705/0.705	358.6	386.1	+7.7%
creative_short	0.393/0.393	262.4	279.1	+6.4%
stepwise_math	0.840/0.840	404.3	440.4	+8.9%
long_code_review	0.808/0.808	393.3	429.7	+9.3%
mean		362.6	394.0	+8.7%

rankaiyx · 2026-08-05T02:42:16Z

Great improvements! TG increased by about 10%.
However, there seem to be some outdated logs; I enabled tensor parallelism, and while the logs indicated the use of the CPU backend, I actually observed a significant performance boost.

Test environment: 4xP100
Test result: TG: 50 -> 55

0.00.066.293 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the -lv N CLI arg)
0.00.589.030 W srv llama_server: -----------------
0.00.589.034 W srv llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.589.034 W srv llama_server: this can be a security risk (cross-origin attacks)
0.00.589.034 W srv llama_server: more info: #25655
0.00.589.035 W srv llama_server: -----------------
0.00.590.492 I srv load_model: loading model '/var/model/gguf/Qwen3.6-27B/Qwen3.6-27B-Q4_0_bartowski_8c66.gguf'

0.10.283.512 W set_sampler: backend sampling not supported with SPLIT_MODE_TENSOR; using CPU

0.11.758.080 I common_speculative_init_result: creating MTP draft context against the target model '/var/model/gguf/Qwen3.6-27B/Qwen3.6-27B-Q4_0_bartowski_8c66.gguf'
0.13.473.411 I srv load_model: loaded multimodal model, '/var/model/gguf/Qwen3.6-27B/Qwen3.6-27B-mmproj-F16_eacf.gguf'
0.13.839.258 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'

0.13.839.650 W spec common_specu: backend offload failed for seq_id=0; using CPU sampler
0.13.839.656 W spec common_specu: backend offload failed for seq_id=1; using CPU sampler
0.13.839.657 W spec common_specu: backend offload failed for seq_id=2; using CPU sampler
0.13.839.673 W spec common_specu: backend offload failed for seq_id=3; using CPU sampler

0.13.959.036 I srv init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.13.959.101 I srv llama_server: model loaded
0.13.959.108 I srv llama_server: listening on http://0.0.0.0:8000
0.22.440.099 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1
0.22.440.789 W common_sampler_init: backend sampling is not compatible with grammar, disabling
0.22.440.806 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0
0.27.862.117 I slot print_timing: id 3 | task 0 | n_decoded = 169, tg = 55.55 t/s, tg_3s = 55.54 t/s
0.30.874.548 I slot print_timing: id 3 | task 0 | n_decoded = 338, tg = 55.82 t/s, tg_3s = 56.10 t/s
0.33.895.179 I slot print_timing: id 3 | task 0 | n_decoded = 507, tg = 55.86 t/s, tg_3s = 55.95 t/s
0.36.912.996 I slot print_timing: id 3 | task 0 | n_decoded = 685, tg = 56.64 t/s, tg_3s = 58.98 t/s
0.38.115.884 I slot print_timing: id 3 | task 0 | prompt eval time = 2375.61 ms / 705 tokens ( 3.37 ms per token, 296.77 tokens per second)
0.38.115.888 I slot print_timing: id 3 | task 0 | eval time = 13296.25 ms / 756 tokens ( 17.59 ms per token, 56.86 tokens per second)
0.38.115.889 I slot print_timing: id 3 | task 0 | total time = 15671.86 ms / 1461 tokens
0.38.115.909 I slot print_timing: id 3 | task 0 | graphs reused = 270
0.38.115.913 I slot print_timing: id 3 | task 0 | draft acceptance = 0.87956 ( 482 accepted / 548 generated), mean len = 2.76
0.38.116.059 I slot release: id 3 | task 0 | stop processing: n_tokens = 1461, truncated = 0
0.38.272.108 I slot get_availabl: id 3 | task -1 | selected slot by LCP similarity, f_sim_best = 0.969 (> 0.100 thold), f_keep = 1.000

ggerganov

I think we can avoid the slot.backend_sampling flag and the related changes by adding an API llama_sampler_copy to llama.h like so:

void llama_sampler_copy(const struct llama_sampler * src, struct llama_sampler * dst) {
    if (!src || !dst) {
        return;
    }

    GGML_ASSERT(src->iface == dst->iface && "llama_sampler_copy: cannot copy between different sampler types");

    // build a temporary sampler carrying src's current state
    llama_sampler * tmp = llama_sampler_clone(src);

    // free dst's old state (frees dst->ctx, including children for a chain)
    if (dst->iface->free) {
        dst->iface->free(dst);
    }

    // transplant tmp's state into dst, then destroy the (now empty) temp shell
    dst->ctx = tmp->ctx;
    tmp->ctx = nullptr;
    delete tmp;
}

This is similar to llama_sampler_clone, but the idea is to keep the pointer intact and just copy the internal context.

Unless I missed something, you should be able to replace this logic in server-context.cpp:

slot.prompt.tokens.keep_first(ckpt.n_tokens);
slot.smpl = std::move(smpl_save);

if (slot.backend_sampling) {
    slot.backend_sampling = llama_set_sampler(
    slot.ctx_tgt, slot.id, common_sampler_get(slot.smpl.get()));
}

with simple unconditional:

slot.prompt.tokens.keep_first(ckpt.n_tokens);
llama_sampler_copy(smpl_save.get(), slot.smpl.get());

This will avoid re-setting the sampler of the llama_context (avoiding sched reserve) and avoiding the extra slot.backend_sampling tracking in the server logic.

LMK if this makes sense.

gaugarg-nv · 2026-08-05T18:57:02Z

I think we can avoid the slot.backend_sampling flag and the related changes by adding an API llama_sampler_copy to llama.h like so:

This will avoid re-setting the sampler of the llama_context (avoiding sched reserve) and avoiding the extra slot.backend_sampling tracking in the server logic.

LMK if this makes sense.

This makes sense as it can help avoid expensive sched reserves. However, the change is going to be more complicated. A simple clone-and-swap won't be enough. We will have to copy the backend sampler state and child samplers explicitly.

I have attempted this change in the latest commit along the same lines. Please take a look.

…uence

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

…ped PR The declaration alone would have been orphaned: its implementation depends on a llama_sampler_i::copy_state interface field that is part of upstream PR ggml-org#25532 ('multi-output backend sampling', 24 files, ~1000 lines) - a squashed feature that predates this fork's real base and is out of scope for a 'zero TQ3 impact' sync. copy_state does not exist anywhere in this tree; a half-port would not compile. Nothing in this codebase currently calls llama_sampler_copy, so dropping the declaration is safe. The full feature belongs in the larger upstream content-catch-up, tracked separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

28 upstream commits since 687e778. One touches ggml-opencl: 689e227 (ggml-org#26428, FA prefill K-tile transpose) -- our own PR, merged upstream this window, so its four files conflicted against the x2ue originals they were carved from. Resolved to ours in all four: x2ue is a strict superset (FA_Q_HALF / FA_O_HALF, the FA*_PROBE_NO_LDS diagnostics and FA_V_LDS_T all landed on top). Verified line-by-line that no ggml-org#26428 content was lost -- the seven upstream lines absent from the merged tree are the same code reshaped by our later commits (#elif rather than #if because our probe branch precedes, mad() rather than *, Q_PRIV_TO_ACC4() around q_priv), all still reading through FA_LK/FA_LK_PAIR, plus one reworded comment. Also verified the automerge kept upstream intact: of the 659 files upstream touched, the merged tree is byte-identical to upstream/master on all but the eight we also modified, and all 247 upstream-added lines in the four auto-merged files (clip.cpp, clip-model.h, llama-context.cpp, test-backend-ops.cpp -- Muse Glimmer ggml-org#26841, multi-output sampling ggml-org#25532, Granite-Switch ggml-org#25107) are present.

…, multi-output backend sampling, pocket-tts Merges 45 upstream commits (0865990..ebb546b, b10362-16-gebb546b7e) into synori/llama-update-mtp-fit. Zero conflicts; all vendored patches carried over untouched. Primary motivation — new Meta architecture: * 62bf73d model: Muse Glimmer Support (ggml-org#26841) LLM_ARCH_MUSE_GLIMMER + src/models/muse-glimmer.cpp + the mtmd vision tower in tools/mtmd/models/muse-glimmer.cpp and conversion/muse_glimmer.py. Other notable changes that touch our public API surface: * dd1ea52 llama : support multi-output backend sampling (ggml-org#25532) llama_context_params gains n_outputs_max_per_seq; llama_sampler_i.backend_init takes it as a third argument; new backend_reset / copy_state vtable slots and llama_sampler_copy(). * 153d324 llama : default load-mode auto, avoids mmap on iGPUs (ggml-org#26081) llama_load_mode gains LLAMA_LOAD_MODE_AUTO = -1 (enum is now signed). * 6e62ba5 mtmd: support pocket-tts (ggml-org#26871) mtmd_gen_inp/mtmd_gen_out gain seed/temp/feats/is_eos; new mtmd_gen_inp_default(); mtmd_helper_gen_audio_step_gen() takes out_stop. * 157b81f model : Granite-Switch Architecture (ggml-org#25107) * 7a20b41 model: MTP support for Nemotron (ggml-org#26725) and cc078b4 Dflash support for nemotron-3.5 (ggml-org#26905) * e23e944 vendor : cpp-httplib 0.53.0, 4c6766f vendor : subprocess.h sync Vendored patches preserved: * 919fde3 feat(rpc): thread-local last_error accessor — intact, upstream touched ggml-rpc.cpp by one unrelated line. * 3679b23 Fixes (RPC) — intact. * 2aa76c7 fix(metal): drop stray kernel_pad_f32 — still applies; upstream has since refactored pad into a templated kernel_pad_impl<T>, so the duplicate definition that referenced the nonexistent kargs_pad.s0..s3 is gone on both sides and nothing had to be re-applied.

* Enable backend sampling with token speculation * Clamp the mask sum before converting it into the sampled index * Add a numeric context parameter declaring the maximum outputs one sequence * More fixes * Don't reuse memory for output views. * Match dist between CPU and GPU * Fix CPU and backend sampling mismatches * Simpify some of the changes * Fix tests on Vulkan * More test fixes * Rebase changes * Rebase and address review comments * Address review comments * Address review comments * Update src/llama-sampler.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

gaugarg-nv requested review from a team, CISC, JohannesGaessler and ggerganov as code owners July 10, 2026 16:36

github-actions Bot added testing Everything test related server ggml changes relating to the ggml tensor library for machine learning labels Jul 10, 2026

gaugarg-nv force-pushed the backend-sampling-multi-output branch from f5e3f22 to 5897112 Compare July 14, 2026 14:59

ggerganov self-assigned this Jul 14, 2026

ggerganov reviewed Jul 15, 2026

View reviewed changes

Comment thread include/llama.h Outdated

gaugarg-nv force-pushed the backend-sampling-multi-output branch from a1b62dd to 466696c Compare July 15, 2026 15:16

ggerganov mentioned this pull request Jul 28, 2026

ggml : set output of view src #25729

Merged

gaugarg-nv force-pushed the backend-sampling-multi-output branch from 762a06e to 89cd866 Compare August 3, 2026 10:48

danbev mentioned this pull request Aug 3, 2026

sampling : support multiple outputs per sequence #19833

Closed

2 tasks

danbev reviewed Aug 4, 2026

View reviewed changes

Comment thread include/llama.h Outdated

gaugarg-nv force-pushed the backend-sampling-multi-output branch from 89cd866 to 4cbb71d Compare August 4, 2026 15:26

danbev reviewed Aug 5, 2026

View reviewed changes

Comment thread src/llama-sampler.cpp Outdated

ggerganov reviewed Aug 5, 2026

View reviewed changes

Comment thread include/llama.h Outdated

ggerganov reviewed Aug 5, 2026

View reviewed changes

Comment thread tools/server/server-context.cpp

ggerganov reviewed Aug 5, 2026

View reviewed changes

Comment thread tools/server/server-context.cpp Outdated

gaugarg-nv force-pushed the backend-sampling-multi-output branch from 4cbb71d to 8d16326 Compare August 5, 2026 18:55

github-actions Bot added the documentation Improvements or additions to documentation label Aug 5, 2026

gaugarg-nv added 14 commits August 10, 2026 18:24

Enable backend sampling with token speculation

fc4fc9b

Clamp the mask sum before converting it into the sampled index

c186a3f

Add a numeric context parameter declaring the maximum outputs one seq…

62dedf4

…uence

More fixes

dbcec66

Don't reuse memory for output views.

aaa886f

Match dist between CPU and GPU

37d606e

Fix CPU and backend sampling mismatches

4dbed2e

Simpify some of the changes

eecc657

Fix tests on Vulkan

3b11240

More test fixes

a5663e1

Rebase changes

9e87478

Rebase and address review comments

1826901

Address review comments

d4c699c

Address review comments

9f5d059

gaugarg-nv force-pushed the backend-sampling-multi-output branch from 9fcc911 to 9f5d059 Compare August 10, 2026 12:57

ggerganov reviewed Aug 10, 2026

View reviewed changes

Comment thread src/llama-sampler.cpp

Update src/llama-sampler.cpp

0ec96eb

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

ggerganov merged commit dd1ea52 into ggml-org:master Aug 10, 2026
26 of 30 checks passed

gaugarg-nv deleted the backend-sampling-multi-output branch August 10, 2026 15:48

gaugarg-nv mentioned this pull request Aug 10, 2026

Address review comment of PR 25532 #26852

Merged

turbo-tan mentioned this pull request Aug 10, 2026

sync: merge upstream/master (3 commits, zero TQ3 impact) turbo-tan/llama.cpp-tq3#72

Merged

jimw567 mentioned this pull request Aug 11, 2026

Jimwu.fix backend sampler hip multi output #26878

Merged

claude Bot mentioned this pull request Aug 11, 2026

fix(container): update image ghcr.io/ggml-org/llama.cpp (server-cuda-b10335 ➔ server-cuda-b10362) drag0n141/home-ops#7223

Merged

1 task

ggerganov mentioned this pull request Aug 11, 2026

tests : update speculative params #26925

Merged

ruixiang63 mentioned this pull request Aug 12, 2026

spec: enable backend sampling for both dflash & dspark #26958

Merged

swiftjiang mentioned this pull request Aug 14, 2026

Server: backend sampling (-bs) gives +48% throughput at 32 slots (706 -> 1046 tok/s) — should it be the default for multi-slot serving? #27050

Open

Navigation Menu

Uh oh!

Backend sampling multi output - #25532

Backend sampling multi output#25532
ggerganov merged 15 commits into
ggml-org:masterfrom
gaugarg-nv:backend-sampling-multi-output

gaugarg-nv commented Jul 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

gaugarg-nv commented Jul 17, 2026

Uh oh!

otacon6530 commented Aug 4, 2026

Uh oh!

Uh oh!

anvme commented Aug 4, 2026

Uh oh!

Pento95 commented Aug 4, 2026 •

edited

Loading

Uh oh!

AmerM137 commented Aug 4, 2026 •

edited

Loading

Uh oh!

rankaiyx commented Aug 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov left a comment •

edited

Loading

Uh oh!

Uh oh!

gaugarg-nv commented Aug 5, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

gaugarg-nv commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

Uh oh!

gaugarg-nv commented Jul 17, 2026

Uh oh!

otacon6530 commented Aug 4, 2026

Uh oh!

Uh oh!

anvme commented Aug 4, 2026

Uh oh!

Pento95 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MTP-only, controlled test (temp=0.01, top-k=64, top-p=0.95).

MTP-only, default temperature

MTP + ngram-mod, default temperature

MTP + ngram-mod, controlled test (temp=0.01)

Uh oh!

AmerM137 commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rankaiyx commented Aug 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ggerganov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gaugarg-nv commented Aug 5, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

gaugarg-nv commented Jul 10, 2026 •

edited

Loading

Pento95 commented Aug 4, 2026 •

edited

Loading

AmerM137 commented Aug 4, 2026 •

edited

Loading

ggerganov left a comment •

edited

Loading