Add backend sampler for penalties sampler by kmorennv · Pull Request #25262 · ggml-org/llama.cpp

kmorennv · 2026-07-03T08:13:42Z

Overview & Motivation

This PR migrates penalties sampling (repeat, frequency, and presence) from the CPU to the GPU backend.

The Problem: Previously, penalty sampling was CPU-only, which forced all subsequent samplers in the chain to execute on the CPU as well.
The Solution: Moving this to the backend allows for continuous GPU-bound sampling. This is highly recommended for modern models like Qwen 3.5/3.6. They directly advice to use penalties sampling https://huggingface.co/Qwen/Qwen3.5-35B-A3B

Performance Impact

Moving penalty sampling to the backend yields a noticeable boost in token generation speed:

OS / GPU	Max Speedup	Key Models Benefiting
Linux (RTX 4000 SFF)	+7.59%	`gpt-oss-20b`, `Qwen3.6-35B`
Windows 11 (RTX 6000 Pro)	+19.40%	`gpt-oss-20b`, `Qwen3.6-35B`

Details performance

OS-CTK-Driver	GPU	Model	Without BS tok/sec	With BS tok/sec	Speedup
Linux-13.2-595.58.03	RTX 4000 SFF	gpt-oss-20b-mxfp4	81.01	87.16	+7.59%
Linux-13.2-595.58.03	RTX 4000 SFF	Qwen3.6-27B UD-Q4_K_XL	13.62	13.84	+1.63%
Linux-13.2-595.58.03	RTX 4000 SFF	Qwen3.6-35B-A3B UD-Q4_K_M	57.87	61.22	+5.80%
Win11-13.2-596.36	RTX 6000 Pro	gpt-oss-20b-mxfp4	305.73	365.06	+19.40%
Win11-13.2-596.36	RTX 6000 Pro	Qwen3.6-27B Q4_K_M	71.84	75.34	+4.86%
Win11-13.2-596.36	RTX 6000 Pro	Qwen3.6-35B-A3B Q4_K_M	237.70	275.66	+15.97%

CTK - CudaToolKit
BS- backend-sampling

Command used to run benchmark:

./build/bin/llama-server -m /gguf/gpt-oss-20b-mxfp4.gguf --temp 1.0 --top-k 0 --top-p 1.0 --min-p 0.0 -dio --port 8033 -np 1 -b 4096 -ub 4096 --repeat-penalty 1.1 --presence-penalty 0 -bs

Core Implementation Steps

1. Backend Integration & Fallback

llama_sampler_penalties now inherits from llama_sampler_backend.
Includes a capability check during initialization. If the selected backend lacks required ggml operations, it falls back to the CPU to preserve compatibility.

2. State Management & Clone Fix

Token history (prev ring buffer and token_count) is still maintained on the CPU via the normal accept() call, but the actual logit transformation is offloaded to the GPU.
Bug Fix: Fixed state cloning by ensuring both the ring buffer and token_count are copied. Previously, an empty cloned token count delayed penalty applications.

3. Sparse Graph Optimization

Instead of modifying the entire vocabulary tensor, the backend graph uses a sparse approach. It only gathers, transforms, and scatters logits for the tokens present in the recent history window, leaving the rest untouched.

4. Support After Top-K / Top-P / Min-P

Initially, the backend assumed full-vocabulary logits. The implementation now detects if a prior sampler reduced the vocabulary (like Top-K).

5. Testing

Added comprehensive backend-vs-CPU verification tests covering combined penalties, repeated prompt tokens, and a larger penalty window (80 tokens).

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, paired with codex.

- Set default value for penalty_last_n based on model context if not specified. - Ensure penalty_last_n and n_prev are non-negative. - Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties. - Implement backend initialization and application logic for penalties, including frequency and presence adjustments.

- Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness. - Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties. - Added `test_backend_penalties_sampling` to validate backend penalties with various configurations. - Enhanced the test suite for better coverage of penalty handling in sampling.

… logits as -Inf and no longer generate NaN.

ggml-gh-bot · 2026-07-03T08:17:51Z

Hi @kmorennv, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 2 open PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

kmorennv · 2026-07-03T08:36:47Z

I think that’s a bot error, since I only have a single open PR.

pwilkin · 2026-07-03T08:46:05Z

@kmorennv Nope :)

#21673

pwilkin · 2026-07-03T08:48:55Z

(just for clarity's sake, this is just an informational message and it's not hard-enforced anywhere and since that's just a draft that doesn't really matter, but technically it's not an error)

kmorennv · 2026-07-03T08:49:32Z

@pwilkin thanks , I filtered for this but only for open --> clear , it was not shown ...

ORippler

Thanks for this nice feature addition!

From my side, I'm unsure if we should support [n_ctx, n_vocab]-case for top/min-p -> penalty sampler. Is this a commonly used sampler chain?

…d tests for invalid values

…f candidates

…on_sampler_init

…e values

…lty_last_n

… initialization

…lary-sized count tensor

…tions

danbev

I found the changes in test-backend-sampler.cpp somewhat hard to follow and think it could possibly be simplified. But I realize this might be subjective and since this PR has been open for a while we can perhaps followup and refactor this later if others feel the same.

ORippler

I found the changes in test-backend-sampler.cpp somewhat hard to follow and think it could possibly be simplified.

+1

ggerganov · 2026-08-03T12:45:03Z

        struct ggml_tensor * probs;
        struct ggml_tensor * sampled;
        struct ggml_tensor * candidates;
+        int64_t              n_vocab;


The n_vocab should not be part of this struct. There is already precedent for passing the n_vocab explicitly during sampler construction when it is needed:

llama.cpp/include/llama.h

Lines 1471 to 1474 in 96278e3

LLAMA_API struct llama_sampler * llama_sampler_init_logit_bias(

int32_t n_vocab,

int32_t n_logit_bias,

const llama_logit_bias * logit_bias);

llama.cpp/include/llama.h

Lines 1376 to 1381 in 96278e3

LLAMA_API struct llama_sampler * llama_sampler_init_mirostat(

int32_t n_vocab,

uint32_t seed,

float tau,

float eta,

int32_t m);

Either try to avoid it completely (the CPU version does not need it) or change the sampler init and pass it there.

ggerganov · 2026-08-03T13:04:50Z

+    const int32_t n_ctx = cparams.n_ctx > 0 ? (int32_t) cparams.n_ctx : llama_model_n_ctx_train(model);
    for (int i = 0; i < (int) cparams.n_seq_max; ++i) {
-        pimpl->samplers[i].reset(common_sampler_init(model, params.sampling));
+        pimpl->samplers[i].reset(common_sampler_init(model, params.sampling, n_ctx));
        pimpl->samplers_seq_config[i] = { i, common_sampler_get(pimpl->samplers[i].get()) };
    }


This is not exactly correct because the llama_context is not yet created and we don't know the exact context that will be allocated. That's the reason similar logic was disabled above:

llama.cpp/common/common.cpp

Lines 1288 to 1297 in 96278e3

//if (params.sampling.penalty_last_n == -1) {

// LOG_TRC("%s: setting penalty_last_n to ctx_size = %d\n", __func__, llama_n_ctx(lctx));

// params.sampling.penalty_last_n = llama_n_ctx(lctx);

//}

//if (params.sampling.dry_penalty_last_n == -1) {

// LOG_TRC("%s: setting dry_penalty_last_n to ctx_size = %d\n", __func__, llama_n_ctx(lctx));

// params.sampling.dry_penalty_last_n = llama_n_ctx(lctx);

//}

In general, I think that supporting a parameter of -1 to mean the "context size" is not worth the added complexity and it's much simpler to have an upper limit for these type of parameters. For example, I don't think there is any reason to penalize in a window of more than 1024 tokens, so setting -1 could mean setting automatically setting it to 1024.

ggerganov · 2026-08-03T13:08:40Z

@kmorennv @ORippler Please follow-up with a PR addressing the 2 comments above. Also need to reflect the struct llama_sampler_data API change in #9289

This matches how it is done for logit_bias and mirostat samplers, see #25262 (comment)

* sampling: enhance penalty handling in common_sampler_init - Set default value for penalty_last_n based on model context if not specified. - Ensure penalty_last_n and n_prev are non-negative. - Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties. - Implement backend initialization and application logic for penalties, including frequency and presence adjustments. * tests: add backend penalties sampling tests and utility functions - Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness. - Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties. - Added `test_backend_penalties_sampling` to validate backend penalties with various configurations. - Enhanced the test suite for better coverage of penalty handling in sampling. * sampling: add support for top-k penalties in backend sampling * sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN. * sampling: enhance penalty comparison tests with masking penalties logic * add comments on padding * sampling: add comments on modifications * add the unit test to cover masked-out token as -INF * validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values * refactor: test functions to share logic and be less verbose * add test to cover case where previously penalized token is not part of candidates * remove comments * remove redundant penalty_last_n initialization and validation in common_sampler_init * add support for penalties in sampler chain with configurable positions * add validation for penalty parameters and enhance tests for non-finite values * add context parameter to common_sampler_init and set default for penalty_last_n * add llama_n_ctx parameter to common_sampler_init for improved sampler initialization * replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor * add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64 * add test for backend penalties sampling after top-p with large history window * remove as unused * add is_disabled method, tensor logits reshape, add rest review suggestions * clarify comment

…-org#26520) This matches how it is done for logit_bias and mirostat samplers, see ggml-org#25262 (comment)

This matches how it is done for logit_bias and mirostat samplers, see ggml-org/llama.cpp#25262 (comment)

* test new flash_attn test * rebase and fix to disable subgrou matrices when max_kv_tile == 0 * delete log output * Add i32 support to cpy and enables the all ops test * restore the non target ci tests * comment out of TODO of build-cpu.yml * fix format

* sampling: enhance penalty handling in common_sampler_init - Set default value for penalty_last_n based on model context if not specified. - Ensure penalty_last_n and n_prev are non-negative. - Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties. - Implement backend initialization and application logic for penalties, including frequency and presence adjustments. * tests: add backend penalties sampling tests and utility functions - Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness. - Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties. - Added `test_backend_penalties_sampling` to validate backend penalties with various configurations. - Enhanced the test suite for better coverage of penalty handling in sampling. * sampling: add support for top-k penalties in backend sampling * sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN. * sampling: enhance penalty comparison tests with masking penalties logic * add comments on padding * sampling: add comments on modifications * add the unit test to cover masked-out token as -INF * validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values * refactor: test functions to share logic and be less verbose * add test to cover case where previously penalized token is not part of candidates * remove comments * remove redundant penalty_last_n initialization and validation in common_sampler_init * add support for penalties in sampler chain with configurable positions * add validation for penalty parameters and enhance tests for non-finite values * add context parameter to common_sampler_init and set default for penalty_last_n * add llama_n_ctx parameter to common_sampler_init for improved sampler initialization * replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor * add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64 * add test for backend penalties sampling after top-p with large history window * remove as unused * add is_disabled method, tensor logits reshape, add rest review suggestions * clarify comment

…-org#26520) This matches how it is done for logit_bias and mirostat samplers, see ggml-org#25262 (comment)

* sampling: enhance penalty handling in common_sampler_init - Set default value for penalty_last_n based on model context if not specified. - Ensure penalty_last_n and n_prev are non-negative. - Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties. - Implement backend initialization and application logic for penalties, including frequency and presence adjustments. * tests: add backend penalties sampling tests and utility functions - Introduced `accept_prompt` and `unique_prompt_tokens` functions to handle prompt acceptance and token uniqueness. - Implemented `compare_penalties_logits` to compare logits from backend and CPU samplers with penalties. - Added `test_backend_penalties_sampling` to validate backend penalties with various configurations. - Enhanced the test suite for better coverage of penalty handling in sampling. * sampling: add support for top-k penalties in backend sampling * sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN. * sampling: enhance penalty comparison tests with masking penalties logic * add comments on padding * sampling: add comments on modifications * add the unit test to cover masked-out token as -INF * validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values * refactor: test functions to share logic and be less verbose * add test to cover case where previously penalized token is not part of candidates * remove comments * remove redundant penalty_last_n initialization and validation in common_sampler_init * add support for penalties in sampler chain with configurable positions * add validation for penalty parameters and enhance tests for non-finite values * add context parameter to common_sampler_init and set default for penalty_last_n * add llama_n_ctx parameter to common_sampler_init for improved sampler initialization * replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor * add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64 * add test for backend penalties sampling after top-p with large history window * remove as unused * add is_disabled method, tensor logits reshape, add rest review suggestions * clarify comment

…-org#26520) This matches how it is done for logit_bias and mirostat samplers, see ggml-org#25262 (comment)

…ml-org#26566) * test new flash_attn test * rebase and fix to disable subgrou matrices when max_kv_tile == 0 * delete log output * Add i32 support to cpy and enables the all ops test * restore the non target ci tests * comment out of TODO of build-cpu.yml * fix format

kmorennv added 7 commits June 16, 2026 17:47

sampling: add support for top-k penalties in backend sampling

af9cf9c

sampling: add fix to ensure stable numerical results. Preserve masked…

3ffeaea

… logits as -Inf and no longer generate NaN.

sampling: enhance penalty comparison tests with masking penalties logic

888f264

add comments on padding

1834d9f

sampling: add comments on modifications

38ad11c

kmorennv requested review from a team and ggerganov as code owners July 3, 2026 08:13

github-actions Bot added the testing Everything test related label Jul 3, 2026

ORippler reviewed Jul 7, 2026

View reviewed changes

ORippler self-assigned this Jul 8, 2026

ORippler changed the title ~~Kmoren/add penalties cu backend~~ Add backend sampler for penalties sampler Jul 9, 2026

kmorennv changed the title ~~Add backend sampler for penalties sampler~~ CUDA: Add backend sampler for penalties sampler Jul 10, 2026

kmorennv added 11 commits July 10, 2026 12:50

add the unit test to cover masked-out token as -INF

d7b0551

validate repeat penalty to ensure it is finite and greater than 0; ad…

2adb957

…d tests for invalid values

refactor: test functions to share logic and be less verbose

760b7e5

add test to cover case where previously penalized token is not part o…

65180ab

…f candidates

remove comments

98b1bea

remove redundant penalty_last_n initialization and validation in comm…

381226d

…on_sampler_init

add support for penalties in sampler chain with configurable positions

91341fe

add validation for penalty parameters and enhance tests for non-finit…

2e54fe3

…e values

add context parameter to common_sampler_init and set default for pena…

ec2f07a

…lty_last_n

add llama_n_ctx parameter to common_sampler_init for improved sampler…

e650b60

… initialization

replace penalty_last_n x n_candidates comparison matrix with a vocabu…

cd20503

…lary-sized count tensor

ORippler requested a review from danbev July 20, 2026 13:31

danbev mentioned this pull request Jul 29, 2026

sampling : reshape logits in backend top_k sampler #26270

Closed

danbev reviewed Jul 29, 2026

View reviewed changes

Comment thread src/llama-sampler.cpp Outdated

Comment thread src/llama-sampler.cpp Outdated

danbev reviewed Jul 29, 2026

View reviewed changes

Comment thread src/llama-sampler.cpp Outdated

danbev reviewed Jul 29, 2026

View reviewed changes

Comment thread src/llama-sampler.cpp Outdated

kmorennv added 2 commits July 29, 2026 22:01

add is_disabled method, tensor logits reshape, add rest review sugges…

6e0e28d

…tions

clarify comment

3eb4dfc

danbev approved these changes Jul 31, 2026

View reviewed changes

ORippler approved these changes Jul 31, 2026

View reviewed changes

ORippler merged commit 96278e3 into ggml-org:master Aug 3, 2026
23 of 26 checks passed

ORippler changed the title ~~CUDA: Add backend sampler for penalties sampler~~ Add backend sampler for penalties sampler Aug 3, 2026

ggerganov reviewed Aug 3, 2026

View reviewed changes

This was referenced Aug 3, 2026

Samplers: Move n_vocab from llama_sampler_data to penalty_sampler #26520

Merged

Samplers: Remove ability to specify "full-context windows" from history-based samplers #26524

Merged

ggerganov added a commit that referenced this pull request Aug 4, 2026

llama : move n_vocab from llama_sampler_data to penalty_sampler (#26520)

935cad6

This matches how it is done for logit_bias and mirostat samplers, see #25262 (comment)

smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026

llama : move n_vocab from llama_sampler_data to penalty_sampler (ggml…

414345a

…-org#26520) This matches how it is done for logit_bias and mirostat samplers, see ggml-org#25262 (comment)

ORippler mentioned this pull request Aug 5, 2026

changelog : libllama API #9289

Open

yomaytk mentioned this pull request Aug 7, 2026

ggml-webgpu: fix CI errors from #25025 and #25262 #26566

Merged

jtrefon pushed a commit to jtrefon/llama.cpp-turboq-mtp that referenced this pull request Aug 9, 2026

llama : move n_vocab from llama_sampler_data to penalty_sampler (#26520)

486a883

This matches how it is done for logit_bias and mirostat samplers, see ggml-org/llama.cpp#25262 (comment)

	LLAMA_API struct llama_sampler * llama_sampler_init_logit_bias(
	int32_t n_vocab,
	int32_t n_logit_bias,
	const llama_logit_bias * logit_bias);

	LLAMA_API struct llama_sampler * llama_sampler_init_mirostat(
	int32_t n_vocab,
	uint32_t seed,
	float tau,
	float eta,
	int32_t m);

	//if (params.sampling.penalty_last_n == -1) {
	// LOG_TRC("%s: setting penalty_last_n to ctx_size = %d\n", __func__, llama_n_ctx(lctx));
	// params.sampling.penalty_last_n = llama_n_ctx(lctx);
	//}

	//if (params.sampling.dry_penalty_last_n == -1) {
	// LOG_TRC("%s: setting dry_penalty_last_n to ctx_size = %d\n", __func__, llama_n_ctx(lctx));
	// params.sampling.dry_penalty_last_n = llama_n_ctx(lctx);
	//}

Uh oh!

Conversation

kmorennv commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview & Motivation

Performance Impact

Core Implementation Steps

1. Backend Integration & Fallback

2. State Management & Clone Fix

3. Sparse Graph Optimization

4. Support After Top-K / Top-P / Min-P

5. Testing

Requirements

Uh oh!

ggml-gh-bot Bot commented Jul 3, 2026

Uh oh!

kmorennv commented Jul 3, 2026

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

kmorennv commented Jul 3, 2026

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danbev left a comment

Choose a reason for hiding this comment

Uh oh!

ORippler left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ggerganov Aug 3, 2026

Choose a reason for hiding this comment

Uh oh!

ggerganov Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Aug 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kmorennv commented Jul 3, 2026 •

edited

Loading

ggerganov Aug 3, 2026 •

edited

Loading