DeepSeek-V4 Checkpointing Fix by danielhanchen · Pull Request #25402 · ggml-org/llama.cpp

danielhanchen · 2026-07-07T11:49:26Z

PR closed - #25202 was the one which fixed the issue and was merged 20 hours ago - and I was using an older llama.cpp version which caused gibberish

Specifically using --cache-type-k/v q8_0 before #25202 caused garbage ie overlayotin kinetic academyléléléléulif after after the PR "The capital of France is Paris." (correct)

Fixes DeepSeek-V4 prompt caching - before --ctx-checkpoints 0 had to be used since prompt caching for DeepSeek-V4-Flash would cause gibberish.

llama.cpp Before PR After PR

Tool calling (temp 1.0, 3 seeds) 4/15 15/15

Used https://huggingface.co/unsloth/DeepSeek-V4-Flash-GGUF UD-Q8_K_XL, UD-Q4_K_XL for testing and other community quants to verify it wasn't a quant issue!

Fixes prompt caching for DeepSeek-V4. Before this, --ctx-checkpoints 0 had to be used since prompt caching for DeepSeek-V4-Flash produced gibberish. - Classify DeepSeek-V4 as RS so context checkpoints stay enabled - seq_rm: evict the 1-token tail on the last-token re-eval so it is not duplicated - state_write/read: include the base and block caches in partial (checkpoint) saves, since they are not recomputable from a partial re-decode

ggml-gh-bot · 2026-07-07T11:53:37Z

Hi @danielhanchen, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

PR Template not respected: Please respect the template when creating a new pull request. Make sure to fill out all required sections.
Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

am17an · 2026-07-07T12:42:19Z


    cparams.n_rs_seq = params.n_rs_seq;
+    // DSV4: default to its 1-token rollback bound so it is classified RS (keeps checkpoints).
+    if (cparams.n_rs_seq == 0 && model.arch == LLM_ARCH_DEEPSEEK4) {


Pretty sure this is not required. Can you try without changing this value

am17an · 2026-07-07T16:26:07Z

@danielhanchen do you have a repro of the gibberish?

remeh · 2026-07-07T18:08:46Z

Hey folks 👋 First things first, thanks a lot @danielhanchen and llama.cpp folks for your continuous & incredible work 🙇 I never had the chance to thank you.

I still have gibberish output despite this PR. For information I'm running llama.cpp main (latest commit c198af4 from Tue Jul 7 18:52:30 2026 +0300) + this patch. I'm running on a Strix Halo with ROCm.

Compilation flags:

cmake -S . -B build \
        -DGGML_HIP=ON \
        -DGGML_HIPBLAS=ON \
        -DGGML_HIP_GRAPHS=ON \
        -DGGML_OPENMP=ON \
        -DHIP_PLATFORM=amd \
        -DGGML_CUDA_FA_ALL_QUANTS=ON \
        -DGGML_HIP_ROCWMMA_FATTN=OFF \
        -DGPU_TARGETS=gfx1151 \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLAMA_OPENSSL=ON \
        -DLLAMA_BUILD_EXAMPLES=OFF \
        --fresh

Here's my configuration if that helps:

/home/remy/opt/llama.cpp/build/bin/llama-server --cache-prompt --host 127.0.0.1 --metrics --min-p 0.0 --mlock --no-mmap --port 55499 --temperature 1.0 --top-p 1.0 --webui-mcp-proxy --alias deepseek-v4-flash --ctx-size 250072 --cache-type-k f16 --cache-type-v f16 --direct-io --flash-attn 1 --fit 1 --model /home/remy/opt/models/unsloth/DeepSeek-V4-Flash-UD-IQ3_XXS.gguf --n-gpu-layers 999 --parallel 1 --threads -1

ANTONBORODA · 2026-07-07T18:24:06Z

Hey folks 👋 First things first, thanks a lot @danielhanchen and llama.cpp folks for your continuous & incredible work 🙇 I never had the chance to thank you.

I still have gibberish output despite this PR. For information I'm running llama.cpp main (latest commit c198af4 from Tue Jul 7 18:52:30 2026 +0300) + this patch. I'm running on a Strix Halo with ROCm.

Compilation flags:
cmake -S . -B build \
        -DGGML_HIP=ON \
        -DGGML_HIPBLAS=ON \
        -DGGML_HIP_GRAPHS=ON \
        -DGGML_OPENMP=ON \
        -DHIP_PLATFORM=amd \
        -DGGML_CUDA_FA_ALL_QUANTS=ON \
        -DGGML_HIP_ROCWMMA_FATTN=OFF \
        -DGPU_TARGETS=gfx1151 \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLAMA_OPENSSL=ON \
        -DLLAMA_BUILD_EXAMPLES=OFF \
        --fresh
Here's my configuration if that helps:
/home/remy/opt/llama.cpp/build/bin/llama-server --cache-prompt --host 127.0.0.1 --metrics --min-p 0.0 --mlock --no-mmap --port 55499 --temperature 1.0 --top-p 1.0 --webui-mcp-proxy --alias deepseek-v4-flash --ctx-size 250072 --cache-type-k f16 --cache-type-v f16 --direct-io --flash-attn 1 --fit 1 --model /home/remy/opt/models/unsloth/DeepSeek-V4-Flash-UD-IQ3_XXS.gguf --n-gpu-layers 999 --parallel 1 --threads -1

Can confirm.
Strix Halo with ROCm is broken from turn one.
I tried both "official" build created by this repo and built it myself with TheROC 7.15 nighlies, nothing helps.

This PR does not help either.

CPU inference works if I set NGL to 0 on the same build.

remeh · 2026-07-07T19:30:40Z

Some additional info in case that helps, broken (on strix halo) even with just these build flags:

cmake -S . -B build \
        -DGGML_HIP=ON \
        -DHIP_PLATFORM=amd \
        -DGPU_TARGETS=gfx1151 \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLAMA_OPENSSL=ON \
        -DLLAMA_BUILD_EXAMPLES=OFF \
        --fresh

ddh0 · 2026-07-07T21:15:26Z

I don't get gibberish with context checkpoints and DSv4 flash on master.

danielhanchen · 2026-07-08T06:29:43Z

Actually I re-checked - #25202 was the one which fixed it and was merged 20 hours ago - and I was using an older llama.cpp version which caused gibberish

Specifically using --cache-type-k/v q8_0 before #25202 caused garbage ie overlayotin kinetic academyléléléléulif after after the PR "The capital of France is Paris." (correct)

For folks who are getting gibberish on an old llama.cpp build - re-get from main and KV cache quantization works!

I conflated the --ctx-checkpoints issue on manual testing - I'll edit the description as well

ANTONBORODA · 2026-07-08T07:34:52Z

Some additional info in case that helps, broken (on strix halo) even with just these build flags:

cmake -S . -B build \
        -DGGML_HIP=ON \
        -DHIP_PLATFORM=amd \
        -DGPU_TARGETS=gfx1151 \
        -DCMAKE_BUILD_TYPE=Release \
        -DLLAMA_OPENSSL=ON \
        -DLLAMA_BUILD_EXAMPLES=OFF \
        --fresh

I opened a followup issue:
#25436

Core DeepSeek-V4-Flash serving work on the llama runtime: - Checkpointing/state-restore: DSV4 recurrent compressor-state save/load with rollback on partial failure (PR ggml-org#25402 lineage); n_rs_seq clamped to the true 1-token direct-rollback bound. - MTP speculative decode: deepseek4mtp draft arch (loader + graph + memory), draft-mtp speculative type, frontier-stash partial-accept rewind, and the opt-in fused chained-draft graph (LLAMA_MTP_FUSED_DRAFT). ~20.8 t/s decode at n_max=2 on GB10. - Refusal-ablation steering: LLAMA_CVEC_FFN_ONLY gate for ffn-only projection. - The MTP fused chained-draft plumbing is named mtp_draft_* (llama_set_mtp_ draft_chain / llama_get_mtp_draft_meta / cparams.mtp_draft_chain).

danielhanchen added a commit to unslothai/llama.cpp that referenced this pull request Jul 7, 2026

Add DeepSeek-V4 checkpointing fix (ggml-org#25402) to prebuilt PR set

ed08956

am17an reviewed Jul 7, 2026

View reviewed changes

ggerganov mentioned this pull request Jul 7, 2026

llama_dsv4: write only used rows in state #25325

Merged

danielhanchen closed this Jul 8, 2026

This was referenced Jul 8, 2026

Eval bug: DSV4-Flash churned-reuse SWA KV-cache exhaustion (crash + stall) #25452

Open

Eval bug: DeepSeek-V4-Flash lightning-indexer materialises a ctx-proportional (4 GiB/layer) score buffer #25468

Closed

allarac mentioned this pull request Jul 9, 2026

Eval bug: DeepSeep V4 garbled output on Strix Halo with ROCm #25436

Open

Navigation Menu

Uh oh!

DeepSeek-V4 Checkpointing Fix - #25402

DeepSeek-V4 Checkpointing Fix#25402
danielhanchen wants to merge 1 commit into
ggml-org:masterfrom
danielhanchen:deepseek-v4-checkpointing-fix

danielhanchen commented Jul 7, 2026 •

edited

Loading

Uh oh!

ggml-gh-bot Bot commented Jul 7, 2026

Uh oh!

am17an Jul 7, 2026

Uh oh!

am17an commented Jul 7, 2026

Uh oh!

remeh commented Jul 7, 2026

Uh oh!

ANTONBORODA commented Jul 7, 2026 •

edited

Loading

Uh oh!

remeh commented Jul 7, 2026

Uh oh!

ddh0 commented Jul 7, 2026 •

edited

Loading

Uh oh!

danielhanchen commented Jul 8, 2026 •

edited

Loading

Uh oh!

ANTONBORODA commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

danielhanchen commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggml-gh-bot Bot commented Jul 7, 2026

Uh oh!

am17an Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

am17an commented Jul 7, 2026

Uh oh!

remeh commented Jul 7, 2026

Uh oh!

ANTONBORODA commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

remeh commented Jul 7, 2026

Uh oh!

ddh0 commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielhanchen commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ANTONBORODA commented Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

danielhanchen commented Jul 7, 2026 •

edited

Loading

ANTONBORODA commented Jul 7, 2026 •

edited

Loading

ddh0 commented Jul 7, 2026 •

edited

Loading

danielhanchen commented Jul 8, 2026 •

edited

Loading