server : make 2 checkpoints near the end of the prompt - #20288
Conversation
Thanks, this helps a lot. From reading the patch, my understanding is that the new logic intentionally keeps two near-end checkpoints:
My guess is that this is a tradeoff between:
Is that the right way to think about it? If so, what is the main practical benefit of still keeping a small tail (for example ~64 tokens) uncheckpointed, instead of trying to checkpoint all the way to the end of the prompt? Is that mainly for robustness when the last user message changes slightly or prompt boundaries shift? Also, in my testing, this behavior does not show up with Qwen3-30B-A3B: replay/regenerate there can have near-zero prompt re-eval, while Qwen3.5-35B-A3B re-evaluates the tail much more noticeably. Is that difference mainly because Qwen3-30B-A3B can reuse the live KV/cache path more directly, while Qwen3.5-35B-A3B relies more on this checkpoint heuristic due to its hybrid/recurrent-like architecture? Finally, do you think near-zero or zero prompt re-eval is achievable in principle for Qwen3.5 regenerate/replay, or is some small amount of tail reprocessing still expected by design? |
Yes, that's correct.
The main restriction is to guarantee that no "reasoning" tokens will get included in the checkpoint because they will be removed for the next user message. I guess 64 tokens is quite big, considering that typically only one "reasoning" token gets added (f.ex |
|
Thanks a lot for this! I just tested this branch and It is working well and reduces the time to first token quite a lot :) |
|
Thanks, could you confirm that changing the secondary checkpoint from 64 -> 4 tokens works OK? |
It seems to be working okay with 4 as well for Qwen3.5 with the default template in both instruct and reasoning modes. |
|
The longest I know is gpt-oss at 3 tokens: Seems like a safe assumption. |
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
* server : make 2 checkpoints near the end of the prompt * cont : adjust checkpoints
Upstream breaks the prompt batch twice near the end so checkpoints can be taken (PR ggml-org#20288, offsets {4 + n_ubatch, 4}). Each break is a separate llama_decode, and a decode boundary drains the whole pipeline -- a cost proportional to stage count on a multi-stage RPC split. Measured on DeepSeek-V4-Flash across 10 stages (8 BC-250s + shredder x2), 15,360-token real-text prompts: the two breaks showed up in the daemon graph traces as gaps of 15.0 s and 10.3 s in a 60 s board span, against a median inter-graph gap of 0.11 s. The deep break only buys a fallback checkpoint n_ubatch tokens further back. Per-turn restores come from the user-message-boundary rule above it, and end-of-prompt continuation from the shallow break, so drop it by default and keep upstream's pair behind LLAMA_CKPT_DEEP_BREAK=1. Prefill 180.3 -> 205.4 t/s warm (+13.9%), 158.1 -> 182.0 cold; the board trace now shows exactly one drain and duty rises 50% -> 66%. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a one Upstream cuts the prompt batch 4 + n_ubatch and 4 tokens before the end so a checkpoint can be snapshotted before a decode, when the memory holds everything except the batch about to run (PR ggml-org#20288). Each cut is a separate llama_decode, and a decode boundary drains the whole pipeline. That cost is proportional to backend count and, crucially, independent of how many tokens the extra decode carries: on a 10-stage RPC split the surviving 4-token break still cost 11.6 s of an 85 s request. Snapshot in post_decode() instead, right after the decode that completes the prompt. That decode has already drained the pipeline so the caller can sample, so the state read rides a synchronization that is paid anyway, and the checkpoint covers the whole prompt instead of stopping 4 tokens short -- a strictly later recovery point. LLAMA_CKPT_PROMPT_BREAKS = 0 (default, deferred) | 1 (shallow break) | 2 (upstream's pair), replacing LLAMA_CKPT_DEEP_BREAK from the previous commit. DeepSeek-V4-Flash, 8 BC-250s + shredder x2, 15,360-token real-text prompts: breaks cold warm 2 (upstream) 158.1 180.3 1 (shallow only) 182.0 205.4 0 (deferred) 180.7 239.8 0 + checkpoints disabled 184.0 243.0 <- ceiling +33% over upstream and within 1.3% of the ceiling; the residual is the 17 MiB state read itself. Restore verified end to end: a request diverging from the cached sequence re-processes 10 tokens of 1210 -- checking checkpoint with [1199, 1199] against 1200... restored context checkpoint (pos_min = 1199, n_past = 1200) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a one Upstream cuts the prompt batch 4 + n_ubatch and 4 tokens before the end so a checkpoint can be snapshotted before a decode, when the memory holds everything except the batch about to run (PR ggml-org#20288). Each cut is a separate llama_decode, and a decode boundary drains the whole pipeline. That cost is proportional to backend count and, crucially, independent of how many tokens the extra decode carries: on a 10-stage RPC split the surviving 4-token break still cost 11.6 s of an 85 s request. Snapshot in post_decode() instead, right after the decode that completes the prompt. That decode has already drained the pipeline so the caller can sample, so the state read rides a synchronization that is paid anyway, and the checkpoint covers the whole prompt instead of stopping 4 tokens short -- a strictly later recovery point. LLAMA_CKPT_PROMPT_BREAKS = 0 (default, deferred) | 1 (shallow break) | 2 (upstream's pair), replacing LLAMA_CKPT_DEEP_BREAK from the previous commit. DeepSeek-V4-Flash, 8 BC-250s + shredder x2, 15,360-token real-text prompts: breaks cold warm 2 (upstream) 158.1 180.3 1 (shallow only) 182.0 205.4 0 (deferred) 180.7 239.8 0 + checkpoints disabled 184.0 243.0 <- ceiling +33% over upstream and within 1.3% of the ceiling; the residual is the 17 MiB state read itself. Restore verified end to end: a request diverging from the cached sequence re-processes 10 tokens of 1210 -- checking checkpoint with [1199, 1199] against 1200... restored context checkpoint (pos_min = 1199, n_past = 1200) Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4 participants
fix #20239 (comment)
In some cases, reprocessing the last 512 tokens of the prompt could be too slow. In other cases it is necessary in order to allow mutating the last user message.
Make 2 checkpoints to satisfy all needs.