convert: add option to create separate dspark GGUF

[am17an]

Overview

Additional information

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, created by DSV4-Flash

[ggerganov]

If I use --no-mtp on the new checkpoint, it is going to skip the DSpark, correct?

Or should we add --no-dspark (or even more general - --no-spec) to make it more semantically correct?

[coder543]

I tried this out, and there seemed to be a mismatch in dflash.embedding_length_out that caused llama-server to abort. The full target uses 16k, but the DSpark might only use 4k for post-hyperconnection embeddings? Maybe I'm doing something wrong.

[am17an]

@coder543 what command did you run? You also need to have #25784 in your tree

[coder543]

Yes, I had #25784 in my build.

./.venv/bin/python \
    convert_hf_to_gguf.py \
    "$source_dir" \
    --dspark \
    --target-model-dir "$source_dir" \
    --outtype bf16 \
    --outfile "$out"

When I launched llama-server, I saw an error:

0.13.407.317 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 16384, kv_unified = 'true'
0.13.407.354 I common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark'
0.13.407.356 I common_speculative_impl_draft_dflash: - n_max=5, n_min=1, p_min=0.00
0.13.407.357 I common_speculative_impl_draft_dflash: - block_size=5, mask_token_id=128799, n_extract=3
llama.cpp/ggml/src/ggml-backend.cpp:272: GGML_ASSERT(offset + size <= ggml_nbytes(tensor) && "tensor read out of bounds") failed
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000e7a6fe557b74 in __GI___wait4 (pid=36254, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x0000e7a6fe557b74 in __GI___wait4 (pid=36254, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x0000e7a6fdaa8e7c in ggml_print_backtrace () from llama.cpp/build/bin/libggml-base.so.0
#2  0x0000e7a6fdaa9020 in ggml_abort () from llama.cpp/build/bin/libggml-base.so.0
#3  0x0000e7a6fdac07cc in ggml_backend_tensor_get_async () from llama.cpp/build/bin/libggml-base.so.0
#4  0x0000e7a6fdc72e34 in llama_context::decode(llama_batch const&) () from llama.cpp/build/bin/libllama.so.0
#5  0x0000e7a6fdc73b60 in llama_decode () from llama.cpp/build/bin/libllama.so.0
#6  0x0000e7a6fe2040e0 in common_context_can_seq_rm(llama_context*) () from llama.cpp/build/bin/libllama-common.so.0
#7  0x0000e7a6fea8d48c in server_context_impl::load_model(common_params&) () from llama.cpp/build/bin/libllama-server-impl.so
#8  0x0000e7a6fe9c8c1c in llama_server(common_params&, int, char**) () from llama.cpp/build/bin/libllama-server-impl.so
#9  0x0000e7a6fe9cb4a0 in llama_server(int, char**) () from llama.cpp/build/bin/libllama-server-impl.so
#10 0x0000e7a6fe4a84c4 in __libc_start_call_main (main=main@entry=0xacee9b531140 <main>, argc=argc@entry=41, argv=argv@entry=0xfffff7619658) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58	../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#11 0x0000e7a6fe4a8598 in __libc_start_main_impl (main=0xacee9b531140 <main>, argc=41, argv=0xfffff7619658, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:360
warning: 360	../csu/libc-start.c: No such file or directory
#12 0x0000acee9b5311b0 in _start ()

  llama-quantize \
    --override-kv dflash.embedding_length_out=int:4096 \
    deepseek-v4-flash-0731-dspark-bf16.gguf \
    deepseek-v4-flash-0731-dspark-bf16-fixed.gguf \
    COPY

This fixed it for me, and DSpark worked correctly after that.

[am17an]

Can you try the latest commit?

[drrros]

@coder543 @am17an guys, can you, please, share a dspark gguf?

[coder543]

@am17an yes, that fixed it!

[am17an]

@drrros
https://huggingface.co/am17an/DeepseekV4-Flash-20260731-DSpark/blob/main/DeepseekV4-Flash-20260731-DSpark.gguf

[ggerganov]

Btw, the base model https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Base is in FP8 format and it fails to convert:

https://github.com/ggml-org/convert/actions/runs/30757137366/job/91521142907#step:5:2303

Would be nice to fix this so we can run PPL measurements.

[am17an]

@ggerganov I only ever convert the MXFP4 stuff, I think that's currently hardcoded, we would need to add some logic to disambiguate FP8/MXFP4

[thomasjfox]

@drrros https://huggingface.co/am17an/DeepseekV4-Flash-20260731-DSpark/blob/main/DeepseekV4-Flash-20260731-DSpark.gguf

Thank you for your work on this, @am17an!

I tried to use this together with the UD-Q8_K_XL from Unsloth, but it triggers an assertion on llama-server startup:

0.00.289.788 I srv    load_model: loading model '/mnt/local/ai-models/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf'
/mnt/local/ai-models/llama.cpp/ggml/src/ggml-backend.cpp:1356: GGML_ASSERT(n_graph_inputs < GGML_SCHED_MAX_SPLIT_INPUTS) failed

System is a dual RTX PRO 6000 Max-Q.

Full backtrace:

0x00007fa606a87902 in __syscall_cancel_arch () from /lib64/libc.so.6
#0  0x00007fa606a87902 in __syscall_cancel_arch () from /lib64/libc.so.6
#1  0x00007fa606a7bb9c in __internal_syscall_cancel () from /lib64/libc.so.6
#2  0x00007fa606a7bbe4 in __syscall_cancel () from /lib64/libc.so.6
#3  0x00007fa606aebb0f in wait4 () from /lib64/libc.so.6
#4  0x00007fa60cf33a33 in ggml_print_backtrace () from /mnt/local/ai-models/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007fa60cf33b7c in ggml_abort () from /mnt/local/ai-models/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007fa60cf4edc2 in ggml_backend_sched_split_graph () from /mnt/local/ai-models/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007fa61024aff9 in llama_context::graph_reserve(unsigned int, unsigned int, unsigned int, llama_memory_context_i const*, bool, unsigned long*) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama.so.0                                                                                                                                                      
#8  0x00007fa61024b502 in llama_context::resolve_fused_ops(llama_memory_context_i const*, unsigned int) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama.so.0
#9  0x00007fa61024c4d3 in llama_context::sched_reserve() () from /mnt/local/ai-models/llama.cpp/build/bin/libllama.so.0
#10 0x00007fa61024e9e6 in llama_context::llama_context(llama_model const&, llama_context_params) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama.so.0
#11 0x00007fa61024fd6d in llama_init_from_model () from /mnt/local/ai-models/llama.cpp/build/bin/libllama.so.0
#12 0x00007fa6107645bf in common_init_result::common_init_result(common_params&, bool) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama-common.so.0
#13 0x00007fa610767106 in common_init_from_params(common_params&, bool) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama-common.so.0
#14 0x00007fa610d31d57 in server_context_impl::load_model(common_params&) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama-server-impl.so
#15 0x00007fa610c618b6 in llama_server(common_params&, int, char**) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama-server-impl.so
#16 0x00007fa610c63922 in llama_server(int, char**) () from /mnt/local/ai-models/llama.cpp/build/bin/libllama-server-impl.so
#17 0x00007fa606a105b5 in __libc_start_call_main () from /lib64/libc.so.6
#18 0x00007fa606a10668 in __libc_start_main_impl () from /lib64/libc.so.6
#19 0x00000000004003b5 in _start ()

Cmdline used:

./llama-server \
    --model /mnt/local/ai-models/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --model-draft /mnt/local/ai-models/Deepseek-V4-Flash-0731-20260731-DSpark.gguf \
    --spec-type draft-dspark \
    --chat-template-kwargs '{"reasoning_effort":"high"}' \
    --threads 16 \
    --jinja \
    --flash-attn on \
    -np 4 \
    --ctx-size 400000 \
    --kv-unified \
    --cache-ram 16384 \
    --metrics \
    --temp 1.0 \
    --top_p 0.95 \
    --min-p 0.0 \
    --fit off \
    --no-mmap \
    --timeout 3600

The draft model loads perfectly fine (and stays unused) when removing the draft-dspark cmdline argument:

0.00.460.036 I srv    load_model: loading model '/mnt/local/ai-models/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf'
1.12.770.807 I common_speculative_init_result: loading draft model '/mnt/local/ai-models/Deepseek-V4-Flash-0731-20260731-DSpark.gguf'
1.13.619.878 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 400128, kv_unified = 'true'
1.13.651.215 I srv  llama_server: model loaded

I'm currently running commit 221f0f6

So I guess the GGUF is not compatible with the UD-Q8_K_XL or is this some other issue?

[coder543]

No, I’ve used it with the UD-Q8_K_XL just fine. I ran into an assertion when the output_weights weren’t on the same CUDA device as the DSpark model, so I had to override the placement of those weights. Maybe that’s what you’re running into?

[thomasjfox]

Found the same report here with dual RTX Pro 6000: #25784 (comment)

The trick was to slightly raise GGML_SCHED_MAX_SPLIT_INPUTS:

diff --git a/ggml/src/ggml-backend.cpp b/ggml/src/ggml-backend.cpp
index 7f4e252dc..02e56711a 100644
--- a/ggml/src/ggml-backend.cpp
+++ b/ggml/src/ggml-backend.cpp
@@ -754,7 +754,7 @@ static bool ggml_is_view_op(enum ggml_op op) {
 #endif
 
 #ifndef GGML_SCHED_MAX_SPLIT_INPUTS
-#define GGML_SCHED_MAX_SPLIT_INPUTS 30
+#define GGML_SCHED_MAX_SPLIT_INPUTS 40
 #endif
 
 #ifndef GGML_SCHED_MAX_COPIES

Toks/s almost doubled with DSpark!

It was 54 toks/s before, now it once hit 101 toks/s and seems to stay around 88 toks/s. Wow! 🥳

[tarruda]

I'm noticing that llama-imatrix has flags to turn on speculative decoding. Should I re-convert the GGUF with dpark/mtp bundled and regenerate imatrix or can I just re-run convert-hf-to-ggu.py passing --dspark?

[tarruda]

Just saw that the separate dspark q8_0 gguf is about 10G. I guess that rules out using dspark if close to memory limit.

[am17an]

That is when it's all in q8. q4 would be half its size

[tarruda]

That is when it's all in q8. q4 would be half its size

It seems that the --dspark output GGUF is mostly MXFP4. When I tried to requantize to Q4_K the size actually increased.

I also tried to re-convert the gguf without specifying --no-mtp (which I expected to include the dspark tensors in the main GGUF), but there doesn't seem to be any size difference from specifying --no-mtp. Do I have to specify some other argument to bundle everything together?

[am17an]

Ah right I forgot everything is in MXFP4. I think --no-mtp would still exclude the Dspark tensors

[tarruda]

Yea what I meant is that --no-mtp/--no-nextn doesn't seem to have any effect, the extra tensors are already excluded and there's no way to bundle them in the main gguf. I was hoping that by bundling in the main GGUF we could save some RAM, that's why I was considering trying it.