GLM-4.7-Flash MTP support

[jacekpoplawski]

Overview

• convert NextN/MTP layers when exporting Glm4MoeLiteForCausalLM
• implement llama_model_deepseek2::graph_mtp

tested trunk-only, mtp-only and combined (default) GGUFs

Additional information

https://huggingface.co/jacek2024/GLM-4.7-Flash-MTP-GGUF

The original idea was to add MTP support for GLM-4.5-Air. But I noticed that GLM-4.7-Flash also has an MTP layer and is much smaller, so I used it as the initial target for learning and getting some feedback.

llama_model_deepseek2::graph_mtp::graph_mtp was implemented by comparing the existing llama_model_deepseek2::graph::graph implementation with llama_model_qwen35::graph_mtp::graph_mtp

benchmarks

GLM-4.7-Flash-MTP-default-Q8_0.gguf

Test	Baseline tok/s	MTP 1 tok/s	MTP 1 speedup	MTP 2 tok/s	MTP 2 speedup	MTP 3 tok/s	MTP 3 speedup
code_python	110.32	129.35	1.17x	123.35	1.12x	115.93	1.05x
code_cpp	111.79	123.67	1.11x	118.58	1.06x	120.60	1.08x
explain_concept	111.71	133.53	1.20x	133.07	1.19x	110.75	0.99x
summarize	111.85	143.02	1.28x	145.68	1.30x	133.21	1.19x
qa_factual	111.82	131.70	1.18x	121.09	1.08x	112.54	1.01x
translation	111.84	133.09	1.19x	124.56	1.11x	114.47	1.02x
creative_short	111.80	130.35	1.17x	119.34	1.07x	108.05	0.97x
stepwise_math	111.90	133.35	1.19x	138.06	1.23x	116.43	1.04x
long_code_review	108.82	122.02	1.12x	114.05	1.05x	99.90	0.92x
Mean	111.32	131.12	1.18x	126.42	1.14x	114.65	1.03x

GLM-4.7-Flash-MTP-default-BF16.gguf

Test	Baseline tok/s	MTP 1 tok/s	MTP 1 speedup	MTP 2 tok/s	MTP 2 speedup	MTP 3 tok/s	MTP 3 speedup
code_python	65.45	84.96	1.30x	82.04	1.25x	78.38	1.20x
code_cpp	65.96	85.10	1.29x	83.11	1.26x	85.16	1.29x
explain_concept	65.65	92.73	1.41x	95.61	1.46x	80.48	1.23x
summarize	66.25	97.46	1.47x	103.75	1.57x	95.42	1.44x
qa_factual	64.96	90.20	1.39x	91.44	1.41x	95.59	1.47x
translation	65.30	93.04	1.42x	93.99	1.44x	80.97	1.24x
creative_short	65.46	87.26	1.33x	85.40	1.30x	74.31	1.14x
stepwise_math	66.73	92.30	1.38x	99.37	1.49x	90.17	1.35x
long_code_review	65.55	86.07	1.31x	83.86	1.28x	80.75	1.23x
Mean	65.70	89.90	1.37x	90.95	1.38x	84.58	1.29x

GLM-4.7-Flash-trunk-only-Q8_0.gguf + GLM-4.7-Flash-MTP-only-BF16.gguf

Test	Baseline tok/s	MTP 1 tok/s	MTP 1 speedup	MTP 2 tok/s	MTP 2 speedup	MTP 3 tok/s	MTP 3 speedup
code_python	110.94	127.55	1.15x	121.80	1.10x	114.97	1.04x
code_cpp	111.72	123.80	1.11x	116.92	1.05x	118.04	1.06x
explain_concept	111.70	131.36	1.18x	127.25	1.14x	106.42	0.95x
summarize	111.86	141.62	1.27x	142.29	1.27x	128.82	1.15x
qa_factual	111.81	130.19	1.16x	118.13	1.06x	109.34	0.98x
translation	111.85	131.81	1.18x	121.43	1.09x	109.22	0.98x
creative_short	111.80	127.97	1.15x	115.52	1.03x	102.03	0.91x
stepwise_math	111.91	134.48	1.20x	138.07	1.23x	115.67	1.03x
long_code_review	108.68	120.80	1.11x	111.74	1.03x	97.08	0.89x
Mean	111.36	129.95	1.17x	123.68	1.11x	111.29	1.00x

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - understanding the graph/MLA, refactoring, benchmark results

[jacekpoplawski]

@am17an I would appreciate any feedback you may have, especially on the helpers split :)

[jacekpoplawski]

rebased on master to include #24832
quantize now works correctly, previously, it was failing like this:
llama_model_quantize: failed to quantize: Bad layer 47 for tensor blk.47.ffn_down_exps.weight. Must be in [0, 47)

[am17an]

the MTP model is just like any other model. So you don't need to helpers like this. Just use the helpers like build_moe_ffn

[jacekpoplawski]

Does it mean I should put all code into one long graph_mtp() ?

[am17an]

what I mean is that you should re-use the helpers in llama-graph as much as possible instead of creating your own. Also I'm not sure if there is a need for a graph_mtp class. You can see how we currently do it for qwen/gemma/step3.5 etc

[jacekpoplawski]

I based my code on qwen35, it has:

    struct graph_mtp : public llm_graph_context {
        graph_mtp(const llama_model & model, const llm_graph_params & params);
    };

I did same but then I split long graph_mtp into smaller functions

[am17an]

@jacekpoplawski is this ready for review?

[jacekpoplawski]

@jacekpoplawski is this ready for review?

I will refactor to one big function this evening

[jacekpoplawski]

@CISC could you check this out maybe? :)

[jacekpoplawski]

@am17an do you have any suggestions on who I should ask for a second review? :)

[am17an]

Wait till @CISC is back I guess?

[arbv]

@CISC Can you please tell me how to load the provided MTP-only GGUF w/o redownloading the complete model GGUF (if that is even possible). Because it is not clear to me at all.

[jacekpoplawski]

@CISC Can you please tell me how to load the provided MTP-only GGUF w/o redownloading the complete model GGUF (if that is even possible). Because it is not clear to me at all.

I think you need -md