Skip to content

convert_hf_to_gguf.py: Refactor how --mtp / --no-mtp is enabled for a model - #25641

Merged
pwilkin merged 1 commit into
ggml-org:masterfrom
tarruda:refactor-convert-to-gguf-mtp-option
Jul 14, 2026
Merged

convert_hf_to_gguf.py: Refactor how --mtp / --no-mtp is enabled for a model#25641
pwilkin merged 1 commit into
ggml-org:masterfrom
tarruda:refactor-convert-to-gguf-mtp-option

Conversation

@tarruda

@tarruda tarruda commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Overview

--no-mtp/--mtp doesn't currently work on Hy3 (#25395) despite the model supporting MTP.

Refactor how that those options are added by adding a flag to the ModelBase class and enabling for Qwen 3.x, Stepfun and Hy3.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: I used codex to show me how this mtp/no-mtp option was enabled and refactored the code myself.

@tarruda
convert_hf_to_gguf: support split MTP export for HY V3
3e6bac3 
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
@tarruda
tarruda requested a review from CISC as a code owner July 14, 2026 00:18
@tarruda tarruda mentioned this pull request Jul 14, 2026
Merged
@satindergrewal

satindergrewal commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Confirming from the Hy3 side: HYV3Model already implements the full no_mtp / mtp_only filtering (the n_nextn hparams guard in set_gguf_parameters plus the filter_tensors checks in conversion/hunyuan.py), so the hardcoded gate in main() was the only thing blocking it. That was my omission in #25395, thanks for catching it. The supports_mtp_export class flag is also a cleaner shape than the growing issubclass chain, and the diff covers all three current MTP families. LGTM.

@pwilkin
pwilkin merged commit cb489bc into ggml-org:master Jul 14, 2026
1 check passed
@tarruda
tarruda deleted the refactor-convert-to-gguf-mtp-option branch July 14, 2026 09:53
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026
@tarruda ROCm FP4 Contributor
convert_hf_to_gguf: support split MTP export for HY V3 (ggml-org#25641)
5475600 
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
@satindergrewal satindergrewal mentioned this pull request Jul 22, 2026
Merged
RehanQasim-dev pushed a commit to aifoundry-org/llama.cpp that referenced this pull request Jul 23, 2026
@tarruda @RehanQasim-dev
convert_hf_to_gguf: support split MTP export for HY V3 (ggml-org#25641)
09a9ac8 
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
RehanQasim-dev pushed a commit to aifoundry-org/llama.cpp that referenced this pull request Jul 23, 2026
@tarruda @RehanQasim-dev
convert_hf_to_gguf: support split MTP export for HY V3 (ggml-org#25641)
9eef749 
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
satindergrewal added a commit to satindergrewal/llama.cpp that referenced this pull request Jul 26, 2026
@satindergrewal @claude
convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM…
8cd7811 
…-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
am17an pushed a commit that referenced this pull request Jul 29, 2026
@satindergrewal @claude
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
7be2c65 
…2) (#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
@zzffesgg58 zzffesgg58 mentioned this pull request Jul 29, 2026
Open
praneshgo pushed a commit to praneshgo/llama.cpp that referenced this pull request Jul 29, 2026
@satindergrewal @claude @praneshgo
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
f14bfdb 
…2) (ggml-org#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
kashif pushed a commit to kashif/llama.cpp that referenced this pull request Aug 2, 2026
@satindergrewal @claude @kashif
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
286d707 
…2) (ggml-org#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
huaxel pushed a commit to huaxel/CachyLLama that referenced this pull request Aug 2, 2026
@satindergrewal @claude @huaxel
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
6cbb3cb 
…2) (ggml-org#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026
@tarruda @smalinin
convert_hf_to_gguf: support split MTP export for HY V3 (ggml-org#25641)
39038ed 
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
smalinin pushed a commit to smalinin/llama.cpp that referenced this pull request Aug 4, 2026
@satindergrewal @claude @smalinin
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
ad0e882 
…2) (ggml-org#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
satindergrewal added a commit to satindergrewal/llama.cpp that referenced this pull request Aug 11, 2026
@satindergrewal @claude
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
fec4d04 
…2) (ggml-org#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
satindergrewal pushed a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026
@tarruda
convert_hf_to_gguf: support split MTP export for HY V3 (ggml-org#25641)
16214c4 
- Add a supports_mtp_export capability to ModelBase so architectures can opt
  into --mtp and --no-mtp without extending a central class allowlist.
- Enable the capability for the existing Qwen3.5/3.6 and Step3.5/3.7
  implementations, and for HY V3, whose converter already supports
  filtering the appended MTP layers.
satindergrewal added a commit to satindergrewal/llama.cpp that referenced this pull request Aug 12, 2026
@satindergrewal @claude
model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.…
64e0606 
…2) (ggml-org#25980)

* model: add NextN/MTP speculative decoding support for GLM_DSA (GLM-5.2)

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target: nextn tensor
loading via the qwen35moe/step35-style presence probe, a graph_mtp
builder (enorm/hnorm/eh_proj + dense MLA + sigmoid-gated MoE with
shared expert + shared head with fallbacks, _s scale tensors passed
for NVFP4), t_h_nextn extraction in the trunk graph, and MTP-context
KV setup: the draft head runs dense MLA, so the MTP context uses a
plain attention KV cache holding only the nextn layer(s) (same
pattern as the hybrid Qwen3.5 MTP context) while the main context
keeps the DSA cache, now filtered to trunk layers only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* convert : support --mtp/--no-mtp export for GlmMoeDsaForCausalLM (GLM-5.2)

Opt GLM-5.2 into the supports_mtp_export contract (post-ggml-org#25641 shape,
mirroring HYV3Model/Step35Model): --no-mtp drops the appended NextN
block (blk.78) and its nextn_predict_layers KV; --mtp keeps only the
NextN block plus shared embeddings/norm/lm_head. Default (bundled)
output is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@am17an am17an am17an approved these changes
@pwilkin pwilkin pwilkin approved these changes
@CISC CISC Awaiting requested review from CISC CISC is a code owner

Assignees

No one assigned

Labels

conversion

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tarruda @satindergrewal @am17an @pwilkin