[Model] Support MiniCPM-V 4.6 by tc-mb · Pull Request #22529 · ggml-org/llama.cpp

tc-mb · 2026-04-29T18:50:14Z

This PR adds support for MiniCPM-V 4.6.

v4.6 reworks the vision tower — the resampler is replaced by a new merger structure, which improves ViT efficiency.
The GGUF conversion is also folded into the standard convert_hf_to_gguf.py flow.

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

ngxson

please change the naming to minicpmv2_6 everywhere for consistency

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

ngxson

some style nits before I can approve this

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

CISC · 2026-05-06T16:30:20Z

Something is wrong with GitHub, can't comment directly on code in this PR any more...

It looks like they updated the tokenizer (see #22700), but we want to keep the old hash so please revert the change of the original chkhsh in convert_hf_to_gguf.py.

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

tc-mb · 2026-05-06T18:07:49Z

chkhsh

ok，fixed

CISC · 2026-05-06T18:28:55Z

To fix https://github.com/ggml-org/llama.cpp/actions/runs/25452560865/job/74674331069?pr=22529

    gguf_writer: gguf.GGUFWriter
    rope_parameters: dict

    def set_gguf_parameters(self):
        super().set_gguf_parameters()  # ty: ignore[unresolved-attribute]
        if "mrope_section" not in self.rope_parameters:
            self.gguf_writer.add_rope_dimension_sections(self._QWEN35_DEFAULT_MROPE_SECTION)

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

tc-mb · 2026-05-06T19:02:09Z

To fix https://github.com/ggml-org/llama.cpp/actions/runs/25452560865/job/74674331069?pr=22529

    gguf_writer: gguf.GGUFWriter
    rope_parameters: dict

    def set_gguf_parameters(self):
        super().set_gguf_parameters()  # ty: ignore[unresolved-attribute]
        if "mrope_section" not in self.rope_parameters:
            self.gguf_writer.add_rope_dimension_sections(self._QWEN35_DEFAULT_MROPE_SECTION)

ok, fixed

* Support MiniCPM-V 4.6 in new branch Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix code bug Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix pre-commit Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix convert Signed-off-by: tc-mb <tianchi_cai@icloud.com> * rename clip_graph_minicpmv4_6 Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use new TYPE_MINICPMV4_6 Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use build_attn to allow flash attention support Signed-off-by: tc-mb <tianchi_cai@icloud.com> * no use legacy code, restored here. Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use the existing tensors name Signed-off-by: tc-mb <tianchi_cai@icloud.com> * unused ctx->model.hparams.minicpmv_version Signed-off-by: tc-mb <tianchi_cai@icloud.com> * use n_merge for slice alignment Signed-off-by: tc-mb <tianchi_cai@icloud.com> * borrow wa_layer_indexes for vit_merger insertion point Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix code style Signed-off-by: tc-mb <tianchi_cai@icloud.com> * Update convert_hf_to_gguf.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com> * use filter_tensors and add model.vision_tower Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix chkhsh Signed-off-by: tc-mb <tianchi_cai@icloud.com> * fix type check Signed-off-by: tc-mb <tianchi_cai@icloud.com> --------- Signed-off-by: tc-mb <tianchi_cai@icloud.com> Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

tc-mb added 2 commits April 29, 2026 23:32

Support MiniCPM-V 4.6 in new branch

2635ba4

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

fix code bug

c6c88e4

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

tc-mb requested review from a team and CISC as code owners April 29, 2026 18:50

fix pre-commit

a86c2ca

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

github-actions Bot added documentation Improvements or additions to documentation examples python python script changes labels Apr 29, 2026

CISC reviewed Apr 29, 2026

View reviewed changes

Comment thread convert_hf_to_gguf.py Outdated

Comment thread convert_hf_to_gguf.py Outdated

ngxson requested changes Apr 29, 2026

View reviewed changes

Comment thread tools/mtmd/models/minicpmv.cpp Outdated

Comment thread tools/mtmd/models/models.h Outdated

Comment thread tools/mtmd/clip-model.h Outdated

Comment thread tools/mtmd/mtmd.cpp Outdated

tc-mb added 6 commits April 30, 2026 09:46

fix convert

8a13c00

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

rename clip_graph_minicpmv4_6

21d09f8

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

use new TYPE_MINICPMV4_6

1f78d3f

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

use build_attn to allow flash attention support

8792deb

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

no use legacy code, restored here.

efabdd8

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

use the existing tensors name

06575b7

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

ngxson reviewed Apr 30, 2026

View reviewed changes

Comment thread tools/mtmd/mtmd-image.h Outdated

Comment thread tools/mtmd/mtmd-image.h Outdated

Comment thread convert_hf_to_gguf.py Outdated

Comment thread convert_hf_to_gguf.py Outdated

Comment thread convert_hf_to_gguf.py Outdated

Comment thread convert_hf_to_gguf.py Outdated

tc-mb added 3 commits May 1, 2026 23:38

unused ctx->model.hparams.minicpmv_version

c2b4e01

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

use n_merge for slice alignment

cddd674

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

borrow wa_layer_indexes for vit_merger insertion point

96996d7

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

ngxson reviewed May 4, 2026

View reviewed changes

Comment thread tools/mtmd/models/minicpmv.cpp Outdated

Comment thread tools/mtmd/models/minicpmv.cpp Outdated

Comment thread tools/mtmd/models/minicpmv.cpp Outdated

fix code style

2c3e6a2

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

ngxson approved these changes May 4, 2026

View reviewed changes

ngxson requested a review from CISC May 4, 2026 14:14

merge upstream master

cd59ba1

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

CISC reviewed May 6, 2026

View reviewed changes

Comment thread convert_hf_to_gguf.py Outdated

Comment thread convert_hf_to_gguf.py Outdated

tc-mb and others added 2 commits May 6, 2026 22:07

Update convert_hf_to_gguf.py

1c2cf30

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

use filter_tensors and add model.vision_tower

72eeefc

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

CISC linked an issue May 6, 2026 that may be closed by this pull request

Misc. bug: convert_hf_to_gguf.py - The BPE pre-tokenizer was not recognized” - Qwen3.5-4B-Base #22700

Closed

fix chkhsh

522ebb7

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

CISC approved these changes May 6, 2026

View reviewed changes

fix type check

76219ec

Signed-off-by: tc-mb <tianchi_cai@icloud.com>

CISC approved these changes May 6, 2026

View reviewed changes

CISC merged commit 2496f9c into ggml-org:master May 6, 2026
49 of 50 checks passed

tc-mb deleted the Support-MiniCPM-V-4.6 branch May 6, 2026 19:59

Conversation

tc-mb commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

CISC commented May 6, 2026

Uh oh!

tc-mb commented May 6, 2026

Uh oh!

CISC commented May 6, 2026

Uh oh!

tc-mb commented May 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants