mtmd: support pocket-tts - #26871
Merged
Merged
Conversation
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample was built as one convolution and one concat per channel, which floods the graph with small nodes and makes kernel launches dominate the decoder. Fold both cases into the column form the seanet decoder already needs: the general case reshapes the kernel to [IC, K * OC] and matmuls it with the input, the depthwise case batches a matmul over the channels so a step scales its own kernel. A single col2im_1d then scatter-adds the columns back to the signal, with the same shape as before, so the overlap-add tail, the streaming state and the bias are untouched. Generation time per frame drops by 80% on CUDA and by 50% on CPU. The output matches the previous implementation sample for sample, with a correlation of 0.999994 and identical frame counts.
The language packs also tune the end-of-speech padding and the padding of short prompts, next to the temperature already carried in the mmproj: french_24l asks for 8 tail frames instead of the guessed 3, english_2026-01 asks for short prompts to be padded with spaces. Write both in the mmproj as clip.gen.audio.frames_after_eos and clip.gen.audio.pad_short_text, keyed on the pack in the conversion script like the temperature. The loader keeps them optional, so a mmproj without them behaves as before. Map semicolons to commas for every pack instead, the reference only asks for it on three of them and it costs nothing elsewhere. Existing mmproj files must be converted again to carry the two keys. On a long french text the port now lands within 2% of the reference: 22.96s against 23.44s, with the same peak level and the same amount of silence.
github-actions
Bot
added
documentation
Contributor
|
Cool! I am rerunning the same tests on this more up-to-date branch |
ggerganov
approved these changes
Aug 11, 2026
CISC
approved these changes
Aug 11, 2026
ServeurpersoCom
approved these changes
Aug 11, 2026
mlogix
added a commit
to SynoriAI/llama.cpp
that referenced
this pull request
Aug 11, 2026
…, multi-output backend sampling, pocket-tts Merges 45 upstream commits (0865990..ebb546b, b10362-16-gebb546b7e) into synori/llama-update-mtp-fit. Zero conflicts; all vendored patches carried over untouched. Primary motivation — new Meta architecture: * 62bf73d model: Muse Glimmer Support (ggml-org#26841) LLM_ARCH_MUSE_GLIMMER + src/models/muse-glimmer.cpp + the mtmd vision tower in tools/mtmd/models/muse-glimmer.cpp and conversion/muse_glimmer.py. Other notable changes that touch our public API surface: * dd1ea52 llama : support multi-output backend sampling (ggml-org#25532) llama_context_params gains n_outputs_max_per_seq; llama_sampler_i.backend_init takes it as a third argument; new backend_reset / copy_state vtable slots and llama_sampler_copy(). * 153d324 llama : default load-mode auto, avoids mmap on iGPUs (ggml-org#26081) llama_load_mode gains LLAMA_LOAD_MODE_AUTO = -1 (enum is now signed). * 6e62ba5 mtmd: support pocket-tts (ggml-org#26871) mtmd_gen_inp/mtmd_gen_out gain seed/temp/feats/is_eos; new mtmd_gen_inp_default(); mtmd_helper_gen_audio_step_gen() takes out_stop. * 157b81f model : Granite-Switch Architecture (ggml-org#25107) * 7a20b41 model: MTP support for Nemotron (ggml-org#26725) and cc078b4 Dflash support for nemotron-3.5 (ggml-org#26905) * e23e944 vendor : cpp-httplib 0.53.0, 4c6766f vendor : subprocess.h sync Vendored patches preserved: * 919fde3 feat(rpc): thread-local last_error accessor — intact, upstream touched ggml-rpc.cpp by one unrelated line. * 3679b23 Fixes (RPC) — intact. * 2aa76c7 fix(metal): drop stray kernel_pad_f32 — still applies; upstream has since refactored pad into a templated kernel_pad_impl<T>, so the duplicate definition that referenced the nonexistent kargs_pad.s0..s3 is gone on both sides and nothing had to be re-applied.
1 task
huaxel
pushed a commit
to huaxel/CachyLLama
that referenced
this pull request
Aug 12, 2026
* adapt the api * text model ok * working impl, need verify and clean up * mtmd: build the pocket-tts transposed convolutions as GEMM + col2im ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample was built as one convolution and one concat per channel, which floods the graph with small nodes and makes kernel launches dominate the decoder. Fold both cases into the column form the seanet decoder already needs: the general case reshapes the kernel to [IC, K * OC] and matmuls it with the input, the depthwise case batches a matmul over the channels so a step scales its own kernel. A single col2im_1d then scatter-adds the columns back to the signal, with the same shape as before, so the overlap-add tail, the streaming state and the bias are untouched. Generation time per frame drops by 80% on CUDA and by 50% on CPU. The output matches the previous implementation sample for sample, with a correlation of 0.999994 and identical frame counts. * flow_temp + frames_after_eos * chunking * mtmd: carry the remaining pocket-tts per-pack settings The language packs also tune the end-of-speech padding and the padding of short prompts, next to the temperature already carried in the mmproj: french_24l asks for 8 tail frames instead of the guessed 3, english_2026-01 asks for short prompts to be padded with spaces. Write both in the mmproj as clip.gen.audio.frames_after_eos and clip.gen.audio.pad_short_text, keyed on the pack in the conversion script like the temperature. The loader keeps them optional, so a mmproj without them behaves as before. Map semicolons to commas for every pack instead, the reference only asks for it on three of them and it costs nothing elsewhere. Existing mmproj files must be converted again to carry the two keys. On a long french text the port now lands within 2% of the reference: 22.96s against 23.44s, with the same peak level and the same amount of silence. * clip.gen.audio.model_variant * clean up code comments * nit: drop the dead flow_temp hparam, the pack table holds the default * update docs * address security problems * less invasive base.py * lint * add mtmd_gen_inp_default * add docs * rm gen_flow_temp --------- Co-authored-by: Pascal <admin@serveurperso.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Most of the discussions happen here: ngxson#108
We decide to support this model after initial Qwen3-TTS support, because it starts to move away from discrete audio codes to passing continuous embd. Supporting continuous embd between stages will be important for future models like chatterbox.
Some decisions / trade-off are taken to make sure the impl isn't too invasive and doesn't add too much complexity for future models:
mtmd_helper_gen_audiomanages the stop condition based on pipeline type. This is because models based on continuous embd doesn't use samplingRequirements