Xsn/pocket tts - #108
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
|
Okay, we just need to add chunking like the reference code No chunking: With chunking: |
|
Cleaned LLM questions for Target architecture : Tried this on long texts with the real gated weights and hit two failure modes: in english the pitch drifts down and the tail turns into structured noise, in french the text just gets cut halfway. Turns out the reference never feeds a whole text to the model, it splits into chunks of max 50 tokens, restarts each chunk from the voice prompt and bounds each one with a budget derived from the token count. I have a patch that does this and it fixes both cases, but a few design calls are yours to make:
Two smaller things while I was at it: the converter raises KeyError on flow_lm.bos_before_voice for english_2026-01 and for the older root revision, and the pipeline only takes a wav reference so the precomputed embeddings shipped in the repo can't be used as voices. |
ggml_conv_transpose_1d has no grouped mode, so the depthwise upsample was built as one convolution and one concat per channel, which floods the graph with small nodes and makes kernel launches dominate the decoder. Fold both cases into the column form the seanet decoder already needs: the general case reshapes the kernel to [IC, K * OC] and matmuls it with the input, the depthwise case batches a matmul over the channels so a step scales its own kernel. A single col2im_1d then scatter-adds the columns back to the signal, with the same shape as before, so the overlap-add tail, the streaming state and the bias are untouched. Generation time per frame drops by 80% on CUDA and by 50% on CPU. The output matches the previous implementation sample for sample, with a correlation of 0.999994 and identical frame counts.
|
Small perf commit using GEMM + COL2IM_1D: the transposed convolutions become one matmul plus a single col2im scatter-add, so the depthwise upsample stops emitting one node per channel : |
|
Testing french_24l I hit another one: the flow temperature and the eos padding are per language pack in the reference, and hardcoding 0.3 for everyone makes the french model draw its noise at the wrong scale. On some seeds the eos head then never fires and one sentence runs to the generation budget, ending on seven seconds of silence. Only english and english_2026-04 are at 0.3, french_24l keeps 0.7 and asks for 8 padding frames. My patch carries both in the mmproj as clip.gen.audio.flow_temperature and clip.gen.audio.frames_after_eos, written by the converter, optional on the loader side. Worst seed on french goes from 155 frames with 7.28s of silence to 95 frames with 0.96s. It needs existing mmproj files to be converted again, so I am holding off until you say whether you want these as GGUF metadata at all. |
The language packs also tune the end-of-speech padding and the padding of short prompts, next to the temperature already carried in the mmproj: french_24l asks for 8 tail frames instead of the guessed 3, english_2026-01 asks for short prompts to be padded with spaces. Write both in the mmproj as clip.gen.audio.frames_after_eos and clip.gen.audio.pad_short_text, keyed on the pack in the conversion script like the temperature. The loader keeps them optional, so a mmproj without them behaves as before. Map semicolons to commas for every pack instead, the reference only asks for it on three of them and it costs nothing elsewhere. Existing mmproj files must be converted again to carry the two keys. On a long french text the port now lands within 2% of the reference: 22.96s against 23.44s, with the same peak level and the same amount of silence.
|
Two things I need from you before cleaning this up. First, the fix adds two fields to mtmd_gen_audio_info so the pipeline can see the per-pack tail length and the short-prompt padding, and that's the only place it widens the public API, so tell me if you'd rather have it some other way. Second, still open from earlier: should the 50 token chunk budget and the 12.5 Hz frame rate come from the GGUF, or stay hardcoded on the cpp side? |
IIRC it has always been the case for mimi encoder / decoder series so might be ok to hard-code it if future model has different values, we can consider the hard-coded value as default and add gguf metadata to overwrite the default |
| self.gguf_writer.add_gen_audio_frames_after_eos( | ||
| _PACK_FRAMES_AFTER_EOS.get(self.dir_model.name, 0)) | ||
| self.gguf_writer.add_gen_audio_pad_short_text( | ||
| _PACK_PAD_SHORT_TEXT.get(self.dir_model.name, False)) |
There was a problem hiding this comment.
this logic seems to be quite fragile, it's better to add a new metadata clip.gen.audio.model_variant and set it to self.dir_model.name
I'll push a commit for this change
|
@ServeurpersoCom looks better now? |
EnglishRef python : llama.cpp MTMD : FrenchRef python : llama.cpp MTMD : Sound good ! |
|
nice, thanks for testing. I'll try to push this branch as upstream PR today |
2 participants
For discussion