[pipelines] text-to-audio pipeline standardization by gante · Pull Request #39796 · huggingface/transformers

gante · 2025-07-30T16:16:03Z

What does this PR do?

⚠️ TODO before merging: after settling on the design, update pipeline usage in model docs accordingly.

This PR standardizes text-to-audio such that the following lines work with most* text-to-audio models:

from transformers import pipeline

synthesiser = pipeline("text-to-audio", "facebook/musicgen-large")
music = synthesiser("A low-fi song with a strong bassline")
synthesiser.save_audio(music, "test.wav")

On most* models where voice control is possible, it is possible to control it through the voice pipeline argument. The valid values for voice are model-dependent, and this argument is documented accordingly

from transformers import pipeline

tts_pipeline = pipeline("text-to-audio", "sesame/csm-1b")
audio = tts_pipeline("I just got bamboozled by my cat.", voice="1")
tts_pipeline.save_audio(audio, "test.wav")

Core changes

Prior to this PR, recent models with text-to-audio capabilities had no pipeline support (e.g. CSM, Dia, Qwen2.5 Omni). There was also not a standardized way to control the voice, if the model generates speech.

With this PR, TextToAudioPipeline:

Uses a processor whenever possible, automatically (as opposed to needing a flag to control it);
Takes a voice argument, which a few models can use out of the box. Whether a model can take voice in the pipeline is specified by properties of the model (if future models have the same properties, they will also have voice support);
Standardizes outputs: ALL models using the pipeline will return {"audio": <np.array with shape (audio_channels, sequence_length)>, "sampling_rate": <int>}. Different models return different array formats, and as a result we can see different saving scripts in the model cards -- the pipeline standardizes it;
Adds a function to save the audio, for convenience. This way, users don't need to learn about soundfile or alternatives. Uses the processor's save_audio whenever it is available.

Model support

Models with out-of-the-box pipeline support:

TTS Models with voice support:
- CSM
- Qwen2.5 Omni
- Bark
TTS Models w/o voice support:
- Dia -- voice is set in the prompt; needs chat templates? (we can hardcode it in the pipeline, though 🤔 )
- FastSpeech2Conformer (model has no voice control)
- SeamlessM4T and variants (model has no voice control)
- Vits (model has no voice control)
TTA Models:
- Musicgen and variants

Models that have special requirements:

SpeechT5 (requires speaker_embeddings argument; we could hide the complexity and accept voice: int, but the voice dataset most commonly used to pull the embeddings from isn't compatible with datasets==4.0.0 💔 )

github-actions · 2025-07-30T16:17:03Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, bark, csm, dia, fastspeech2_conformer, qwen2_5_omni, speecht5

HuggingFaceDocBuilderDev · 2025-07-30T16:29:06Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ebezzam · 2025-07-30T16:58:55Z

+            audio (`dict[str, Any]` or `list[dict[str, Any]]`):
+                The audio returned by the pipeline. The dictionary (or each dictionary, if it is a list) should
+                contain two keys:
+                - `"audio"`: The audio waveform.


Cool to have saving function 👍

Maybe useful to mention that the audio is expected to have shape (nb_channels, audio_length)? (unless self.processor expects something else...)

And are we sure it isn't a tensor? Otherwise can also add something like this?

I guess synthesiser and tts_pipeline try to ensure this dimension order, but maybe useful in case people gets issues saving.

cf this comment

ebezzam · 2025-07-30T17:02:34Z

+            voice (`str`, *optional*):
+                The voice to use for the generation, if the model is a text-to-speech model that supports multiple
+                voices. Please refer to the model docs in transformers for model-specific examples.


How about something more generic like preset or style? At the moment, TTS models mainly make use of this feature (Bark calls it voice_preset) but maybe music generation models may use this feature at one point?

wdyt @eustlb ?

agree! preset is good IMO

ebezzam · 2025-07-30T17:12:47Z

-        # Or we need to postprocess to get the waveform
+                raise ValueError(
+                    f"Unexpected keys in the audio output format: {audio.keys()}. Expected one of "
+                    "`waveform` or `audio`"


and "sequences"

ebezzam · 2025-07-30T17:15:05Z

+                waveform = waveform.unsqueeze(0)
+            if len(waveform.shape) == 2:  # (bsz, audio_length) -> (bsz, audio_channels=1, audio_length)
+                waveform = waveform.unsqueeze(1)
+


do we need a condition if self.audio_channels > 1 but missing batch dimension?

eustlb

Thank you @gante for the good work! 🤗
My comments (open to discussion!!!, more details in the review):

move from voice to preset, that should be passed blindly to the processor. This way, we offload all the logic to the processor, with the advantage of reducing maintenance and keeping in general enough to include new approaches in the future
centralize saving function in audio_utils

eustlb · 2025-07-30T16:33:48Z

        self.parent.assertTrue(torch.allclose(output_from_past_slice, output_from_no_past_slice, atol=1e-3))


-class BarkModelTester:


Strange this was here, that's an easy unbloat! 😅

eustlb · 2025-07-30T16:36:23Z

+
+    @slow
+    @require_torch
+    def test_csm_model(self):


yes yes yes! Thanks for adding it

eustlb · 2025-07-30T16:38:15Z

        output_audio: Optional[bool] = False,
        **kwargs,
-    ) -> Union[GenerateNonBeamOutput, torch.LongTensor]:
+    ) -> Union[CsmGenerateOutput, torch.LongTensor, list[torch.FloatTensor]]:


Nice catch!

eustlb · 2025-07-30T16:44:50Z

+            if audio_value.ndim == 2 and audio_value.shape[0] in (1, 2):
+                # (nb_channels, audio_length) -> (audio_length, nb_channels), as expected by `soundfile`
+                audio_value = np.transpose(audio_value)


I don't get why you're adding this for CSM and Dia. The fact that the models are mono is more of a design choice than a parameter, and such things should not change.

Nevertheless, if we add a save_audio in audio_utils and rely on this function for saving audio in the processor, it will make sense to have it there, WDYT?

Agree with centralizing/standardizing a save_audio function in audio_utils to avoid code duplication in different processors.

Another idea @eustlb, in several audio libraries I've seen the use of an Audio objects to group together useful properties and methods. WDYT? Here's an example below (quickly drafted with ChatGPT):

import soundfile as sf import numpy as np import os class Audio: def __init__(self, waveform: np.ndarray, sample_rate: int, path: str = None): """ Args: waveform: NumPy array of shape (channels, samples). sample_rate: Sampling rate in Hz. path: Optional original file path. """ if waveform.ndim == 1: waveform = waveform[np.newaxis, :] # Convert to (1, samples) elif waveform.shape[0] > waveform.shape[1]: # Assume it's (samples, channels) — transpose to (channels, samples) waveform = waveform.T self.waveform = waveform self.sample_rate = sample_rate self.path = path @classmethod def load(cls, path: str): """ Load audio from file (converted to (channels, samples)). """ if not os.path.isfile(path): raise FileNotFoundError(f"Audio file not found: {path}") data, sample_rate = sf.read(path, always_2d=True) # (samples, channels) waveform = data.T # Transpose to (channels, samples) return cls(waveform, sample_rate, path) def save(self, path: str): """ Save audio to file (converted to (samples, channels)). """ data = self.waveform.T # Convert to (samples, channels) sf.write(path, data, self.sample_rate) self.path = path def duration_seconds(self) -> float: return self.waveform.shape[1] / self.sample_rate # can have others things for spectrogram, MFCC, augmentations, etc which can call functional versions

It can make working audio with much more convenient (not having to pass sample rate each time and channel dimension consistency). And can be used alongside functional approach

eustlb · 2025-07-30T16:57:40Z

+    def _get_sampling_rate(self, sampling_rate: Optional[int] = None) -> Optional[int]:
+        """
+        Get the sampling rate from the model config, generation config, processor, or vocoder. Can be overridden by
+        `sampling_rate` in `__init__`.
+        """


I don't think we should be able to override it. Like the number of channels, that's more of a fixed parameter for a certain model than something that users would be able to play with.

Moreover, I don't see a model where it would make sense to have it in generation config (bark has it but it's silent 🤔).

Sampling rate should live in two places:

processor → to prepare the inputs in the correct sampling rate

config → to indicate on which sampling rate the model has been trained

priority order should therefore be model config > processor (+ special case of the vocoder)

Agree with priority being in model config

Agreed with not allowing overwrites! (the option to overwrite it was present before, so I left it -> I will remove it then 👍 )

eustlb · 2025-07-30T16:58:39Z

+            sampling_rate = self.vocoder.config.sampling_rate

-        if self.sampling_rate is None:
+        if sampling_rate is None:


detail but can be safely removed

Suggested change

if sampling_rate is None:

eustlb · 2025-07-30T17:05:22Z

@@ -43,7 +43,7 @@

 from ...cache_utils import Cache


Trusting you on that one!

eustlb · 2025-07-30T17:09:36Z

+            for attr in ["audio_channels", "num_audio_channels"]:
+                audio_channels = getattr(obj, attr, None)
+                if audio_channels is not None:
+                    break
+        # WARNING: this default may cause issues in the future. We may need a more precise way of detecting the number
+        # of channels.
+        audio_channels = audio_channels or 1


That's ok I think, stereo is exceptional. It's almost always not specified, with convention of being mono in this situation.

eustlb · 2025-07-30T17:17:05Z

+            voice = str(kwargs.pop("voice", "0"))
+            if not voice.isdigit():
+                logger.warning(
+                    f"With {self.model.name_or_path}, the voice pipeline argument must be a digit. Got voice={voice}, "
+                    "using voice=0 instead."
+                )
+                voice = "0"
+            conversation = [{"role": voice, "content": [{"type": "text", "text": text[0]}]}]


hum that's a bit misleading for CSM since the voice id does not refer to a predefined voice but rather a voice given a conventional context (cf this doc example - the model can be fined-tuned to do predefined voices though! cf here).

Here, setting a voice ID without providing a conversational context will just pick a random voice.

In the future I see multiple different ways of setting this voice, mainly either via a custom way of doing voice cloning, either by using a predefined voice identifier (number/ name/ etc). In both situations that's something that will end up being done in the processor.

What we can do is move from voice to preset, which will directly be passed to the processor and actualise current processors __call__ so that they can handle this preset kwarg.

eustlb · 2025-07-30T17:21:36Z

+                output_list.append(output_dict)
+            return output_list
+
+    def save_audio(self, audio: Union[dict[str, Any], list[dict[str, Any]]], path: Union[str, list[str]]):


Can we move this to audio_utils.py?
The best would be to redefine AudioInput type as a dict with keys audio (typed as former AudioInput) and sampling_rate (float) so that we finally move from having audio in the lib as an array with sampling rate. This should not be breaking as it's only type hinting. This might require removing AudioInput in favor of the current type hinting to places where it's currently been used.

See #39796 (comment) for idea on moving away from audio as independent array

vasqu

Just for me so I can make a proper review after vacay :D

Regarding dia with voice - it does not use the voice (encoded into codebooks) within the encoder but as prefix to the decoder. Since it's an encoder-decoder architecture I'm not sure whether a chat template would really be suitable here. However, the processor should be able to take in audio and text to process it into the correct parameters, e.g. input ids and decoder input ids

vasqu · 2025-07-31T17:34:21Z

+    @require_torch
+    def test_dia_model(self):
+        """Tests Dia with the text-to-audio pipeline"""
+        speech_generator = pipeline(task="text-to-audio", model="buttercrab/dia-v1-1.6b", framework="pt")


Suggested change

speech_generator = pipeline(task="text-to-audio", model="buttercrab/dia-v1-1.6b", framework="pt")

speech_generator = pipeline(task="text-to-audio", model="nari-labs/Dia-1.6B-0626", framework="pt")

Let's use the official ckpt imo

gante added 8 commits July 28, 2025 17:57

basic refactor

7788a27

tmp commit

2421868

tmp

33ead43

all pipeline tests green

d7fb429

tmp commit

5714b8b

tmp commit

6ef93c5

mvp

85d9c5b

dia

779e1e9

gante requested review from ebezzam and eustlb July 30, 2025 16:17

ebezzam added the Audio label Jul 30, 2025

ebezzam reviewed Jul 30, 2025

View reviewed changes

eustlb reviewed Jul 30, 2025

View reviewed changes

vasqu reviewed Jul 31, 2025

View reviewed changes

eustlb mentioned this pull request Sep 10, 2025

Fix the issue that csm model cannot work with pipeline mode. #39349

Merged

5 tasks

ebezzam mentioned this pull request Oct 21, 2025

Add Vocos model #39403

Open

ebezzam mentioned this pull request Nov 21, 2025

Fix processor usage + add chat_template support to TTS pipeline, and shift common chat template logic to base class. #42326

Merged

ebezzam mentioned this pull request Nov 28, 2025

Convenient default behavior for pipeline TTS usage. #42473

Merged

evalstate mentioned this pull request Apr 29, 2026

Cumulative feature and defect updates from recent Transformers PRs evalstate/transformers#42

Open

		self.parent.assertTrue(torch.allclose(output_from_past_slice, output_from_no_past_slice, atol=1e-3))


		class BarkModelTester:

	speech_generator = pipeline(task="text-to-audio", model="buttercrab/dia-v1-1.6b", framework="pt")
	speech_generator = pipeline(task="text-to-audio", model="nari-labs/Dia-1.6B-0626", framework="pt")

Uh oh!

Conversation

gante commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Core changes

Model support

Uh oh!

github-actions Bot commented Jul 30, 2025

Uh oh!

HuggingFaceDocBuilderDev commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eustlb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vasqu left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gante commented Jul 30, 2025 •

edited

Loading

ebezzam Jul 31, 2025 •

edited

Loading

vasqu left a comment •

edited

Loading