Implement VibeVoice by pengzhiliang · Pull Request #40546 · huggingface/transformers

pengzhiliang · 2025-08-29T12:47:34Z

What does this PR do?

Merge the model from https://github.com/microsoft/VibeVoice/tree/main

HF:
https://huggingface.co/microsoft/VibeVoice-1.5B

ebezzam

@pengzhiliang thanks for the PR! This is an exciting model to add 🔥

My first comments are mainly on rearranging content to be consistent with the other models in Transformers, and creating a modular file to better optimize copying components from other models in Transformers.

There are also some other files to modify:

in src/transformers/models/auto
in docs
and eventually some tests (for which a lot of code can be copied from other models)

As an example of typical files to create/modify, you can check out the Qwen2.5-Omni PR, which is also multimodal.

fakerybakery · 2025-09-06T22:37:22Z

Hopefully this PR can get merged, since the deletion of the VibeVoice repo not sure the original authors can continue contributing to this PR

fakerybakery · 2025-10-15T01:33:05Z

Glad to see that someone has picked this PR up, thanks 🤗

github-actions · 2026-07-22T09:52:42Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-22T09:53:02Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-22T09:53:24Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-22T10:30:03Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-22T10:30:53Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-22T15:05:51Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-22T15:06:01Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-22T16:13:43Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

ebezzam

@eustlb an idea for getting the desired labels for training via the chat template.

I'm not super convinced (also because we don't have a precedent) but happy to get your thoughts!

ebezzam · 2026-07-22T16:16:43Z

+        # Default to the feature extractor's own sampling rate
+        kwargs.setdefault("sampling_rate", self.feature_extractor.sampling_rate)


ugly hack to avoid the warning:

[transformers] It is strongly recommended to pass the `sampling_rate` argument to `VibeVoiceAcousticTokenizerFeatureExtractor()`. Failing to do so can result in silent errors that might be hard to debug.

because _defaults needs to be empty to satisfy this new style check

You can add a trf ignore instead of this workaround

thanks, and I wonder if we really need to keep this type of guard for future feature extractors? otherwise new models will always need this ignore

cc @eustlb maybe something for #44394

We need it, it's an opinionated choice that's been made and that is still of great importance IMO. It means that the style check will have to get adapted to take this into account thought

ebezzam · 2026-07-22T16:29:36Z

+            # NOTE: mask padding through the attention mask, as the padding and EOS token share the same ID
+            if "attention_mask" in data:
+                labels[data["attention_mask"] == 0] = -100


because padding and EOS tokens are the same as per Qwen2 here

ebezzam · 2026-07-22T16:37:25Z

+{%- if not add_generation_prompt %}
+{%- if has_target_audio %}{{ audio_token }}{{ audio_eos_token }}{% endif %}{{ eos_token }}
+{%- endif %}"""


one idea for distinguishing between training/inference regime

add_generation_prompt=True (inference) -> no eos_token

add_generation_prompt=False (training) -> append eos_token

here is a script to see the chat template rendering before tokenization.

there is one TRL chat template that does this, but in reality TRL doesn't have a strict requirement on the chat template as they often create their own to make sure they are compatible with training. See their collection of chat templates here and guidelines here.

although I'm still not sure if this is the best approach. As Higgs still sets add_generation_prompt=True for training here

add_generation_prompt=True on higgs shouldn't be there, nice catch !!
nevertheless we got lucky there because eos comes from the codebooks, so this has no detrimental effect

ahhhhh it's really tricky here, I'd be tempted to say yes, let's use add_generation_prompt for that but it means overloading a parameter for a behavior it was not intended for...

Nevertheless TRL does use it to that end, so I am ok with that!

ebezzam · 2026-07-22T16:43:29Z

+            # Processor kwargs are passed separately from Jinja2 template kwargs.
+            processor_kwargs = kwargs.pop("processor_kwargs", None) or {}
+            chat_template_kwargs = {
+                "tokenize": True,
+                "return_dict": True,
+                "add_generation_prompt": True,
+                "processor_kwargs": processor_kwargs,
+                **kwargs,
+            }
            output = preprocessor.apply_chat_template(
                text.messages,
-                tokenize=True,
-                return_dict=True,
-                **kwargs,
+                **chat_template_kwargs,


(this file in general needs a revamp, but I think worth adding things as we see them?)

We should be passing "add_generation_prompt": True? As do all of Higgs audio examples and if we do go with using add_generation_prompt to distinguish between training/inference for Vibevoice

as well as passing processor_kwargs separately

github-actions · 2026-07-22T17:08:30Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-22T17:32:08Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-24T15:16:29Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-24T15:18:01Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-24T15:18:45Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

eustlb

Clean 👏👏👏 Great work!!

eustlb · 2026-07-27T08:15:47Z

    [conversation1, conversation2],
-    add_generation_prompt=True,
    tokenize=True,


eustlb · 2026-07-27T08:39:20Z

+        if not is_soundfile_available():
+            raise ImportError("Please install `soundfile` to save audio files.")


updating to main you'll be able to use requires_backends(self, ["soundfile"])

github-actions · 2026-07-27T08:58:37Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-27T08:59:29Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-27T09:07:25Z

Thank you for your contribution 🤗!

CI Security Gate — automatic approval blocked

This PR was not automatically approved for CI because the security gate failed.

Possible reasons:

The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
A changed file is outside the allowed directories (src/, tests/, docs/, utils/), has a disallowed extension (only .py, .txt, .md permitted outside tests/ and docs/), or is not .md/.yml inside docs/
A new high-severity security issue was detected in the changed Python files (Bandit check)

See the workflow run for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

github-actions · 2026-07-27T09:08:19Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto, csm, vibevoice, vibevoice_acoustic_tokenizer

github-actions · 2026-07-27T09:34:15Z

CI recap

Dashboard: View test results in Grafana
Latest run: 30251954355:2
Result: success | Jobs: 16 | Tests: 175,446 | Failures: 1 | Duration: 9h 49m

ebezzam added New model Audio labels Aug 29, 2025

ebezzam requested changes Aug 29, 2025

View reviewed changes

twobob mentioned this pull request Sep 5, 2025

[Feature] [New Model]: VibeVoice-1.5B : A Frontier Long Conversational Text-to-Speech Model sgl-project/sglang#9697

Closed

2 tasks

Galigator added a commit to Galigator/transformers that referenced this pull request Oct 8, 2025

Apply PR huggingface#40546

dca2cc5

ebezzam added 8 commits October 13, 2025 18:37

Remove audio tokenizer from main model.

396bb2c

Separate semantic model.

fa6100c

Switch to modular and remove debug path.

24e9f13

Start conversion script for main model, clean up semantic modeling.

84c27ad

Update semantic modular.

d7d409e

Single conversion script.

06313d0

Clean up semantic tokenizer modeling.

4af74e9

Update modular.

47fdd93

ebezzam added 15 commits October 15, 2025 14:23

More semantic cleanup.

8de03e7

Cleanup and refactor.

8e69a36

Cleanup and standardization.

8a6b033

Merge branch 'main' into main

8771cb3

More cleanup and standardization.

4504fe1

Some simplification.

b44c3a8

Cleaner streaming conv.

62d5da1

Nits.

e8fa0ae

Nits.

42fa4df

Cache like past_key_values

627ce76

Further cleanup and optimization.

f2bfc71

Update modular and generate.

446a19c

Standardize config.

cd26b19

Update acoustic tokenizer config and encoder.

f41d701

Switch to new acoustic cache object.

56e9eb7

Merge branch 'main' of github.com:pengzhiliang/transformers into main

f36c334

Address modular comments.

957902f

Merge branch 'main' into main

7485602

Update chat template to handle training regime

f7f807e

ebezzam reviewed Jul 22, 2026

View reviewed changes

Functional approach for weightless rms norm.

e982a71

Add trf-ignore

8ecb0ce

Update docs.

8be50ea

Merge branch 'main' into main

8503b4a

eustlb approved these changes Jul 27, 2026

View reviewed changes

Merge branch 'main' into main

e3d7755

Switch to requires_backend

a8604db

		# Default to the feature extractor's own sampling rate
		kwargs.setdefault("sampling_rate", self.feature_extractor.sampling_rate)

		if not is_soundfile_available():
		raise ImportError("Please install `soundfile` to save audio files.")

Uh oh!

Conversation

pengzhiliang commented Aug 29, 2025 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

ebezzam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fakerybakery commented Sep 6, 2025

Uh oh!

fakerybakery commented Oct 15, 2025

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

ebezzam left a comment

Choose a reason for hiding this comment

Uh oh!

ebezzam Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ebezzam Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 22, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 24, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 24, 2026

CI Security Gate — automatic approval blocked

Uh oh!

github-actions Bot commented Jul 24, 2026

pengzhiliang commented Aug 29, 2025 •

edited by github-actions Bot

Loading

ebezzam Jul 22, 2026 •

edited

Loading

ebezzam Jul 22, 2026 •

edited

Loading