Add xcodec2 model - #44178
Merged
Merged
Conversation
Contributor
Author
|
run-slow: wav2vec2_bert |
Contributor
|
This comment contains models: ["models/wav2vec2_bert", "models/xcodec2"] |
Contributor
CI ResultsCommit Info
Model CI Report❌ 2 new failed tests from this PR 😭
|
ebezzam
commented
Jun 24, 2026
| self.tdnn_kernel = tdnn_kernel | ||
| self.tdnn_dilation = tdnn_dilation | ||
| self.xvector_output_dim = xvector_output_dim | ||
| self.position_embeddings_type = position_embeddings_type |
Contributor
Author
There was a problem hiding this comment.
not being used, it is rather passed to the method for preparing the config to try different types (see here)
Contributor
Author
|
run-slow: wav2vec2_bert |
Contributor
|
This comment contains models: ["models/wav2vec2_bert", "models/xcodec2"] |
Contributor
Author
|
run-slow: wav2vec2_bert |
Contributor
|
This comment contains models: ["models/wav2vec2_bert", "models/xcodec2"] |
Contributor
CI ResultsCommit Info
Model CI Report❌ 1 new failed tests from this PR 😭
|
Contributor
Author
|
run-slow: wav2vec2_bert |
Contributor
|
This comment contains models: ["models/wav2vec2_bert", "models/xcodec2"] |
Contributor
CI ResultsCommit Info
Model CI Report❌ 1 new failed tests from this PR 😭
|
Contributor
Author
|
run-slow: wav2vec2_bert |
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, dac, encodec, higgs_audio_v2_tokenizer, mimi, pe_audio, qwen2_5_omni, qwen3_omni_moe, seamless_m4t, wav2vec2_bert, xcodec, xcodec2 |
Contributor
|
This comment contains models: ["models/wav2vec2_bert", "models/xcodec2"] |
ebezzam
enabled auto-merge
June 25, 2026 07:38
stevhliu
pushed a commit
to stevhliu/transformers
that referenced
this pull request
Jul 30, 2026
* Add xcodec model * code formatting * typo xcodec2 name * add xcodec2 in init file * fix import * fix weight_norm init * remove unused import * add convert file * add ModelOutput class * nit * fix device issue * fix forward * nit * doc draft * draft test * match tensor with the orignal implementation * Add doc file for xcodec2 * finish model doc for xcodec2 * update doc * working xcodec2 * add test file for xcodec2 * nit * xcodec2 use EncodecFeatureExtractor * Standardize with Xcodec. * Address some PR comments and standardize. * Remove Sequential. * Remove weight norm from model definition. * Remove padding. * Better use of modular and better init. * Style and format checks. * Address some modeling tests. * Better use of modular for Attention and cleanup. * Remove asserts, expose params in config. * Clean up internal and better docstrings. * Fix example. * Make clear how self.causal is used * Add padding consistent to original. * Clean up and matching integration test. * Update docs and clean up. * Correct doc location among audio models. * Remove flash tests. * Skip flash attention tests. * Switch to processor. * New feature extractor. * Update modular. * Add feature extractor. * Clean feature extractor and expose some parameters. * Clean up feature extractor and add torch support (wip). * Clean up seamless feature extractor. * Modular cleanup. * Remove processor. * Update docs. * Fix copy paths. * Fix modeling tests for audio_spectrogram input. * Fix torch support with new spectrogram torch utility. * Repo consistency. * Feature extraction tests. * Make style happy * Remove unprotected import. * Another unprotected import. * Remove more unprotected torches. * zero_mean_unit_var_norm needed for a test * Modify Vocos component to be able to use modular later. * Update modular * Make style happy. * Regenerate modular, update rope config. * Update init weights * Add to init weights * FIxed _init_weights * Integration passing! * Cleanup * Better use of modular, and cleanup. * Update config after mering with main, and other nits. * Address tests and other nits. * Nits, better wav2vec2 bert init * CLean up feature extraction and other nits * doc nits * Nit * Simplify feature extraction. * Torch compiile compatibility. * Rename output entries, semantic encoder. Move weight norm utilities. Only keep necessary layers of semantic encoder. * Add batch test and spectrogram padding for equivalence. * Refactoring. * Remove defensive trimming. * Make style happy. * Apply suggestion from @vasqu Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Fix feature extraction tests, remove integration since original doesn't have one and we are implicitly testing with the model output integration tests. * Update date. * Address docs comments. * Apply suggestion from @vasqu Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Apply suggestion from @vasqu Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Address various comments. * Add import for RopeParameters * Address feature extractor comments. * Adding note on spectrogram computation. * Address smaller modeling comments. * Address various modeling comments. * Style fixes after merge. * better api for xcodec forward * Better module naming in Qwen2.5 Omni * Address various comments. * Add comments, add audio utils, address nits. * Keep current audio input names, various nits. * Add casting for cpu * Move utils for testing audio. * Update checkpoint name. * FIx for torch compile * Set test config for offload to pass. * Increase allowed num layers. * Config that passes cpu offload. * test config for disk/cpu offload * test nits * update date --------- Co-authored-by: Deep-Unlearning <steven@macbook-pro-de-steven.taildb5d.ts.net> Co-authored-by: Deep-unlearning <steven@huggingface.co> Co-authored-by: Deep-unlearning <Deep-unlearning> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Re-opening #37868
TODO
Original checkpoint: https://huggingface.co/HKUSTAudio/xcodec2
Original modeling code: https://huggingface.co/HKUSTAudio/xcodec2/blob/main/modeling_xcodec2.py