fix: AttributeError for Qwen3_omni_moe by Vallabh-1504 · Pull Request #43593 · huggingface/transformers

Vallabh-1504 · 2026-01-29T13:01:01Z

What does this PR do?

This PR fixes a crash when initializing Qwen3OmniMoeTalkerCodePredictorConfig due to a missing attribute reference.

Specifically, it:

Removes the reference to the non-existent use_sliding_window attribute, which was causing an AttributeError.
Adds the missing max_window_layers initialization (defaulting to 28). The existing layer_types logic relies on this attribute, and without it, sliding_window causes a secondary crash.
Adds a new test case (test_code_predictor_config_init) to ensure the configuration initializes correctly.

Fixes #43531

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case. (Issue sliding_window issue with Qwen3-MoE models #43531)
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Rocketknight1 · 2026-01-29T15:06:48Z

cc @vasqu since you commented on the issue!

vasqu

max_window_layers seems like an unnecessary attribute added + let's move the test, please no new files

vasqu · 2026-01-29T15:51:49Z

        rope_parameters: int | None = None,
        attention_bias: bool | None = False,
        sliding_window: int | None = None,
+        max_window_layers: int = 28,


Seems irrelevant?

vasqu · 2026-01-29T15:52:09Z

The test should be still under test_modeling_xxx

Vallabh-1504 · 2026-01-29T16:34:22Z

hi, @vasqu, thanks for the review.
when I was running pytest I was getting an error regarding max_window_layers.
This attribute is used in the layer_types list comprehension (around line 612).

    def __getattribute__(self, key):
        if key != "attribute_map" and key in super().__getattribute__("attribute_map"):
            key = super().__getattribute__("attribute_map")[key]
>       return super().__getattribute__(key)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       AttributeError: 'Qwen3OmniMoeTalkerCodePredictorConfig' object has no attribute 'max_window_layers'

src\transformers\configuration_utils.py:163: AttributeError

vasqu · 2026-01-29T16:59:28Z

        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
-        self.sliding_window = sliding_window if self.use_sliding_window else None


Ok I checked what happened and it seems it was unintentionally added in #41541, but since I'm not super familiar with this model I'd rather wait for @zucchini-nlp to answer here

Imo we should just do self.sliding_window = sliding_window (use_sliding_window was never used at all and should be removed from the docstring) - max_window_layers should be removed alongside it (not reintroduced)

Also the changes should be done in modular and then reapplied via python utils/modular_model_converter.py qwen3_omni_moe

Indeed a bad copy, no need for use_sliding_window. Model always uses sliding layers together with full attention

zucchini-nlp · 2026-01-30T08:46:53Z

        **kwargs,
    ):
        self.sliding_window = sliding_window
+        self.max_window_layers = max_window_layers


it is a recurring pattern in many qwen-multimodal models and I think authors incorrectly copied it when adding Qwen3-Omni-MoE. The official checkpoints have max_window_layers saved in config therefore we don't see errore at inference

IMO we need to keep it to match the model's default behavior and to match with docstring

zucchini-nlp · 2026-02-03T09:41:41Z

@Vallabh-1504 hey, any updates on the PR? Would be great to merge it before the next planned release, which I believe should be around this week

Vallabh-1504 · 2026-02-03T14:54:32Z

hi @zucchini-nlp, I was working on it but hitting a bit of a wall.

I was applying changes through modular_qwen3_omni_moe.py, as previously i did manual changes which was wrong. but for some reason, the modular_model_converter.py keeps regenerating the use_sliding_window logic in the final config, even after i removed it from modular_qwen3_omni_moe.py.

kinda stuck in a loop where the generator keeps the code i'm trying to remove.

I also don't know what to do with the max_window_layers, as this is also not initialized anywhere.

Can you guide me through this. If i'm missing a step or if there's some cache with the converter?

zucchini-nlp · 2026-02-03T15:04:52Z

Smth like this should work, we need to delete unused attributed explicitly and re-assign the "differing" attr. For max_window_layers, let's keep it without deleting

https://github.com/Vallabh-1504/transformers/blob/9041720191cdef9348f09f7c1695db606223d353/src/transformers/models/qwen3_omni_moe/modular_qwen3_omni_moe.py#L571-L572

Vallabh-1504 · 2026-02-03T16:55:39Z

@zucchini-nlp, I have applied the changes as requested!

zucchini-nlp

One tiny comment and let's wait for @vasqu's review

zucchini-nlp · 2026-02-03T17:31:08Z

        self.num_code_groups = num_code_groups
        self.vocab_size = vocab_size
        self.max_position_embeddings = max_position_embeddings
        self.hidden_size = hidden_size
        self.intermediate_size = intermediate_size
        self.num_hidden_layers = num_hidden_layers
        self.num_attention_heads = num_attention_heads
-        self.sliding_window = sliding_window if self.use_sliding_window else None
+        self.sliding_window = sliding_window
+        self.max_window_layers = max_window_layers


nit: duplicate

zucchini-nlp · 2026-02-03T17:31:36Z

@@ -473,6 +475,7 @@ def __init__(
        **kwargs,
    ):
        self.sliding_window = sliding_window
+        self.max_window_layers = max_window_layers


not needed, super assigns it already which is why we got duplicates in auto-generated code :)

zucchini-nlp · 2026-02-03T17:35:58Z

+class TestQwen3OmniMoeCodePredictorConfig(unittest.TestCase):
+    def test_code_predictor_config_init(self):
+        """
+        Test that Qwen3OmniMoeTalkerCodePredictorConfig initializes correctly
+        and accepts max_window_layers while removing use_sliding_window.
+        """
+
+        config = Qwen3OmniMoeTalkerCodePredictorConfig(


ideally we need a complete model test for the 'TalkerModel' model. But I realize that the model doesn't follow transformers standards and we'll skip anyway many tests from ModelTesterMixin, or override a lot of them

I'm fine with deleting the current test in that case, @vasqu wdyt?

Imo, we can add a small regression test under

transformers/tests/models/qwen3_omni_moe/test_modeling_qwen3_omni_moe.py

Line 61 in d75ecdb

class Qwen3OmniMoeThinkerForConditionalGenerationTester:

at least, we don't need a separate class for this

Just a tad weird because the naming is weird but if we don't test it we ought to repeat it in a refactor 😬

HuggingFaceDocBuilderDev · 2026-02-03T17:40:45Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

vasqu

Don't have much to add onto @zucchini-nlp's comments, I'd just prefer to move the test under the general tester class

vasqu · 2026-02-03T18:29:48Z

+class TestQwen3OmniMoeCodePredictorConfig(unittest.TestCase):
+    def test_code_predictor_config_init(self):
+        """
+        Test that Qwen3OmniMoeTalkerCodePredictorConfig initializes correctly
+        and accepts max_window_layers while removing use_sliding_window.
+        """
+
+        config = Qwen3OmniMoeTalkerCodePredictorConfig(


Imo, we can add a small regression test under

transformers/tests/models/qwen3_omni_moe/test_modeling_qwen3_omni_moe.py

Line 61 in d75ecdb

class Qwen3OmniMoeThinkerForConditionalGenerationTester:

at least, we don't need a separate class for this

Just a tad weird because the naming is weird but if we don't test it we ought to repeat it in a refactor 😬

github-actions · 2026-02-04T10:26:19Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: qwen3_omni_moe

## Description  The main reason is huggingface/transformers#43593. I've checked the changelog and run it through the agent - no breaking changes for WWB or LLM_bench were introduced. Changes to llm_bench hooks with the new transformers: **v5.4.0** — _get_initial_cache_position was removed and _prefill stopped writing cache_position; the inputs_embeds slicing in _prefill switched to computing next_sequence_length instead of mutating model_kwargs["inputs_embeds"] (https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/generation/utils.py#L3852). **v5.11.0** — _sample gained pad_token_id = pad_token_id.to(input_ids.device) right after unfinished_sequences is created (https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/generation/utils.py#L2760). **v5.13.0** — _beam_search's end-of-iteration cache reorder now scans ALL_CACHE_NAMES, has an elif hasattr(cache, "reorder_cache") branch, and raises ValueError for unreorderable caches (https://github.com/huggingface/transformers/blob/v5.13.0/src/transformers/generation/utils.py#L3185). <details> <summary>Diffs:</summary> 5.3 vs 5.4 ```python 8c8 < self, --- > self: "GenerativePreTrainedModel", 32,34c32 < # Always directly slice the inputs_embeds if present, as `prepare_inputs_for_generation` never need them full and `_get_initial_cache_position` < # rely on its size explicitly. For input_ids, we need to use `next_sequence_length` to slice later instead of explicit slicing, < # as some model need them full for correct input preparation inside `prepare_inputs_for_generation` (i.e. audio models) --- > # It will be sliced as input_embeds = inputs_embeds[:, -next_sequence_length:, :] in `prepare_inputs_for_generation` 36c34 < model_kwargs["inputs_embeds"] = inputs_embeds[:, past_length:, :] --- > next_sequence_length = model_kwargs["inputs_embeds"].shape[1] - past_length 47,49d44 < # The cache is already taken into account in `_get_initial_cache_position`, so the length is only the new tokens if we slice < effective_input_length = next_sequence_length if next_sequence_length is not None else input_ids.shape[1] < model_kwargs = self._get_initial_cache_position(effective_input_length, input_ids.device, model_kwargs) 62c57 < torch._dynamo.config.cache_size_limit = 64 --- > getattr(torch, "_dynamo").config.cache_size_limit = 64 85,87d79 < model_kwargs["cache_position"] = torch.arange( < past_length, current_length, dtype=torch.long, device=input_chunk.device < ) 97,99d88 < model_kwargs["cache_position"] = torch.arange( < input_ids.shape[1], dtype=torch.long, device=input_ids.device < ) 107c96 < self, --- > self: "GenerativePreTrainedModel", 293c282 < self, --- > self: "GenerativePreTrainedModel", ``` 5.8 vs 5.9 (comment): ```python 32c32 < # It will be sliced as input_embeds = inputs_embeds[:, -next_sequence_length:, :] in `prepare_inputs_for_generation` --- > # It will be sliced as inputs_embeds = inputs_embeds[:, -next_sequence_length:, :] in `prepare_inputs_for_generation` ``` 5.10 vs 5.11: ```python 164a165,171 > > # `pad_token_id` is created on `inputs_tensor.device` in `_prepare_special_tokens`. For multimodal models > # (e.g. BLIP-2, LLaVA) sharded across devices via `device_map="auto"`, `inputs_tensor` (e.g. `pixel_values` > # on the vision encoder) and `input_ids` (on the language model) can live on different devices, so we need to > # realign `pad_token_id` with `input_ids` to avoid cross-device ops below. > if pad_token_id is not None: > pad_token_id = pad_token_id.to(input_ids.device) ``` 5.12 vs 5.13: ```python 559c559,560 < if model_kwargs.get("past_key_values") is not None: --- > if any(cache_key in model_kwargs for cache_key in ALL_CACHE_NAMES): > cache_key = next(cache_key for cache_key in ALL_CACHE_NAMES if cache_key in model_kwargs) 562c563,565 < model_kwargs["past_key_values"] = self._reorder_cache(model_kwargs["past_key_values"], beam_idx) --- > model_kwargs[cache_key] = self._reorder_cache(model_kwargs[cache_key], beam_idx) > elif hasattr(model_kwargs[cache_key], "reorder_cache"): > model_kwargs[cache_key].reorder_cache(beam_idx) 564c567,569 < model_kwargs["past_key_values"].reorder_cache(beam_idx) --- > raise ValueError( > f"{self.__class__.__name__} cannot use beam search with a cache currently, as the cache cannot be reordered" > ) ``` </details> Added new modules that will work with transformers >=5.4, <=5.13. Limitations: Optimum-Intel still supports transformers < 5.1 only. Benchmarks allow that version too, so it is not a breaking change. ## Checklist: - [x] This PR follows [GenAI Contributing guidelines](https://github.com/openvinotoolkit/openvino.genai?tab=contributing-ov-file#contributing).  - [ ] Tests have been updated or added to cover the new code.  - [ ] This PR fully addresses the ticket.  - [ ] I have made corresponding changes to the documentation.

vasqu reviewed Jan 29, 2026

View reviewed changes

zucchini-nlp reviewed Jan 30, 2026

View reviewed changes

zucchini-nlp mentioned this pull request Feb 3, 2026

Add Qwen3-Omni model registration to AutoModel and AutoModelForConditionalGeneration classes #43684

Closed

zucchini-nlp added the for patch Tag issues / labels that should be included in the next patch label Feb 3, 2026

correct-fix-through-modular

d75ecdb

Vallabh-1504 force-pushed the fix-attribute-error branch from 9041720 to d75ecdb Compare February 3, 2026 16:51

zucchini-nlp approved these changes Feb 3, 2026

View reviewed changes

vasqu approved these changes Feb 3, 2026

View reviewed changes

lvliang-intel mentioned this pull request Feb 4, 2026

[Feature]: Support Omni model quantization intel/auto-round#1387

Closed

fix-duplicate

2d186c4

zucchini-nlp enabled auto-merge (squash) February 4, 2026 10:36

zucchini-nlp merged commit 257a602 into huggingface:main Feb 4, 2026
19 checks passed

avasenin-14 mentioned this pull request Jul 8, 2026

[wwb] Bump transformers version to <=5.13 openvinotoolkit/openvino.genai#4115

Merged

4 tasks

Uh oh!

Conversation

Vallabh-1504 commented Jan 29, 2026

What does this PR do?

Before submitting

Who can review?

Uh oh!

Rocketknight1 commented Jan 29, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vallabh-1504 commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp Jan 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Feb 3, 2026

Uh oh!

Vallabh-1504 commented Feb 3, 2026

Uh oh!

zucchini-nlp commented Feb 3, 2026

Uh oh!

Vallabh-1504 commented Feb 3, 2026

Uh oh!

zucchini-nlp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Feb 3, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Feb 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Vallabh-1504 commented Jan 29, 2026 •

edited

Loading

zucchini-nlp Jan 30, 2026 •

edited

Loading