Fix eager experts for DeepSeekV2 by Rocketknight1 · Pull Request #43288 · huggingface/transformers

Rocketknight1 · 2026-01-14T14:50:19Z

The eager expert computation for DeepSeekV2 inherited from Qwen2MoE, but they pass expert weights differently. Qwen2MoE selects experts before forward(), it passes top_k_weights with shape (num_tokens, num_top_k). DeepSeekV2 does not, so top_k_weights has shape (num_tokens, num_experts). The result is that DeepSeekV2 selected incorrect experts when _experts_implementation == "eager" and the output was garbage.

This PR overrides the forward() method for DeepSeek so we select the right experts!

cc @IlyasMoutawwakil, fixes #43224

HuggingFaceDocBuilderDev · 2026-01-14T14:59:07Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

IlyasMoutawwakil · 2026-01-14T15:09:25Z

thanks @Rocketknight1 i was surprised as well when writing batched_mm and grouped_mm that top_k_weights can have two possible shapes while the eager path seemed very standardized across MoEs. so i added support for both shapes without touching the eager path (assuming that it works fine somehow 😅).

Rocketknight1 · 2026-01-14T15:12:19Z

Yeah, I'm not sure how this ever worked! I wonder how long this bug has existed for

IlyasMoutawwakil · 2026-01-14T15:15:59Z

not for long, i see it was introduced in #42456

Rocketknight1 · 2026-01-14T15:16:03Z

cc @ydshieh more flaky tests here, I think possibly Hub issues again?

Rocketknight1 · 2026-01-14T16:48:49Z

run-slow: deepseek_v2

github-actions · 2026-01-14T16:50:04Z

This comment contains run-slow, running the specified jobs:

models: ["models/deepseek_v2"]
quantizations: []

github-actions · 2026-01-14T16:56:50Z

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

Rocketknight1 · 2026-01-14T16:59:17Z

@IlyasMoutawwakil tests are green! Can you approve it so I can merge?

vasqu · 2026-01-14T17:28:38Z

        super().__init__(config)
        self.num_experts = config.n_routed_experts

+    def forward(


Sorry I have to say that this is likely the wrong fix and will make things incompatible with things like fp8, we should change the gates instead.

Essentially, this scatter changes the shapes

transformers/src/transformers/models/deepseek_v2/modular_deepseek_v2.py

Line 299 in f92c553

topk_weight = torch.zeros_like(router_logits).scatter_(1, topk_idx, topk_weight)

We should simply remove that line and everything else should be resolved then (without these changes here)

good catch ! standardizing the router's outputs would definitely make things easier

Yes, agreed! I wasn't too familiar with the code and your fix seems like a better one

I don't understand how the change breaks FP8 compatibility, though? If you look at the modeling file, it only updates one slice

Because the weights are exchanged as is and the forward needs to be overwritten --> it has to expect the same input as the base version in the modeling code. FP8 currently assumes the version where we do not use the scatter

Reference:

transformers/src/transformers/integrations/finegrained_fp8.py

Lines 530 to 563 in 38e5987

# We follow the mixtral "eager" moe implementation at

# https://github.com/huggingface/transformers/blob/457048fbfdba9a7dee8bd03328c62f49e57b95f9/src/transformers/models/mixtral/modular_mixtral.py#L148

# The core changes in this FP8 version should only relate to how we call the linear projections

def forward(

self,

hidden_states: torch.Tensor,

top_k_index: torch.Tensor,

top_k_weights: torch.Tensor,

) -> torch.Tensor:

final_hidden_states = torch.zeros_like(hidden_states)

with torch.no_grad():

expert_mask = torch.nn.functional.one_hot(top_k_index, num_classes=self.num_experts)

expert_mask = expert_mask.permute(2, 1, 0)

expert_hit = torch.greater(expert_mask.sum(dim=(-1, -2)), 0).nonzero()

for expert_idx in expert_hit:

expert_idx = expert_idx[0]

if expert_idx == self.num_experts:

continue

top_k_pos, token_idx = torch.where(expert_mask[expert_idx])

current_state = hidden_states[token_idx]

gate, up = self.linear(

current_state, self.gate_up_proj[expert_idx], self.gate_up_proj_scale_inv[expert_idx]

).chunk(2, dim=-1)

current_hidden_states = self.act_fn(gate) * up

current_hidden_states = self.linear(

current_hidden_states, self.down_proj[expert_idx], self.down_proj_scale_inv[expert_idx]

)

routing_weights = top_k_weights[token_idx, top_k_pos, None]

current_hidden_states = current_hidden_states * routing_weights.to(current_hidden_states.dtype)

final_hidden_states.index_add_(0, token_idx, current_hidden_states.to(final_hidden_states.dtype))

return final_hidden_states

so with this deepseekv2 will have its top_k_weights with shape (num_tokens, num_top_k) ? like mixtral, qwen moe, etc

Yup, exactly

github-actions · 2026-01-15T15:15:51Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: deepseek_v2

Rocketknight1 · 2026-01-15T15:22:47Z

run-slow: deepseek_v2

github-actions · 2026-01-15T15:24:32Z

This comment contains run-slow, running the specified jobs:

models: ["models/deepseek_v2"]
quantizations: []

Rocketknight1 · 2026-01-15T15:28:56Z

@vasqu confirmed in local testing that your fix also recovers the correct output!

github-actions · 2026-01-15T15:39:28Z

CI Results

Workflow Run ⚙️

✅ No failing test specific to this PR 🎉 !

vasqu

Thank you :)

IlyasMoutawwakil

LGTM ! thanks !

@vasqu

* Fix eager experts for DeepSeekV2 * Use @vasqu's fix instead * Propagate modular

Rocketknight1 and others added 2 commits January 14, 2026 14:46

Fix eager experts for DeepSeekV2

5f75d26

Merge branch 'main' into fix_deepseek_v2_experts

f92c553

Rocketknight1 marked this pull request as ready for review January 14, 2026 14:50

vasqu reviewed Jan 14, 2026

View reviewed changes

yiliu30 mentioned this pull request Jan 15, 2026

[Feat] Re-quantization support for FP8 DeepSeek R1 w/ transformers v5.0 intel/auto-round#1248

Closed

3 tasks

Rocketknight1 added 2 commits January 15, 2026 15:14

Use @vasqu's fix instead

0395a1a

Propagate modular

a5b076c

vasqu approved these changes Jan 15, 2026

View reviewed changes

IlyasMoutawwakil approved these changes Jan 15, 2026

View reviewed changes

vasqu merged commit 396caf5 into main Jan 15, 2026
21 checks passed

vasqu deleted the fix_deepseek_v2_experts branch January 15, 2026 15:48

IlyasMoutawwakil mentioned this pull request Jan 23, 2026

Fix some MoE routers #43445

Merged

5 tasks

SangbumChoi pushed a commit to SangbumChoi/transformers that referenced this pull request Jan 23, 2026

Fix eager experts for DeepSeekV2 (huggingface#43288)

5fe4a55

* Fix eager experts for DeepSeekV2 * Use @vasqu's fix instead * Propagate modular

pasta-paul mentioned this pull request May 22, 2026

[deepseek_v4] Add DeepseekV4NextNPredictor class for MTP draft-head support #46127

Open

	# We follow the mixtral "eager" moe implementation at
	# https://github.com/huggingface/transformers/blob/457048fbfdba9a7dee8bd03328c62f49e57b95f9/src/transformers/models/mixtral/modular_mixtral.py#L148
	# The core changes in this FP8 version should only relate to how we call the linear projections
	def forward(
	self,
	hidden_states: torch.Tensor,
	top_k_index: torch.Tensor,
	top_k_weights: torch.Tensor,
	) -> torch.Tensor:
	final_hidden_states = torch.zeros_like(hidden_states)
	with torch.no_grad():
	expert_mask = torch.nn.functional.one_hot(top_k_index, num_classes=self.num_experts)
	expert_mask = expert_mask.permute(2, 1, 0)
	expert_hit = torch.greater(expert_mask.sum(dim=(-1, -2)), 0).nonzero()

	for expert_idx in expert_hit:
	expert_idx = expert_idx[0]
	if expert_idx == self.num_experts:
	continue
	top_k_pos, token_idx = torch.where(expert_mask[expert_idx])
	current_state = hidden_states[token_idx]
	gate, up = self.linear(
	current_state, self.gate_up_proj[expert_idx], self.gate_up_proj_scale_inv[expert_idx]
	).chunk(2, dim=-1)
	current_hidden_states = self.act_fn(gate) * up
	current_hidden_states = self.linear(
	current_hidden_states, self.down_proj[expert_idx], self.down_proj_scale_inv[expert_idx]
	)

	routing_weights = top_k_weights[token_idx, top_k_pos, None]
	current_hidden_states = current_hidden_states * routing_weights.to(current_hidden_states.dtype)
	final_hidden_states.index_add_(0, token_idx, current_hidden_states.to(final_hidden_states.dtype))

	return final_hidden_states

Uh oh!

Conversation

Rocketknight1 commented Jan 14, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Jan 14, 2026

Uh oh!

IlyasMoutawwakil commented Jan 14, 2026

Uh oh!

Rocketknight1 commented Jan 14, 2026

Uh oh!

IlyasMoutawwakil commented Jan 14, 2026

Uh oh!

Rocketknight1 commented Jan 14, 2026

Uh oh!

Rocketknight1 commented Jan 14, 2026

Uh oh!

github-actions Bot commented Jan 14, 2026

Uh oh!

github-actions Bot commented Jan 14, 2026

CI Results

Uh oh!

Rocketknight1 commented Jan 14, 2026

Uh oh!

vasqu Jan 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IlyasMoutawwakil Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Rocketknight1 Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

Rocketknight1 Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

IlyasMoutawwakil Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

vasqu Jan 15, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jan 15, 2026

Uh oh!

Rocketknight1 commented Jan 15, 2026

Uh oh!

github-actions Bot commented Jan 15, 2026

Uh oh!

Rocketknight1 commented Jan 15, 2026

Uh oh!

github-actions Bot commented Jan 15, 2026

CI Results

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

IlyasMoutawwakil left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vasqu Jan 14, 2026 •

edited

Loading

IlyasMoutawwakil left a comment •

edited

Loading