Skip to content

Fix DeepSeek V2 dense MLP hook semantics - #1651

Merged
jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
koriyoshi2041:fix/deepseek-v2-dense-mlp-hooks
Aug 13, 2026
Merged

Fix DeepSeek V2 dense MLP hook semantics#1651
jlarson4 merged 2 commits into
TransformerLensOrg:devfrom
koriyoshi2041:fix/deepseek-v2-dense-mlp-hooks

Conversation

@koriyoshi2041

Copy link
Copy Markdown
Contributor

Closes #1645.

DeepSeek V2 uses dense gated MLPs for the prefix selected by first_k_dense_replace, but the adapter currently gives every layer MoE boundary aliases. This adds optional dense projection bridges and resolves the compatibility aliases to gate_proj, up_proj, and down_proj when those modules exist, while sparse layers keep their existing block-boundary aliases.

The regression coverage checks exact hook identity and width on a tiny dense layer, and verifies that a later sparse layer retains its current behavior.

Tests:

  • uv run pytest -q tests/unit/model_bridge/supported_architectures/test_deepseek_v2_adapter.py tests/integration/model_bridge/test_deepseek_v2_adapter.py (44 passed)
  • uv run mypy transformer_lens/model_bridge/supported_architectures/deepseek_v2.py
  • make check-format
  • git diff --check

@jlarson4 jlarson4 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on @koriyoshi2041! Looks good to me, just one note about a bug in the test suite.

Comment thread tests/integration/model_bridge/test_deepseek_v2_adapter.py Outdated
@jlarson4

Copy link
Copy Markdown
Collaborator

Thanks for fixing that up @koriyoshi2041! Great work, merging now

@jlarson4
jlarson4 merged commit 79ad5a2 into TransformerLensOrg:dev Aug 13, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug Report] DeepSeek V2 dense layers expose MLP boundary tensors under neuron-hook names

2 participants