Skip to content

Fix gated Qwen W_Q analysis weights - #1653

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:dev-4.xfrom
emerardd:fix/qwen-gated-wq
Aug 12, 2026
Merged

Fix gated Qwen W_Q analysis weights#1653
jlarson4 merged 1 commit into
TransformerLensOrg:dev-4.xfrom
emerardd:fix/qwen-gated-wq

Conversation

@emerardd

Copy link
Copy Markdown
Contributor

Description

Fixes #1652.

Qwen3.5 and Qwen3Next use a gated query projection whose rows are interleaved as query and gate values within each head. TransformerBridge.W_Q previously exposed the complete projection as if every row belonged to the query, so its final dimension was 2 * cfg.d_head. Weight-space analyses such as composition scores, QK analysis, and SVD therefore consumed query-plus-gate data under a query-only interface.

The adapters attempted to remove the gate rows in preprocess_weights(), but that helper matched Hugging Face-style keys while the real processing path supplies TransformerLens-style state-dict keys. More importantly, preprocessing the parameter would mutate the live projection and remove the gate needed by the model forward.

This change:

  • makes AttentionBridge.W_Q return a query-only per-head view when gated_q_proj is enabled;
  • leaves the underlying query-and-gate projection unchanged, preserving forward behavior and hook_q_gate;
  • removes the unreachable gated-query preprocessing helper and tests built around synthetic key formats;
  • adds download-free, config-only integration coverage for Qwen3.5 and Qwen3Next through build_bridge_from_module() and process_weights();
  • checks exact query-row values, multi-layer W_Q shape, forward parity, gate-hook behavior, live-weight preservation, ordinary Qwen3 as a control, and the multimodal adapter's shared gated-query contract.

No new dependencies are required.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Screenshots

Not applicable.

Checklist:

  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Validation

  • Full unit suite: 5260 passed, 55 skipped, 54 deselected, 10 xfailed
  • Affected and adjacent Qwen/attention coverage: 146 passed
  • New download-free integration regressions: 5 passed
  • Formatting: pycln, isort, and Black pass
  • Type checking: Success: no issues found in 431 source files
  • git diff --check passes

@emerardd
emerardd marked this pull request as ready for review August 12, 2026 08:16
@jlarson4 jlarson4 linked an issue Aug 12, 2026 that may be closed by this pull request
1 task
@jlarson4

Copy link
Copy Markdown
Collaborator

Looks good, great resolution for this issue @emerardd!

@jlarson4
jlarson4 merged commit b8aca0f into TransformerLensOrg:dev-4.x Aug 12, 2026
49 of 50 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug Report] Qwen3.5/Qwen3Next expose query-gate rows as double-width W_Q

2 participants