Skip to content

Commit 8188ca1

Browse files
justinchubyCopilot
andcommitted
fix: add Q/K norm handling to BitNetAttention.forward
Address Copilot reviewer comment: BitNetAttention.forward was missing the optional Q/K normalization path (q_norm/k_norm) that the base Attention class supports. Added the full q_norm/k_norm handling (both full and per-head modes) between Q/K/V projections and RoPE, matching the base Attention.forward logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
1 parent be515c4 commit 8188ca1

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

src/mobius/models/bitnet.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,19 @@ def forward(
6161
key_states = self.k_proj(op, hidden_states)
6262
value_states = self.v_proj(op, hidden_states)
6363

64+
# Optional Q/K normalization (inherited from base Attention)
65+
if self.q_norm is not None and self.k_norm is not None:
66+
if self._qk_norm_full:
67+
query_states = self.q_norm(op, query_states)
68+
key_states = self.k_norm(op, key_states)
69+
else:
70+
query_states = op.Reshape(query_states, [0, 0, -1, self.head_dim])
71+
key_states = op.Reshape(key_states, [0, 0, -1, self.head_dim])
72+
query_states = self.q_norm(op, query_states)
73+
key_states = self.k_norm(op, key_states)
74+
query_states = op.Reshape(query_states, [0, 0, -1])
75+
key_states = op.Reshape(key_states, [0, 0, -1])
76+
6477
# RoPE
6578
if position_embeddings is not None:
6679
query_states = apply_rotary_pos_emb(

0 commit comments

Comments
 (0)