From 086e4c1c53c758ee57970c25e293c06c3df14462 Mon Sep 17 00:00:00 2001 From: Loi Nguyen Date: Fri, 31 Jul 2026 04:12:06 +0700 Subject: [PATCH 1/2] docs: demonstrate W_pos migration --- docs/source/content/migrating_to_v3.md | 28 ++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/source/content/migrating_to_v3.md b/docs/source/content/migrating_to_v3.md index 26b736d32..e0591fac4 100644 --- a/docs/source/content/migrating_to_v3.md +++ b/docs/source/content/migrating_to_v3.md @@ -223,3 +223,31 @@ Weight-matrix rows return **raw** HuggingFace weights by default. `HookedTransfo | `model.W_pos` | `bridge.pos_embed.W_pos` | Raw weight (also `bridge.pos_embed.weight`). `center_writing_weights` centers `W_pos` in default HT loads, so it matches HT's only under matching processing (`enable_compatibility_mode()`, or HT loaded with no processing). | | `model.W_E_pos` | `torch.cat([bridge.W_E, bridge.pos_embed.W_pos], dim=0)` | No single accessor — concatenate the token + positional matrices. Same weight-processing caveat as `W_pos` (both `W_E` and `W_pos` are centered writing-weights). | | `HookedTransformer.from_pretrained_no_processing(name)` | `TransformerBridge.boot_transformers(name, no_processing=True)` | Both load raw weights, so these match. | + +```python +import torch + +from transformer_lens import HookedTransformer +from transformer_lens.model_bridge import TransformerBridge + +model = HookedTransformer.from_pretrained( + "gpt2", device="cpu", dtype=torch.float32 +) +bridge = TransformerBridge.boot_transformers( + "openai-community/gpt2", device="cpu", dtype=torch.float32 +) +bridge.enable_compatibility_mode() + +W_pos = bridge.pos_embed.W_pos +W_E_pos = torch.cat([bridge.W_E, W_pos], dim=0) + +assert W_pos.shape == (bridge.cfg.n_ctx, bridge.cfg.d_model) +assert W_E_pos.shape == ( + bridge.cfg.d_vocab + bridge.cfg.n_ctx, + bridge.cfg.d_model, +) +torch.testing.assert_close(W_pos, model.W_pos) +torch.testing.assert_close(W_E_pos, model.W_E_pos) +``` + +The equality checks use matching weight processing: `enable_compatibility_mode()` centers the bridge's writing weights in the same way as a default `HookedTransformer` load. A raw bridge load instead matches `HookedTransformer.from_pretrained_no_processing`; see [Will my numbers match HookedTransformer?](#will-my-numbers-match-hookedtransformer) for the broader rule. From c31501604f332798bcd734b00a2d17765db02525 Mon Sep 17 00:00:00 2001 From: Loi Nguyen Date: Sat, 1 Aug 2026 04:34:47 +0700 Subject: [PATCH 2/2] Clarify positional weight migration example --- docs/source/content/migrating_to_v3.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/source/content/migrating_to_v3.md b/docs/source/content/migrating_to_v3.md index 51037842e..340d4eb6d 100644 --- a/docs/source/content/migrating_to_v3.md +++ b/docs/source/content/migrating_to_v3.md @@ -240,6 +240,8 @@ Weight-matrix rows return **raw** HuggingFace weights by default. `HookedTransfo | `model.all_head_labels()` | `bridge.all_head_labels` | This is a property on the bridge, so omit the call parentheses. | | `model.set_tokenizer(tokenizer)` | `TransformerBridge.boot_transformers(name, tokenizer=tokenizer)` | A bridge's tokenizer is fixed when it boots. Reboot to change it; assigning `bridge.tokenizer` directly bypasses tokenizer/config wiring. | +The following example demonstrates the `W_pos` and `W_E_pos` equivalents under matching weight processing: + ```python import torch