Fix masked causal loss in TransformerBridge - #1608
Open
emerardd wants to merge 1 commit into
Open
Conversation
emerardd
marked this pull request as ready for review
August 5, 2026 07:50
4 tasks
sohv
added a commit
to sohv/TransformerLens
that referenced
this pull request
Aug 5, 2026
…input TransformerBridge.forward() did not derive position_ids from a supplied attention_mask, so left-padded input silently got the wrong absolute positions — no error, no NaN, just wrong logits and a wrong loss. On gpt2 the loss for one prompt moved from 4.503170 unpadded to 11.154946 with three left pads, while HookedTransformer stays invariant (drift ~1e-06). Right padding was never affected, since causality already protects it. transformer_bridge.py derived position_ids only for batched *list* input, so pre-tokenized tensors fell through to HF's plain arange and the padding offset was never removed. This extends the same correction the list-input branch already applies: when a mask is supplied, position_ids are absent, and the mask indicates left padding, positions are computed from the mask. An explicitly supplied position_ids still wins. The bridge was also inconsistent with itself before this — the same batch gave different logits depending on whether it was passed as strings or token IDs (max |logit diff| 4.142e+01) — and enable_compatibility_mode(), which documents "HookedTransformer-equivalent numerics", diverged on left-padded input while matching exactly on unpadded input. Adds integration regression tests covering logit invariance under both padding sides, the same property in compatibility mode, and agreement between the derived and an explicitly supplied position_ids. The right-padding cases are controls: they pass with and without the fix. The tests live in the integration tier because left padding makes a fully masked query row, which the Native attention path turns into NaN until the masked-softmax fix in TransformerLensOrg#1608 lands. Fixes TransformerLensOrg#1609. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Propagate the forward-pass
attention_maskinto TransformerBridge causal loss so padding transitions do not affectreturn_type="loss",return_type="both", or per-token loss.The fix also:
seq_len - 1shape.Fixes #1607
Type of change
Screenshots
Not applicable.
Validation
4 passed102 passed, 1 skippedgit diff --check: cleanuv run mypy .: success on 422 source filesChecklist