Skip to content

Fix masked causal loss in TransformerBridge - #1608

Open
emerardd wants to merge 1 commit into
TransformerLensOrg:dev-4.xfrom
emerardd:fix/bridge-loss-attention-mask
Open

Fix masked causal loss in TransformerBridge#1608
emerardd wants to merge 1 commit into
TransformerLensOrg:dev-4.xfrom
emerardd:fix/bridge-loss-attention-mask

Conversation

@emerardd

@emerardd emerardd commented Aug 5, 2026

Copy link
Copy Markdown

Description

Propagate the forward-pass attention_mask into TransformerBridge causal loss so padding transitions do not affect return_type="loss", return_type="both", or per-token loss.

The fix also:

  • moves the loss mask to the logits device when needed;
  • uses NaN-safe masking for ignored token losses;
  • zeros fully masked Native attention pattern entries so left padding cannot poison later layers; and
  • corrects the per-token loss return annotation to reflect its seq_len - 1 shape.

Fixes #1607

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Screenshots

Not applicable.

Validation

  • Regression tests: 4 passed
  • Affected Bridge unit tests: 102 passed, 1 skipped
  • Changed files: pycln, isort, and Black clean
  • git diff --check: clean
  • uv run mypy .: success on 422 source files

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • Documentation is not required for this internal correctness fix
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective
  • The complete unit-test suite passes locally; only the affected unit-test surface was run
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

@emerardd
emerardd marked this pull request as ready for review August 5, 2026 07:50
sohv added a commit to sohv/TransformerLens that referenced this pull request Aug 5, 2026
…input

TransformerBridge.forward() did not derive position_ids from a supplied
attention_mask, so left-padded input silently got the wrong absolute positions —
no error, no NaN, just wrong logits and a wrong loss. On gpt2 the loss for one
prompt moved from 4.503170 unpadded to 11.154946 with three left pads, while
HookedTransformer stays invariant (drift ~1e-06). Right padding was never
affected, since causality already protects it.

transformer_bridge.py derived position_ids only for batched *list* input, so
pre-tokenized tensors fell through to HF's plain arange and the padding offset
was never removed. This extends the same correction the list-input branch
already applies: when a mask is supplied, position_ids are absent, and the mask
indicates left padding, positions are computed from the mask. An explicitly
supplied position_ids still wins.

The bridge was also inconsistent with itself before this — the same batch gave
different logits depending on whether it was passed as strings or token IDs
(max |logit diff| 4.142e+01) — and enable_compatibility_mode(), which documents
"HookedTransformer-equivalent numerics", diverged on left-padded input while
matching exactly on unpadded input.

Adds integration regression tests covering logit invariance under both padding
sides, the same property in compatibility mode, and agreement between the
derived and an explicitly supplied position_ids. The right-padding cases are
controls: they pass with and without the fix. The tests live in the integration
tier because left padding makes a fully masked query row, which the Native
attention path turns into NaN until the masked-softmax fix in TransformerLensOrg#1608 lands.

Fixes TransformerLensOrg#1609.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant