🚨 Enable SDPA (and other attention backends) for T5 and propagate to the T5 family by jiqing-feng · Pull Request #47014 · huggingface/transformers

jiqing-feng · 2026-07-02T01:54:46Z

What this PR does

Refactors the T5 attention stack to route through ALL_ATTENTION_FUNCTIONS,
so T5 can dispatch to sdpa / eager instead of the old eager-only path, and
propagates the change to the copied-from family.

Now _supports_sdpa = True / _supports_attention_backend = True and running
test_eager_matches_sdpa_inference:

t5 (reference), mt5, udop, pop2piano
pix2struct — also required migrating the vision-tower attention
(Pix2StructVisionAttention), a standard dense attention that was not on the
interface, otherwise SDPA is unusable for the full model.
umt5 — its attention is its own dense implementation (not # Copied from
T5); migrated the same way.

switch_transformers (modular, inherits T5 attention) is regenerated so its
modeling matches the refactored T5, but it stays eager-only (_supports_sdpa
not set): it is not on the SDPA path, this only propagates the eager refactor.

Alignment applied to migrated modules: attention goes through
ALL_ATTENTION_FUNCTIONS (with eager_attention_forward fallback), self.scaling,
is_causal; the relative position bias is folded into the additive attention mask;
forwards use **kwargs: Unpack[TransformersKwargs] + @can_return_tuple /
@auto_docstring / @merge_with_config_defaults / @capture_outputs, dropping the
manual return_dict / output_attentions / output_hidden_states resolution.

🚨 Behavior changes

With sdpa, softmax no longer force-upcasts to fp32 as the old path did;
numerics stay within the eager_matches_sdpa tolerance.
output_attentions removed from internal block/layer tuple returns; attentions
are collected via capture_outputs.

test_eager_matches_sdpa_inference is not skipped for any enabled model.

longt5 stays not-SDPA-enabled

The regular LongT5Attention (copied from T5) is on the interface, but the encoder
always runs the block-sparse LongT5LocalAttention / LongT5TransientGlobalAttention.
These operate on 5D blocked tensors (batch, num_blocks, heads, block_len, 3*block_len),
while sdpa_attention_forward assumes 4D (batch, heads, seq, dim) — they cannot go
through the interface at all. LongT5EncoderModel is purely block-sparse, so the
framework refuses to switch its attention implementation:

LongT5EncoderModel does not support setting its attention implementation
dynamically, because it does not follow the functional approach based on
AttentionInterface

Forcing _supports_sdpa = True makes the encoder-only sdpa tests fail (fp16
mean relative difference: nan). Same precedent as pegasus_x (block-sparse
encoder + regular decoder), which also sets _supports_sdpa = False.

Testing

utils/check_copies.py passes; full sweep green:

tests/models/{t5,mt5,umt5,udop,pop2piano,longt5,pix2struct,switch_transformers}/test_modeling_*.py
-> 1817 passed, 1958 skipped, 18 xfailed, 6 xpassed, 16699 subtests passed, 0 failed

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

Rocketknight1 · 2026-07-02T11:52:08Z

Attention dispatch so cc @ArthurZucker @Cyrilvallez

vasqu · 2026-07-02T14:07:43Z

WIll check it out when I have time but it's a big PR so expect delays 🙏

Cyrilvallez · 2026-07-08T09:14:57Z

This competes with #46946

vasqu

Only checked t5 becuse everything else is dependent on that (so the same comments apply), after t5 is solid I will make a scan through all models

vasqu · 2026-07-17T13:41:46Z

                causal_mask = mask[:, :, :, : key_states.shape[-2]]
+                if causal_mask.dtype == torch.bool:
+                    # `sdpa` may materialize a boolean mask (True = keep). Turn it into an additive float mask so it
+                    # can be folded into the relative position bias, just like the `eager` float mask.
+                    causal_mask = torch.where(
+                        causal_mask,
+                        torch.tensor(0.0, device=causal_mask.device, dtype=position_bias.dtype),
+                        torch.finfo(position_bias.dtype).min,
+                    )


Suggested change

causal_mask = mask[:, :, :, : key_states.shape[-2]]

if causal_mask.dtype == torch.bool:

# `sdpa` may materialize a boolean mask (True = keep). Turn it into an additive float mask so it

# can be folded into the relative position bias, just like the `eager` float mask.

causal_mask = torch.where(

causal_mask,

torch.tensor(0.0, device=causal_mask.device, dtype=position_bias.dtype),

torch.finfo(position_bias.dtype).min,

)

if causal_mask.dtype == torch.bool:

# `sdpa` may materialize a boolean mask (True = keep). Turn it into an additive float mask so it

# can be folded into the relative position bias, just like the `eager` float mask.

causal_mask = torch.where(

causal_mask,

torch.tensor(0.0, device=causal_mask.device, dtype=position_bias.dtype),

torch.finfo(position_bias.dtype).min,

)

I don't think we need to slice anymore (might be wrong)

I think we should do the bool -> float conversion at mask creation time tbh

(1) Dropped the slice for t5/umt5 (no-op there since kv_len already matches key_states.shape[-2]); kept it on pix2struct because its text stack still builds the mask the old way and removing it breaks test_eager_matches_sdpa_inference — will remove it with the pix2struct refactor follow-up. (2) Doing the bool→float conversion in masking_utils would hit every model and other backends rely on the boolean mask (sdpa's is_causal fast path). Did you mean a T5-specific mask_function that returns the additive bias directly? Happy to do that if there's an existing pattern.

No no, I dont mean in the mask function itself but when we create in T5 so in the stack / model modules after creation, do the conversion directly there.

With the current approach there's no bool→float conversion needed anymore — since position_bias is passed to the attention interface separately instead of being folded into the mask, we don't need an additive float mask. eager just gets the float mask and sdpa gets the bool mask, each handled natively by its backend. So there's nothing to convert after mask creation.

vasqu · 2026-07-17T13:42:57Z

+        # T5 uses a relative attention bias that is added to the attention scores. This is passed as the additive
+        # attention mask so that it works with the different attention implementations. As it is always non-`None`,
+        # `is_causal` is never inferred, so the causal behavior is fully encoded in the bias itself.
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            if self.config._attn_implementation != "sdpa":
+                raise ValueError(
+                    "T5 adds a relative position bias on top of the attention scores, which is only supported by the "
+                    f"`eager` and `sdpa` attention implementations, but got `{self.config._attn_implementation}`."
+                )
+            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]


Suggested change

# T5 uses a relative attention bias that is added to the attention scores. This is passed as the additive

# attention mask so that it works with the different attention implementations. As it is always non-`None`,

# `is_causal` is never inferred, so the causal behavior is fully encoded in the bias itself.

attention_interface: Callable = eager_attention_forward

if self.config._attn_implementation != "eager":

if self.config._attn_implementation != "sdpa":

raise ValueError(

"T5 adds a relative position bias on top of the attention scores, which is only supported by the "

f"`eager` and `sdpa` attention implementations, but got `{self.config._attn_implementation}`."

)

attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]

attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(

self.config._attn_implementation, eager_attention_forward

)

Imo

This was the old format

The flags will already indicate what is supported so we can just go ahead

Done, applied to umt5 and pix2struct too.

vasqu · 2026-07-17T13:44:33Z

    _no_split_modules = ["T5Block"]
    _keep_in_fp32_modules = ["wo"]

+    _supports_attention_backend = True


Suggested change

_supports_attention_backend = True

would avoid this for now, enc-dec are more complicated

Removed for t5, umt5 and pix2struct. We can revisit once enc-dec is settled.

vasqu · 2026-07-17T13:45:04Z

+    _supports_flash_attn = False
+    _supports_flex_attn = False


We can add a small comment here as to why == relative bias is not possible

Altho in the future we might be able to use FA4 to integrate the score mod along

Added a comment: T5 folds the relative position bias into the additive mask, which flash can't take and flex would need a dedicated score_mod for. (Noted the FA4 / score_mod possibility, cc @zucchini-nlp below.)

I think based on @zucchini-nlp we could actually change the strategy a bit:

Do not add mask and bias directly

Pass the bias and mask separately

Do not convert your mask manually anymore

That would also enable flex as side effect

Done — reworked exactly along those lines: mask and bias are no longer added together, position_bias is passed to the attention interface as a separate kwarg, and all the manual mask conversion is gone. eager and sdpa each handle the bias and mask natively, and flex is now enabled as a side effect. Applied across the whole T5 family.

vasqu · 2026-07-17T13:45:41Z

-        **kwargs,
+        **kwargs: Unpack[TransformersKwargs],
    ):
        use_cache = use_cache if use_cache is not None else self.config.use_cache


Suggested change

use_cache = use_cache if use_cache is not None else self.config.use_cache

already handled by decorator

Removed, the decorator already handles it.

vasqu · 2026-07-17T13:47:03Z

@@ -668,8 +714,6 @@
                raise ValueError("You have to initialize the model with valid token embeddings")
            inputs_embeds = self.embed_tokens(input_ids)


Let's simplify here

The gradient ckpting is handled by the decorator

The input ids and embeds checks can be simplified by a lot (see other encoder-decoder models like bart)

The value error on inputs embeds that embed tokens is overkill we can skip that if completely

Simplified as suggested, same for umt5. pix2struct's text stack still uses the old decorator style, so I'll migrate it in a follow-up to keep this PR focused on the SDPA enablement.

vasqu · 2026-07-17T13:47:37Z

+                and_mask_function=dummy_and_mask_function,
            )
        else:
            attention_mask = create_bidirectional_mask(


relative bias only on decoder side?

Relative bias is on both encoder and decoder. The dummy_and_mask_function is only needed on the decoder's create_causal_mask to force the mask to be materialized as a float so the bias can be folded in (otherwise sdpa may take the is_causal shortcut). The encoder uses create_bidirectional_mask which is already additive.

Based on #47014 (comment) and the model addition of inkling, we also dont need this workaround anymore

Right — dropped that workaround. Now that the bias is passed separately (like inkling does), there's nothing left to work around here.

vasqu · 2026-07-17T13:47:48Z


        encoder_extended_attention_mask = None
        if self.is_decoder and encoder_hidden_states is not None:
            encoder_extended_attention_mask = create_bidirectional_mask(


Cross-attention has no relative bias (has_relative_attention_bias=False), so no dummy mask function is needed here.

vasqu · 2026-07-17T13:48:09Z

@@ -881,19 +892,16 @@
        >>> last_hidden_states = outputs.last_hidden_state
        ```"""
        use_cache = use_cache if use_cache is not None else self.config.use_cache


Suggested change

use_cache = use_cache if use_cache is not None else self.config.use_cache

vasqu · 2026-07-17T13:48:20Z

@@ -1049,7 +1050,6 @@
        >>> # studies have shown that owning a dog is good for you.
        ```"""
        use_cache = use_cache if use_cache is not None else self.config.use_cache


ok last time mention

Removed here too, and in T5ForQuestionAnswering for consistency. Thanks for the nudge.

zucchini-nlp · 2026-07-17T14:52:14Z

FWIW, we might take advantage of recent updates to accommodate Inkling, and support FlexAttn with position bias. I didn't check in details, might be wrong if these two position-biases are added up in different ways

transformers/src/transformers/models/inkling/modeling_inkling.py

Lines 602 to 609 in 290d5c4

    
           # The relative position bias flows through the attention interface as a `position_bias` (duh) 
        
           # kwarg that only the eager path consumes; other backends need a score_mod/kernel 
        
           _supports_flash_attn = False 
        
           _supports_sdpa = True 
        
           _supports_flex_attn = True 
        
           _can_compile_fullgraph = False 
        
           _supports_attention_backend = False 
        
           _keys_to_ignore_on_load_unexpected = [r"model\.mtp\..*"]

vasqu · 2026-07-17T14:57:20Z

You could only do this with flex attention using score mod afaik, but flex attention with score mod was quite slow; things might have changed by then but their mask creation is the biggest blocker to make flex attention viable.

Edit: This might be ok when triggered via FA4 but still fear the mask creation tbh

vasqu · 2026-07-17T15:04:56Z

Ok yea checked, we added it to the score mod. We can do it for sure, just doubting the perf a bit

transformers/src/transformers/integrations/flex_attention.py

Line 271 in 290d5c4

position_bias: torch.Tensor | None = None,

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

jiqing-feng · 2026-07-20T07:04:06Z

Great pointer. Given the perf concern around flex mask creation, I'd keep this PR focused on sdpa and add flex support (via score_mod) as a follow-up once we can benchmark it. Works for you both?

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

vasqu · 2026-07-20T16:30:02Z

@jiqing-feng I haven't taken a look at inkling too deeply before this but I noticed it reduces a lot of friction if we follow that process, see #47014 (comment) and #47014 (comment)

The tl;dr:

No mask workarounds (force creation, float conversion)
Pass position bias separately to the mask as kwarg to the interface
Side effect: it also enables flex attention (even though the perfs there are likely not super nice)

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

jiqing-feng · 2026-07-21T02:57:21Z

@jiqing-feng I haven't taken a look at inkling too deeply before this but I noticed it reduces a lot of friction if we follow that process, see #47014 (comment) and #47014 (comment)

The tl;dr:

No mask workarounds (force creation, float conversion)

Pass position bias separately to the mask as kwarg to the interface

Side effect: it also enables flex attention (even though the perfs there are likely not super nice)

Following the inkling approach did remove a lot of friction: no more mask workarounds (no forced creation, no float conversion), and the position bias is now passed to the interface as a separate kwarg alongside the mask. Flex attention falls out as a side effect too. Reworked t5 along these lines and propagated it across the whole T5 family.

vasqu

Smaller comments overall, only bigger thing I noticed was that fullgraph was set to false now which would be nice to solve instead 👀

vasqu · 2026-07-21T15:04:36Z

And ig the same here re output recorder. Just fmi why it wasnt refactored

It's orthogonal to the attention-backend work and a fairly heavy refactor, so I'd do it in a dedicated follow-up. It's more than the vision tower — the whole model (text decoder included) still uses the legacy output_attentions/return_dict path with no _can_record_outputs, and being a two-tower model it means reworking three forwards and registering recorders for both towers. Happy to fold it in if you'd rather.

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

github-actions · 2026-07-27T02:14:18Z

CI recap

Dashboard: View test results in Grafana
Latest run: 30059557292:2
Result: success | Jobs: 16 | Tests: 174,496 | Failures: 0 | Duration: 8h 24m

jiqing-feng · 2026-07-27T02:32:35Z

Hi @vasqu . I've fixed your last comment, please rerun the tests. Thanks!

vasqu · 2026-07-28T14:09:05Z

run-slow: blip_2, longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

github-actions · 2026-07-28T14:10:54Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/blip_2", "models/longt5", "models/mt5", "models/pix2struct", "models/pop2piano", "models/switch_transformers", "models/t5", "models/udop", "models/umt5"]
quantizations: []

HuggingFaceDocBuilderDev · 2026-07-28T14:22:23Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-07-28T14:59:56Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	643875a0	workflow commit (merge commit)
PR	aad66e65	branch commit (from PR)
main	ebea912f	base commit (on `main`)

Model CI Report

❌ 2 new failed tests from this PR 😭

longt5:
tests/models/longt5/test_modeling_longt5.py::LongT5ModelIntegrationTests::test_summarization (❌ ⟹ ❌)
switch_transformers:
tests/models/switch_transformers/test_modeling_switch_transformers.py::SwitchTransformerModelIntegrationTests::test_small_logits (✅ ⟹ ❌)

yao-matrix · 2026-07-28T18:21:56Z

@jiqing-feng , pls check CI, thx

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

github-actions · 2026-07-29T05:38:13Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: blip_2, longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

jiqing-feng · 2026-07-29T05:40:58Z

This is a real bug on main that the PR fixes.

The stack loop never writes position_bias back from the layer output (t5 does position_bias = layer_outputs[1]). Only block 0 has has_relative_attention_bias=True, so blocks 1–11 fall through to position_bias = torch.zeros(...) and attention becomes uniform. Dropped in #40132. switch_transformers is the only model in the family affected (umt5 has a per-layer bias by design).

Proof: I checked out 7938e91faa^ (right before #40132) and ran the same input — its output is bit-identical to this PR's. So the PR restores the original numerics rather than changing them.

Greedy generation with google/switch-base-8:

prompt	`main`	this PR
`The human walks into a bar and orders a <extra_id_0>`	`'.'`	`'drink.'`
`The capital of France is <extra_id_0>.`	`'..'`	`'Paris.'`

('drink.' is what the skipped test_small_generate asserts.)

The test barely moved because it feeds torch.ones((32, 64)) — all tokens identical, so the attention distribution hardly matters. With real inputs the encoder output differs by up to 3.93 (absmax 4.66). That's why it kept passing after #40132.

I regenerated the ("cuda", 8) expectations, which now match the pre-#40132 values exactly. Note these come from an A100 — on A100 the PR actually passes the old values too, so the diff is hardware spread, not a behaviour change. (None, None) is unchanged: it predates #40132 and already encodes the correct behaviour.

`longt5::test_summarization`

Pre-existing, fails the same on main. Root cause is in get_expanded_tied_weights_keys:

if not getattr(self.config, "tie_word_embeddings", False):
    return {}

_tied_weights_keys mixes structural aliases with the output embedding:

{'encoder.embed_tokens.weight': 'shared.weight',   # structural, must always tie
 'decoder.embed_tokens.weight': 'shared.weight',   # structural, must always tie
 'lm_head.weight': 'shared.weight'}                # what tie_word_embeddings governs

This checkpoint has tie_word_embeddings=False, so all three are dropped and the encoder/decoder embeddings stay randomly initialised (std 1.00 vs 8.92 for shared.weight), hence the gibberish output. Introduced in #41541; affects any encoder-decoder checkpoint with tie_word_embeddings=False.

Only dropping the entries that point at get_output_embeddings() fixes it — test_summarization then passes. Since that's a core modeling_utils change unrelated to attention dispatch, I've kept it out of this PR and will open a separate one. (Fixed in #47620)

(longt5::test_inference_hidden_states also fails on main — unrelated, 0.0035 abs diff against atol=1e-4.)

github-actions · 2026-07-29T05:42:21Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: blip_2, longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

vasqu · 2026-07-29T16:17:36Z

Fair enough, I was mainly concerned about the switch transformer failure, checking slow ci and then merge

vasqu · 2026-07-29T16:17:40Z

run-slow: blip_2, longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

github-actions · 2026-07-29T16:19:18Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/blip_2", "models/longt5", "models/mt5", "models/pix2struct", "models/pop2piano", "models/switch_transformers", "models/t5", "models/udop", "models/umt5"]
quantizations: []

github-actions · 2026-07-29T21:44:20Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	83d9ebf1	workflow commit (merge commit)
PR	b35db480	branch commit (from PR)
main	3d7b75a8	base commit (on `main`)

⚠️ Model CI failed to report results

The test failure analysis could not be completed. Please check the workflow run for details.

vasqu · 2026-07-29T22:00:22Z

I will check again tomorrow, I think we are safe but I dont want to merge with broken slow ci

vasqu · 2026-07-30T11:26:59Z

run-slow: blip_2, longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

github-actions · 2026-07-30T11:28:07Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: blip_2, longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

github-actions · 2026-07-30T11:28:38Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/blip_2", "models/longt5", "models/mt5", "models/pix2struct", "models/pop2piano", "models/switch_transformers", "models/t5", "models/udop", "models/umt5"]
quantizations: []

github-actions · 2026-07-30T11:51:31Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	d74d3f2d	workflow commit (merge commit)
PR	c8352f73	branch commit (from PR)
main	6b8369fe	base commit (on `main`)

Model CI Report

❌ 1 new failed tests from this PR 😭

longt5:
tests/models/longt5/test_modeling_longt5.py::LongT5ModelIntegrationTests::test_summarization (❌ ⟹ ❌)

…the T5 family (huggingface#47014) * enable t5 sdpa Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix format Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix comments Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * update comments Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix mask and position Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix use_cache and _supports_attention_backend Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix _can_compile_fullgraph Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix class register Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix tests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * Update src/transformers/models/t5/modeling_t5.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/t5/modeling_t5.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/mt5/modeling_mt5.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/pix2struct/modeling_pix2struct.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/pop2piano/modeling_pop2piano.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/pop2piano/modeling_pop2piano.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * Update src/transformers/models/pop2piano/modeling_pop2piano.py Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com> * fix comments Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix name Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix tyests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix test Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix tests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix test Signed-off-by: jiqing-feng <jiqing.feng@intel.com> --------- Signed-off-by: jiqing-feng <jiqing.feng@intel.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>

jiqing-feng added 5 commits July 1, 2026 02:03

enable t5 sdpa

42d8877

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

update

2b821a2

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

update

a172063

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

update

e76fc48

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

fix format

0109081

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

jiqing-feng changed the title ~~enable t5 sdpa~~ 🚨 Enable SDPA (and other attention backends) for T5 and propagate to the T5 family Jul 2, 2026

jiqing-feng marked this pull request as ready for review July 2, 2026 06:45

github-actions Bot requested review from ArthurZucker and zucchini-nlp July 2, 2026 06:45

vasqu self-assigned this Jul 2, 2026

Cyrilvallez mentioned this pull request Jul 8, 2026

T5: add SDPA and Flash Attention 2 support #46946

Closed

vasqu reviewed Jul 17, 2026

View reviewed changes

fix comments

5e1fe40

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

jiqing-feng added 2 commits July 20, 2026 00:10

update comments

6a662b5

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

Merge branch 'main' into sdpa

ac191ba

vasqu mentioned this pull request Jul 20, 2026

🚨[wav2vec2] Support attn_implementation=sdpa dispatch #46196

Open

fix mask and position

44ded18

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

Merge branch 'main' into sdpa

ff7200e

vasqu reviewed Jul 21, 2026

View reviewed changes

jiqing-feng added 2 commits July 21, 2026 22:43

fix use_cache and _supports_attention_backend

0a33998

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

fix _can_compile_fullgraph

b2c3fa9

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

Merge branch 'main' into sdpa

aad66e6

qgallouedec mentioned this pull request Jul 28, 2026

Add Flash Attention 2.0 for T5 Family #27441

Closed

vasqu approved these changes Jul 28, 2026

View reviewed changes

fix test

82fabfd

Signed-off-by: jiqing-feng <jiqing.feng@intel.com>

Merge branch 'main' into sdpa

b35db48

Merge branch 'main' into sdpa

c8352f7

vasqu added this pull request to the merge queue Jul 30, 2026

Merged via the queue into huggingface:main with commit 560f36c Jul 30, 2026
114 of 115 checks passed

		@@ -668,8 +714,6 @@
		raise ValueError("You have to initialize the model with valid token embeddings")
		inputs_embeds = self.embed_tokens(input_ids)

Uh oh!

Conversation

jiqing-feng commented Jul 2, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does

🚨 Behavior changes

longt5 stays not-SDPA-enabled

Testing

Uh oh!

Rocketknight1 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vasqu commented Jul 2, 2026

Uh oh!

Cyrilvallez commented Jul 8, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiqing-feng Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Jul 17, 2026

Uh oh!

vasqu commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vasqu commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jiqing-feng commented Jul 2, 2026 •

edited by github-actions Bot

Loading

Rocketknight1 commented Jul 2, 2026 •

edited

Loading

jiqing-feng Jul 21, 2026 •

edited

Loading

vasqu commented Jul 17, 2026 •

edited

Loading

vasqu commented Jul 17, 2026 •

edited

Loading