[Feature]: FlashAttention 3 support

[orellavie1212]

🚀 The feature, motivation and pitch

As you know, FA3 promises 1.5x~ improvements
Dao-AILab/flash-attention@7ef2484

Alternatives

No response

Additional context

No response

[simon-mo]

Yes. Actively looking.

Update: It seem Dao-AILab/flash-attention#1268 is merged. The integration is now ready for testing.

[Navanit-git]

any updates??

[Ushnish-Sengupta]

This would be amazing. FA3 is apparently part of the reason behind Together Inference Engine leapfrogging vllm by a lot!

[ehartford]

Very interested in this

[freddifederica]

+1

+1

[badrisnps]

+1

[simon-mo]

A small progress update here. @felixzhu555 did some testing. FA3 is mostly a training kernel at the moment and only benefit extremely large batch size with long context.

[tmoon]

Any further update on this?

[yuvalkk2002]

+1

[hitcoogle]

+1

[taegeonum]

+1

[Isaac4real]

+1