[PyTorch] Add sliding window support to FlashAttention - #551

Merged

ptrendx merged 21 commits into

NVIDIA:mainfrom

cyanguwa:fa/sliding_window

Dec 16, 2023

Commits on Dec 6, 2023

add sliding window to FA
cyanguwa
committed
69f568e
fix forward logic
cyanguwa
committed
8f7121f
fix lint
cyanguwa
committed
f37e872
change bert test to causal as unfused does not support padding
cyanguwa
committed
37f1469
fix FlashAttention for v2-2.3 versions
cyanguwa
committed
e278579
Merge branch 'main' into fa/sliding_window
cyanguwa
authored
37c1caf

Commits on Dec 7, 2023

verify FA swa works
cyanguwa
committed
49e147e

Commits on Dec 8, 2023

merge main
cyanguwa
committed
72447a1
fix mask related restrictions and duplicate code after merge
cyanguwa
committed
97f807f
fix swa test
cyanguwa
committed
603cb52

Commits on Dec 13, 2023

Merge branch 'main' into fa/sliding_window
cyanguwa
authored
ce15516

Commits on Dec 15, 2023

add docstring for get_swa func
cyanguwa
committed
e2fcf1d
move repeated code into a function
cyanguwa
committed
0f6b235
Merge branch 'main' into fa/sliding_window
cyanguwa
authored
f91d80d
revert mask change
cyanguwa
committed
67dd2d9
add determinism filter and fix FA warning message
cyanguwa
committed
576112d
add message for determinism filter
cyanguwa
committed
9c7de54
Merge branch 'main' into fa/sliding_window
cyanguwa
authored
c8afd21
simplify check_set_window_size()
cyanguwa
committed
53bf1f5
fix check_set_window_size in transformer layers
cyanguwa
committed
24252a2
fix indent
cyanguwa
committed
87c9826