Skip to content

[Kernel]Remove recast iter from sliding window attention kernel. - #997

Open
amd-nprotaso wants to merge 15 commits into
ROCm:mainfrom
amd-nprotaso:main
Open

[Kernel]Remove recast iter from sliding window attention kernel.#997
amd-nprotaso wants to merge 15 commits into
ROCm:mainfrom
amd-nprotaso:main

Conversation

@amd-nprotaso

Copy link
Copy Markdown
Contributor

#933 was merged, now we can use add_offset without recast_iter, with equivalent performance:

B seq win prev ms cur ms prev/cur prev TF cur TF
1 1024 (512,0) 0.0297 0.0300 0.99x 217.6 215.3
1 2048 (512,0) 0.0353 0.0350 1.01x 427.3 430.6
1 2048 (1024,0) 0.0469 0.0475 0.99x 549.9 543.0
1 4096 (512,0) 0.0734 0.0697 1.05x 440.0 463.3
1 4096 (1024,0) 0.1049 0.1012 1.04x 573.9 594.5
1 8192 (512,0) 0.1581 0.1616 0.98x 422.0 412.7
1 8192 (1024,0) 0.2325 0.2398 0.97x 554.8 538.0
1 8192 (4096,0) 0.5402 0.5459 0.99x 763.5 755.5
1 16384 (4096,0) 1.1505 1.1471 1.00x 836.4 838.9
1 32768 (8192,0) 4.3768 4.3778 1.00x 879.4 879.2
1 65536 (8192,0) 9.2751 9.2596 1.00x 889.2 890.7
1 131072 (8192,0) 19.0403 19.0032 1.00x 895.2 896.9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant