Skip to content

vulkan: parallelize DSV4 Lightning Indexer prefill - #3

Open
Mushoz wants to merge 1 commit into
Nathanw1014:strix-halo-vulkan-betafrom
Mushoz:strix-halo-vulkan-lightning-indexer
Open

vulkan: parallelize DSV4 Lightning Indexer prefill#3
Mushoz wants to merge 1 commit into
Nathanw1014:strix-halo-vulkan-betafrom
Mushoz:strix-halo-vulkan-lightning-indexer

Conversation

@Mushoz

@Mushoz Mushoz commented Aug 13, 2026

Copy link
Copy Markdown

Same disclaimer as my previous PR :)

I have measured the following performance improvements:

209 -> 214 improvement at 32k
175 -> 190 improvement at 64k

As this PR improves the lightning indexer, the improvement will grow as context depth increases as the time spent on the lightning indexer increases with increasing context length. Micro benches confirm that the performance increase hold at least until 512k context length:

Source depth kv Baseline Optimized Reduction
0 512 4.23 ms 2.14 ms 49.5%
8k 2560 17.10 ms 10.09 ms 41.0%
16k 4608 29.97 ms 17.62 ms 41.2%
32k 8704 50.51 ms 32.94 ms 34.8%
64k 16896 94.36 ms 64.87 ms 31.3%
128k 33280 174.96 ms 128.76 ms 26.4%
256k 66048 352.73 ms 250.88 ms 28.9%
512k 131584 696.79 ms 495.90 ms 28.8%

It's a single commit this time.

@github-actions github-actions Bot added documentation Improvements or additions to documentation Vulkan testing ggml labels Aug 13, 2026
@Mushoz

Mushoz commented Aug 13, 2026

Copy link
Copy Markdown
Author

Whoops, accidentally posted while I was still writing the description. Will edit the PR. My apologies for the early post.

Edit: Done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation ggml testing Vulkan

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant