CUDA: Various fixes to cpy.cu - #25000
Conversation
Extracted from ggml-org#24072
N = 8 * (65535 - 1) + 1 = 524273
Fall-back should share code, also relax y-z constraint to be inclusive
github-actions
Bot
added
testing
|
Just built and merged it into koboldcpp to test. |
Is this E2E comparison between CUDA/VULKAN? Or isolated to the changes of this PR |
|
It's cuda with this PR vs Vulkan. The test settings are Previously, the Cuda path would assert, but the vulkan path was fine. No changes to the speed of vulkan after this PR (which is expected) The s/IT metrics are taken during the generation. (this gen takes about 10 mins to complete) |
If you have an nsys report I could try to take a look, but current CUDA perf priorities are still for LLM inference. Though we are looking to improve PP perf a bit, and that should align with image/video gen workloads from the characteristics |
|
Happy to hear it's no longer crashing! |
|
I think further speed improvements need to come from stablediffusion.cpp. But for the time being, this is functional :) |
|
Thanks for looking into this and grats on 25000! Stable-diffusion.cpp maintainers should be aware of this fix now. I understand they run an older version of GGML, but with any luck should be able to incorporate these changes without issue. 😊👍 |
* Add failing test-case to test-backend-ops Extracted from ggml-org#24072 * Minimize repro with help of AI N = 8 * (65535 - 1) + 1 = 524273 * Port and adjust workaround from LostRuins@0ba7983 Fall-back should share code, also relax y-z constraint to be inclusive * Add test-case + fallback also for y dim * Fix x-guards which is 2^{31}-1, so inlusive of INT_MAX * Fix overflow problems for transposed copy kernel
* Add failing test-case to test-backend-ops Extracted from ggml-org#24072 * Minimize repro with help of AI N = 8 * (65535 - 1) + 1 = 524273 * Port and adjust workaround from LostRuins@0ba7983 Fall-back should share code, also relax y-z constraint to be inclusive * Add test-case + fallback also for y dim * Fix x-guards which is 2^{31}-1, so inlusive of INT_MAX * Fix overflow problems for transposed copy kernel
5 participants
Overview
Closes #24072 and does some more cleanup to the copy functions while going at it
Additional information
Concrete changes
<=INT_MAX/<=USHRT_MAX/<=USHRT_MAX, so they actually match cuda constraints, see https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/compute-capabilities.html#compute-capabilities-table-device-and-streaming-multiprocessor-sm-information-per-compute-capabilityRequirements