ggml : address integer overflows in binary ops CUDA implementation

[fairydreaming]

Overview

This PR addresses the problem of integer overflows when processing large tensors with GGML binary OPs in CUDA backend. It's not a major overhaul, but rather a least-intrusive way of improving the current implementation.

Fixes #24643

Additional information

The problem was addressed by:

• changing int type usage to uint32_t to double usable integer range,
• adding casts so that multiplication result of two uint32_t won't overflow,
• using asserts to verify before kernel launches that passed uint32_t parameters (and their products inside the kernels) will stay inside uint32_t type range,
• making sure that i0 variable doesn't overflow when incremented in loop iterations,

I ran ./bin/test-backend-ops (all tests passed), ran failing tests from #24643 (passed) and compared performance of example small and large tensor f32 addition in old and new implementations (was the same new impl after adding size_t casts is slightly slower, ~1.2% for small tensors, ~0.3% for large ones).

Additionally I compared kernel register usage in old and new implementation. In most kernels (126) register usage was the same, in 17 kernels it was lower by 1-2 registers and in 10 kernels it was higher by 1-2 registers. With added size_t casts: unchanged in 74 kernels, lower by 1-2 registers in 15 kernels, higher by 1-4 registers in 64 kernels.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, discussed with AI possible ways of improving the implementation

[ORippler]

Could you share the measured performance numbers?

[fairydreaming]

@ORippler performance:

Without PR:

$ ./bin/test-backend-ops perf -o "ADD"
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97247 MiB):
  Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes, VRAM: 97247 MiB
Testing 2 devices

Backend 1/2: CUDA0
  Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
  Device memory: 97247 MB (96640 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0):              966420 runs -     1.04 us/run -       48 kB/run -   44.10 GB/s
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1],nf=1,perm1=0,src_overlap=0):            113378 runs -     8.86 us/run -    24576 kB/run - 2645.75 GB/s
  Backend CUDA0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

With PR:

$ ./bin/test-backend-ops perf -o "ADD"
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97247 MiB):
  Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes, VRAM: 97247 MiB
Testing 2 devices

Backend 1/2: CUDA0
  Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
  Device memory: 97247 MB (96640 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0):              892710 runs -     1.13 us/run -       48 kB/run -   40.62 GB/s
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1],nf=1,perm1=0,src_overlap=0):            113378 runs -     8.84 us/run -    24576 kB/run - 2650.32 GB/s
  Backend CUDA0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

Looks like I missed 10% perf degradation for very small tensors caused by size_t() casts, investigating.

[fairydreaming]

@ORippler Added missing size_t casts, now performance is even higher than original:

$ ./bin/test-backend-ops perf -o "ADD"
ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97247 MiB):
  Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes, VRAM: 97247 MiB
Testing 2 devices

Backend 1/2: CUDA0
  Device description: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition
  Device memory: 97247 MB (96640 MB free)

ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0):             1007370 runs -     0.99 us/run -       48 kB/run -   46.03 GB/s
ggml_backend_cuda_graph_compute: CUDA graph warmup complete
  ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1],nf=1,perm1=0,src_overlap=0):            113378 runs -     8.86 us/run -    24576 kB/run - 2645.82 GB/s
  Backend CUDA0: OK
Backend 2/2: CPU
  Skipping CPU backend
2/2 backends passed
OK

Register usage: unchanged in 74 kernels, lower by 1-2 registers in 15 kernels, higher by 1-4 registers in 64 kernels.

Edit: I did a clean recompile of the project and now perf is again a little lower (43.21 GB/s for small matrix, so ~1.2% slower, 2631.88 GB/s for large, so 0.3% slower). Honestly I have no idea what causes these fluctuations.

[ORippler]

Edit: I did a clean recompile of the project and now perf is again a little lower (43.21 GB/s for small matrix, so ~1.2% slower, 2631.88 GB/s for large, so 0.3% slower). Honestly I have no idea what causes these fluctuations.

The perf benchmarks do not factor out:

• Cache hits (cache is not invalidated/flushed)
• Clocking differences
• cudaGraph reuse

which may affect performance and introduce r2r variation. At 1 and 8 us/run this is pretty fast already (typically one cannot go faster than 2-5 us as at these are the minimums pro/epilogue overheads of launching a cuda kernel (constructing & tearing down CTAs etc.)

[fairydreaming]

@ORippler Yeah, I checked clock values of my Max-Q and it was doing pretty wild swings. Locked it to 1410 MHz, increased test time from 1s to 10s and now got this (also added some more tests):

Test	master GB/s	this GB/s	change
ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0)	28.27	29.55	+4.53%
ADD(type=f32,ne=[4096,256,1,1],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0)	1651.68	1638.44	-0.80%
ADD(type=f32,ne=[4096,256,16,1],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0)	1327.07	1322.73	-0.33%
ADD(type=f32,ne=[4096,256,16,4],nr=[1,1,1,1],nf=1,perm1=0,src_overlap=0)	1238.83	1236.64	-0.18%
ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1],nf=1,perm1=1,src_overlap=0)	28.41	29.75	+4.72%
ADD(type=f32,ne=[4096,256,1,1],nr=[1,1,1,1],nf=1,perm1=1,src_overlap=0)	759.65	756.38	-0.43%
ADD(type=f32,ne=[4096,256,16,1],nr=[1,1,1,1],nf=1,perm1=1,src_overlap=0)	848.00	847.67	-0.04%
ADD(type=f32,ne=[4096,256,16,4],nr=[1,1,1,1],nf=1,perm1=1,src_overlap=0)	854.89	854.88	-0.00%
ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1],nf=1,perm1=0,src_overlap=0)	1626.74	1629.70	+0.18%

[ggerganov]

Apart from the performance concerns/analysis, I am in favor of pretty much casting all (or most) i32*i32 to 64-bits. It's a rule I follow in the Metal backend as well.

[ORippler]

Apart from the performance concerns/analysis, I am in favor of pretty much casting all (or most) i32*i32 to 64-bits. It's a rule I follow in the Metal backend as well.

Generally agree. However, doing ALU in 64-bit requires different HW than 32-bit, and thus perf implications depend on HW (for NVGPUs one wants to use 32-bit ALU where possible/reasonable). Anyways for this instance it's a non-issue.

E2E perf numbers

``` ggml_cuda_init: found 1 CUDA devices (Total VRAM: 97250 MiB): Device 0: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, compute capability 12.0, VMM: yes, VRAM: 97250 MiB | model | size | params | backend | ngl | fa | dio | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | --: | --: | --------------: | -------------------: | | qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | CUDA | -1 | 1 | 1 | pp512 | 7567.30 ± 123.94 | | qwen35moe 35B.A3B Q4_K - Medium | 20.60 GiB | 34.66 B | CUDA | -1 | 1 | 1 | tg128 | 246.50 ± 2.59 | | qwen35moe 35B.A3B NVFP4 | 19.51 GiB | 34.66 B | CUDA | -1 | 1 | 1 | pp512 | 8792.63 ± 24.38 | | qwen35moe 35B.A3B NVFP4 | 19.51 GiB | 34.66 B | CUDA | -1 | 1 | 1 | tg128 | 215.35 ± 2.16 | | gemma4 26B.A4B NVFP4 | 16.45 GiB | 25.23 B | CUDA | -1 | 1 | 1 | pp512 | 10716.34 ± 188.49 | | gemma4 26B.A4B NVFP4 | 16.45 GiB | 25.23 B | CUDA | -1 | 1 | 1 | tg128 | 168.72 ± 0.52 | | gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | -1 | 1 | 1 | pp512 | 17384.51 ± 124.35 | | gpt-oss 20B MXFP4 MoE | 11.27 GiB | 20.91 B | CUDA | -1 | 1 | 1 | tg128 | 373.90 ± 1.48 | | gemma4 26B.A4B Q4_K - Medium | 15.90 GiB | 25.23 B | CUDA | -1 | 1 | 1 | pp512 | 9645.72 ± 223.84 | | gemma4 26B.A4B Q4_K - Medium | 15.90 GiB | 25.23 B | CUDA | -1 | 1 | 1 | tg128 | 214.47 ± 0.89 |

build: bf95565 (9674)

• ./scripts/compare-llama-bench.py -b 74ade52 -c binary-ops-overflows --tool llama-bench -i llama-bench.sqlite
  | CPU | Model | Test | t/s 74ade52 | t/s binary-ops-overflows | Speedup |
  |:----------------------------|:-------------------------|:-------|----------------:|---------------------------:|----------:|
  | INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B NVFP4 | pp512 | 10742.21 | 10716.34 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B NVFP4 | tg128 | 168.61 | 168.72 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B Q4_K_M | pp512 | 9624.92 | 9645.72 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | gemma4 26B.A4B Q4_K_M | tg128 | 214.42 | 214.47 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | gpt-oss 20B MXFP4 MoE | pp512 | 17330.25 | 17384.51 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | gpt-oss 20B MXFP4 MoE | tg128 | 373.81 | 373.90 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B NVFP4 | pp512 | 8791.00 | 8792.63 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B NVFP4 | tg128 | 213.90 | 215.35 | 1.01 |
  | INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B Q4_K_M | pp512 | 7578.87 | 7567.30 | 1.00 |
  | INTEL(R) XEON(R) GOLD 6542Y | qwen35moe 35B.A3B Q4_K_M | tg128 | 245.85 | 246.50 | 1.00 |

[ORippler]

Thanks for bearing with the multi-review-rounds