Skip to content

ggml-webgpu: add support for NVFP4 - #25143

Merged
yomaytk merged 1 commit into
ggml-org:masterfrom
yomaytk:add-nvfp4
Jun 30, 2026
Merged

ggml-webgpu: add support for NVFP4#25143
yomaytk merged 1 commit into
ggml-org:masterfrom
yomaytk:add-nvfp4

Conversation

@yomaytk

@yomaytk yomaytk commented Jun 29, 2026

Copy link
Copy Markdown
Member

Overview

This PR adds NVFP4 support to ggml-webgpu. The following table shows the performance of CISCai/gemma-4-31B-it-NVFP4-turbo-GGUF on M2 and NVIDIA Tesla-V100.

M2:

Model Test master (t/s) PR (t/s)
gemma4 31B NVFP4 pp512 13.61 ± 0.78 44.06 ± 8.07
gemma4 31B NVFP4 tg128 0.90 ± 0.06 2.51 ± 0.01

Tesla-V100:

Model Test master (t/s) PR (t/s)
gemma4 31B NVFP4 pp512 1.39 ± 0.00 30.38 ± 0.09
gemma4 31B NVFP4 tg128 1.04 ± 0.00 3.35 ± 0.04

Since WebGPU doesn't seem to provide direct access to the GPU instructions of the host device, NVFP4 likely offers no performance advantage over k-quants on WebGPU backend.
The primary motivation is ecosystem coverage — NVFP4 GGUFs are increasingly available on huggingface, and this ensures ggml-webgpu can run them rather than falling back to CPU backend.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, I used AI to generate some boilerplate code.

@yomaytk
yomaytk requested a review from a team as a code owner June 29, 2026 18:21
@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning WebGPU labels Jun 29, 2026
@reeselevine

Copy link
Copy Markdown
Contributor

nice, I agree that if right now people are using nvfp4 on webgpu they should probably be pointed towards using a k-quant instead, but still nice to have some acceleration if for some reason they insist on using them :)

@yomaytk
yomaytk merged commit 6c5de1c into ggml-org:master Jun 30, 2026
39 of 42 checks passed
@yomaytk
yomaytk deleted the add-nvfp4 branch June 30, 2026 08:20
zengde added a commit to zengde/llama.cpp that referenced this pull request Jun 30, 2026
ggml-webgpu: add support for NVFP4 (ggml-org#25143)
turbo-tan pushed a commit to turbo-tan/llama.cpp-tq3 that referenced this pull request Jul 1, 2026
DFveloper pushed a commit to DFveloper/aikar-engine that referenced this pull request Jul 3, 2026
adrianhoehne pushed a commit to adrianhoehne/llama.cpp that referenced this pull request Jul 5, 2026
CowboyTim pushed a commit to aardbeiplantje/llama.cpp that referenced this pull request Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ggml changes relating to the ggml tensor library for machine learning WebGPU

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants