ggml: add Q2_0 quantization support (CPU)

[khosravipasha]

Overview

This PR adds Q2_0 support for CPU. Main motivation is to support Ternary Bonsai models (1.7B, 4B, 8B) and upcoming models. This PR is CPU only (ARM NEON + generic scalar fallback).
This completes the Q1_0, Q2_0, Q4_0, Q8_0 family.
We have the x86, Metal, CUDA, and Vulkan backends ready to submit later.

Notes

• Format: Each group of 64 weights shares one fp16 scale d; weights are packed at 2 bits each with
  mapping of {0,1,2,3} => {-1,0,+1,+2} * d

• Our models natively support group size 128; however, it was requested to do group size 64 for the official Q2_0 format (see discussion #22019), so this PR uses 64.

• We plan to also maintain a sibling group-128 variant (PQ2_0) in our fork since the
  0.125 extra bpw becomes significant on larger models. If you have a cleaner way to do this, please let us know. For future release we will pack the models into both Q2_0 and PQ2_0 formats.

• Why not use TQ1_0 / TQ2_0? They support group size 256, our models are group size 128, and also cpu-only and harder to accelerate on Metal/CUDA, etc. More info in: discussion #22019

Speed/Correctness Summary

• Speeds: All on Mac M4 Pro (ARM NEON, -t 8, -ngl 0).
• Correctness: Compare unpacked (from FP16 gguf) logits vs packed (Q2_0 gguf) logits on.
  More details and raw outputs in appendix.

Size	Q2_0 size (vs F16)	tg128 Q2_0 / F16	pp512 Q2_0 / F16	Mean KLD	Same top-1
1.7B	461.79 MiB / 3.20 GiB	117.20 / 48.70 t/s	170.51 / 200.15 t/s	0.000204	99.392 %
4B	1.05 GiB / 7.49 GiB	55.00 / 27.23 t/s	68.31 / 80.64 t/s	0.000130	99.373 %
8B	2.15 GiB / 15.25 GiB	30.14 / 14.88 t/s	36.36 / 38.98 t/s	0.000120	99.314 %

Additional Info

Model Repos:

• prism-ml/Ternary-Bonsai-1.7B-gguf
• prism-ml/Ternary-Bonsai-4B-gguf
• prism-ml/Ternary-Bonsai-8B-gguf

Links + Evals

More info on the models and working demos can be found below:

• Demo Repo: https://github.com/PrismML-Eng/Bonsai-demo
• Whitepaper: ternary-bonsai-8b-whitepaper.pdf

pip install -U "huggingface_hub[cli]"

# F16 reference + Q2_0 group-64 GGUF (swap 1.7B -> 4B / 8B for other sizes)
hf download prism-ml/Ternary-Bonsai-1.7B-gguf Ternary-Bonsai-1.7B-F16.gguf      --local-dir models
hf download prism-ml/Ternary-Bonsai-1.7B-gguf Ternary-Bonsai-1.7B-Q2_0_g64.gguf --local-dir models

Each repo has three gguf variants:

• Q2_0_g64.gguf: the new group-64 format this PR adds. The _g64 suffix is for
  convenience; it will be renamed to plain Q2_0 once these PRs merge. Use this file with this PR.
• Q2_0.gguf: the old Q2_0 from our fork (group 128); predates the group-64 change
  and does not load with this PR. Will be deleted/renamed once this PR merges.
• PQ2_0.gguf: the sibling format with group size 128 we keep maintaining in our fork (fork-only,
  not part of this PR).

Testing

Tested on Mac M4 Pro, 48 GB. Two CPU routes: ARM NEON and Generic Scalar Fallback
(generic built by steering ggml to the portable path: GGML_SYSTEM_ARCH=UNKNOWN, NEON
arch/arm/quants.c not compiled).

Pack to Q2_0 (from F16 GGUF)

./build/bin/llama-quantize --pure models/Ternary-Bonsai-1.7B-F16.gguf models/Ternary-Bonsai-1.7B-Q2_0_g64.gguf Q2_0

llama_model_quantize_impl: model size  =  3280.93 MiB (16.00 BPW)
llama_model_quantize_impl: quant size  =   461.79 MiB (2.25 BPW)

Speed Benchmarks Details

./build/bin/llama-bench -m <model.gguf> -t 8 -ngl 0 -p 512 -n 128

ARM NEON

| model           |       size |     params | backend | threads |  test |            t/s |
| --------------- | ---------: | ---------: | ------- | ------: | ----: | -------------: |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | CPU     |       8 | pp512 | 170.51 ± 2.48  |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | CPU     |       8 | tg128 | 117.20 ± 0.17  |
| qwen3 1.7B F16  |   3.20 GiB |     1.72 B | CPU     |       8 | pp512 | 200.15 ± 7.18  |
| qwen3 1.7B F16  |   3.20 GiB |     1.72 B | CPU     |       8 | tg128 |  48.70 ± 1.44  |
| qwen3 4B Q2_0   |   1.05 GiB |     4.02 B | CPU     |       8 | pp512 |  68.31 ± 0.30  |
| qwen3 4B Q2_0   |   1.05 GiB |     4.02 B | CPU     |       8 | tg128 |  55.00 ± 0.12  |
| qwen3 4B F16    |   7.49 GiB |     4.02 B | CPU     |       8 | pp512 |  80.64 ± 0.47  |
| qwen3 4B F16    |   7.49 GiB |     4.02 B | CPU     |       8 | tg128 |  27.23 ± 0.26  |
| qwen3 8B Q2_0   |   2.15 GiB |     8.19 B | CPU     |       8 | pp512 |  36.36 ± 0.15  |
| qwen3 8B Q2_0   |   2.15 GiB |     8.19 B | CPU     |       8 | tg128 |  30.14 ± 0.50  |
| qwen3 8B F16    |  15.25 GiB |     8.19 B | CPU     |       8 | pp512 |  38.98 ± 1.01  |
| qwen3 8B F16    |  15.25 GiB |     8.19 B | CPU     |       8 | tg128 |  14.88 ± 0.03  |

Generic Scalar Fallback (1.7B, small -p 16 -n 8)

| model           |       size |     params | backend | threads | test |           t/s |
| --------------- | ---------: | ---------: | ------- | ------: | ---: | ------------: |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | CPU     |       8 | pp16 | 24.70 ± 0.32  |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | CPU     |       8 |  tg8 | 19.31 ± 0.30  |

KL Kernel Accuracy Test Details (Q2_0 g64 vs F16, packed vs unpacked)

# Step 1: save F16 reference logits
./build/bin/llama-perplexity -m models/Ternary-Bonsai-1.7B-F16.gguf \
  -f datasets/wikitext-2-raw/wiki.test.raw -c 512 --chunks 20 \
  --save-all-logits models/f16_logits.bin

# Step 2: KL Q2_0 vs F16
./build/bin/llama-perplexity -m models/Ternary-Bonsai-1.7B-Q2_0_g64.gguf \
  -f datasets/wikitext-2-raw/wiki.test.raw -c 512 --chunks 20 \
  --kl-divergence --kl-divergence-base models/f16_logits.bin

ARM NEON: Q2_0 g64 vs F16, by size

| Metric      |       1.7B       |        4B        |        8B        |
| ----------- | ---------------- | ---------------- | ---------------- |
| Same top p  | 99.392 ± 0.109 % | 99.373 ± 0.111 % | 99.314 ± 0.116 % |
| Mean KLD    | 0.000204 ± 4e-6  | 0.000130 ± 2e-6  | 0.000120 ± 3e-6  |
| Maximum KLD |        0.008953  |        0.002888  |        0.004727  |

1.7B: full statistics (ARM NEON vs F16)

====== KL divergence statistics ======
Mean    KLD:   0.000204 ±   0.000004
Maximum KLD:   0.008953
99.9%   KLD:   0.003159
99.0%   KLD:   0.001131
95.0%   KLD:   0.000647
90.0%   KLD:   0.000475
10.0%   KLD:   0.000001
 5.0%   KLD:  -0.000000
 1.0%   KLD:  -0.000003
 0.1%   KLD:  -0.000024
Minimum KLD:  -0.000041
Same top p: 99.392 ± 0.109 %

1.7B: Generic Scalar Fallback vs F16 (matches NEON)

====== KL divergence statistics ======
Mean    KLD:   0.000203 ±   0.000004
Maximum KLD:   0.003817
99.9%   KLD:   0.002900
99.0%   KLD:   0.001182
95.0%   KLD:   0.000626
90.0%   KLD:   0.000482
10.0%   KLD:   0.000001
 5.0%   KLD:  -0.000000
 1.0%   KLD:  -0.000004
 0.1%   KLD:  -0.000035
Minimum KLD:  -0.000046
Same top p: 99.059 ± 0.135 %

Requirements

• I have read and agree with the contributing guidelines: Yes
• AI usage disclosure: NEON paths were generated with AI help to follow Q1_0 NEON path but with Q2_0 logic, manually verified correctness using KL-test as above. Have been using the packed models and its working well as expected.

[bri-prism]

Ran the CPU correctness checks for this Q2_0 (group 64) type on two architectures, both green:

test-quantize-fns (round-trip + vec_dot against the scalar reference):

arch	CPU	result
x86-64	AMD EPYC (Zen 4, AVX-512)	✅ pass, 0 failures (q2_0 exercised)
ARM64	Apple Silicon (NEON + dotprod)	✅ pass, 0 failures (q2_0 exercised)

Both built clean from this branch (CPU backend, -DGGML_NATIVE=ON). On x86 the dot resolves to the generic/scalar path (no x86 SIMD kernel in this PR); on ARM the NEON ggml_vec_dot_q2_0_q8_0 is exercised directly. test-backend-ops is GPU-only and skips the CPU backend, so test-quantize-fns is the relevant CPU gate here.

No issues found on the CPU side.

[CISC]

Just so you are prepared, our current workload means this will take some time to get reviewed, and #22836 is first in the quant queue. :)

[khosravipasha]

@CISC, sounds good, thanks for the heads up.
Should we change the GGML_TYPE_Q2_0 = 43 since that one will be 42 (and +1 to the other enums).
Is the other one almost ready to merge? its been around a while.

[CISC]

@CISC, sounds good, thanks for the heads up. Should we change the GGML_TYPE_Q2_0 = 43 since that one will be 42 (and +1 to the other enums). Is the other one almost ready to merge? its been around a while.

Hard to say unfortunately, new quants in particular are hard to allocate reviewer/time for.

No point in updating enums until conflicts are merged.

[max-krasnyansky]

This looks very clean to me.
Very similar to Q4_0 in terms of the overall integration (kernels, blocks, etc).
I'm interested in adding this to ggml-hexagon since it maps pretty well.
Unfortunately, the compute would have to be done in FP16 (HMX) and INT8 (HVX) but the memory footprint reduction will definitely help speed things up.

@khosravipasha Please rebase with the latest master and I'll approve the workflow to kick the CI

[khosravipasha]

@max-krasnyansky
Thanks, just rebased and fixed the small merge conflict in tests/test-quantize-fns.cpp.
Reran some of our tests and should be ok to try now rebase did not break anything.

For now if you want to try with one of the uploaded models use the _g64 version (Ternary-Bonsai-8B-Q2_0_g64.gguf). The Q2_0's are the ones with group size 128 from our fork, after our PRs get merged will replace with official Q2_0 from here.

Nice, ggml-hexagon sounds exciting, what format is nicer for it, the scales also be int16 or int8?

[max-krasnyansky]

@max-krasnyansky Thanks, just rebased and fixed the small merge conflict in tests/test-quantize-fns.cpp. Reran some of our tests and should be ok to try now rebase did not break anything.

For now if you want to try with one of the uploaded models use the _g64 version (Ternary-Bonsai-8B-Q2_0_g64.gguf). The Q2_0's are the ones with group size 128 from our fork, after our PRs get merged will replace with official Q2_0 from here.

Nice, ggml-hexagon sounds exciting, what format is nicer for it, the scales also be int16 or int8?

Block size 32 would be the most ideal for ggml-hexagon at this point. It's be the easiest to map to our new 32x32 weight repack format.

[khosravipasha]

Seems there is some conflicts now, will resolve them.

@max-krasnyansky Main issue with group size 32 will have large overhead will become 2.5 bpw.

[khosravipasha]

Fixed two merge conflicts
-arch-fallback.h: nvfp4 removed something in fallbacks and causing conflict should be good now.
-llama-model-loader.cpp: Picked up some refactoring related to LLAMA_FTYPE_PREFIX in llama_ftype_name().

[cfournel]

We need this PR !

[max-krasnyansky]

Fixed two merge conflicts -arch-fallback.h: nvfp4 removed something in fallbacks and causing conflict should be good now. -llama-model-loader.cpp: Picked up some refactoring related to LLAMA_FTYPE_PREFIX in llama_ftype_name().

Sounds good. I can work with 64 and 128 too.

[Green-Sky]

Tried this with sd.cpp and your ternary flux2 klein 4B qats.

But obscenely slow on cpu right now.

[ValdikSS]

@khosravipasha letting you know that Ternary-Bonsai-27B-Q2_0.gguf does not work on x86_64 CPU in llama.cpp with your patch right now (b10015, commit 12127de)

$ ./llama-server --no-warmup -dev none -m Ternary-Bonsai-27B-Q2_0.gguf

0.00.008.023 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.009.134 W srv  llama_server: -----------------
0.00.009.141 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.009.141 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.009.142 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.00.009.142 W srv  llama_server: -----------------
0.00.010.773 I srv    load_model: loading model 'Ternary-Bonsai-27B-Q2_0.gguf'
0.00.064.719 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.064.725 E gguf_init_from_reader: failed to read tensor data
0.00.068.645 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-Q2_0.gguf
0.00.068.649 E llama_model_load_from_file_impl: failed to load model
0.00.068.672 E common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
0.00.110.225 E gguf_init_from_reader: tensor 'output_norm.weight' has offset 337715200, expected 357580800
0.00.110.227 E gguf_init_from_reader: failed to read tensor data
0.00.113.696 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-Q2_0.gguf
0.00.113.698 E llama_model_load_from_file_impl: failed to load model
0.00.113.703 E cmn  common_init_: failed to load model 'Ternary-Bonsai-27B-Q2_0.gguf'
0.00.113.706 E srv    load_model: failed to load model, 'Ternary-Bonsai-27B-Q2_0.gguf'
0.00.113.708 I srv    operator(): operator(): cleaning up before exit...
0.00.114.452 E srv  llama_server: exiting due to model loading error

[khosravipasha]

@ValdikSS
Need to use Ternary-Bonsai-1.7B-Q2_0_g64.gguf instead as mentioned in the PR.

Currently models ending in Q2_0.gguf is from our fork which has different formatting, waiting for all main backends to merge here (only one left is cuda) to do replacement and migration (i.e. renaming Q2_0_g64 to Q2_0). Was not expecting these to merge before our new launch.

[ggerganov]

Format: Each group of 64 weights shares one fp16 scale d; weights are packed at 2 bits each with mapping of {0,1,2,3} => {-1,0,+1,+2} * d

@khosravipasha For my understanding, is it correct that the Bonsai ternary models never utilize the +2 value of the Q2_0 container? Looking at the whitepaper, this seems to be implied, but want to make sure this is correct:

[khosravipasha]

@ggerganov Yes, that's correct current models only use the {0,1,-1}. But we pack it into 2-bit as it was easier to accelerate on Metal/CUDA. That being said we might do 2-bit models too (or mixed quants of tensors) in the future that utilizes the 2 so Q2_0 should be useful for those.