metal: Q2_0 backend

[khosravipasha]

Overview

Follow up PR after merging of Q2_0 CPU PR[#24448].
This PR adds the Metal backend support for Q2_0.

Supported Models

Supported Models: Ternary-Bonsai-{8B, 4B, 1.7B}-Q2_0_g64.gguf, more coming very soon.

Note: In our huggingface repos ggufs ending with _Q2_0_64.gguf are packed for this PR, Q2_0.gguf ones are older packing from our fork that have group 128, after all backends merge will rename and replace with the official Q2_0(group size 64)).

# downloads to models/Ternary-Bonsai-<size>-Q2_0_g64.gguf
hf download prism-ml/Ternary-Bonsai-1.7B-gguf Ternary-Bonsai-1.7B-Q2_0_g64.gguf --local-dir models
hf download prism-ml/Ternary-Bonsai-4B-gguf  Ternary-Bonsai-4B-Q2_0_g64.gguf  --local-dir models
hf download prism-ml/Ternary-Bonsai-8B-gguf  Ternary-Bonsai-8B-Q2_0_g64.gguf  --local-dir models

Correctness: Metal vs CPU logits KL

The ensure correctness we run the model twice: 1) Q2_0 CPU 2) Q2_0 Metal. We store the logits for both and compare the KL divergence. Using the standard llama-perplexity tool.

# reference logits from the CPU kernel
./build/bin/llama-perplexity -m models/Ternary-Bonsai-<size>-Q2_0_g64.gguf \
  -f datasets/wikitext-2-raw/wiki.test.raw -c 512 --chunks 20 -ngl 0 --save-all-logits cpu.bin
# Metal vs CPU
./build/bin/llama-perplexity -m models/Ternary-Bonsai-<size>-Q2_0_g64.gguf \
  -f datasets/wikitext-2-raw/wiki.test.raw -c 512 --chunks 20 -ngl 99 \
  --kl-divergence --kl-divergence-base cpu.bin

size	Mean KLD	Maximum KLD	Same top-1
1.7B	0.000000 ± 0.000000	0.000062	99.980 %
4B	0.000000 ± 0.000000	0.000067	99.980 %
8B	0.000000 ± 0.000000	0.000056	99.980 %

Metal Q2_0 matches the CPU reference (Mean KLD ~0, top-1 ≥99.9%).

Speeds: llama-bench Metal

./build/bin/llama-bench -m models/Ternary-Bonsai-<size>-Q2_0_g64.gguf -ngl 99 -p 512 -n 128 -r 5

| model           |       size |     params | backend  | ngl |  test |               t/s |
| --------------- | ---------: | ---------: | -------- | --: | ----: | ----------------: |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | MTL,BLAS |  99 | pp512 |  2148.81 ±  4.82  |
| qwen3 1.7B Q2_0 | 461.79 MiB |     1.72 B | MTL,BLAS |  99 | tg128 |   246.01 ±  0.57  |
| qwen3 4B Q2_0   |   1.05 GiB |     4.02 B | MTL,BLAS |  99 | pp512 |   856.51 ±  0.74  |
| qwen3 4B Q2_0   |   1.05 GiB |     4.02 B | MTL,BLAS |  99 | tg128 |   124.05 ±  1.44  |
| qwen3 8B Q2_0   |   2.15 GiB |     8.19 B | MTL,BLAS |  99 | pp512 |   457.85 ±  1.45  |
| qwen3 8B Q2_0   |   2.15 GiB |     8.19 B | MTL,BLAS |  99 | tg128 |    77.31 ±  0.37  |

Requirements

• I have read and agree with the contributing guidelines: Yes

• AI usage disclosure: Initial boilerplate code for Metal kernels were generated with AI for Q2_0 group size 64, manually reviewed all changes, and verified correctness using KL-test as above. Have been using the packed models and its working well as expected. Tested again after rebasing with new changes to make sure model is behaving as expected.

[khosravipasha]

rebased to fix some merge conflicts, few things were moved (e.g. kernel_get_rows_q2_0)

[ggerganov]

@khosravipasha Could you rebase to latest master to see if this will fix the CI?

[khosravipasha]

@ggerganov thanks, rebased with recent master.
Did not need any changes, rebased cleanly.

Running the CI on our fork as well PrismML-Eng#60

[khosravipasha]

This time more tests passed, some failures in ubuntu (x64, ubuntu-22.04). From glacing at it does not seem relate to this PR.