hexagon: MUL_MAT, MUL_MAT_ID, FLASH_ATTN and GDN cleanup and optimizations for latest models by max-krasnyansky · Pull Request #23989 · ggml-org/llama.cpp

max-krasnyansky · 2026-06-01T18:47:16Z

Overview

Fairly large PR but mostly due to the inner kernel updates.
HVX matmuls have lots of hand-unrolled code (i.e all those vec_dots).
I could technically split the GATE_DELTA_NET but I did lots of testing on the latest so I'd rather keep it.
The main goal here is to continue improving these core Ops for latest models and use-cases.

Main features:

Support for F32 x F32 -> F32 MUL_MAT
- Lots of F32 2D Weights in gemma-4 Q4 GGUFs
  (need to followup with unsloth and bartowski, seems like F16 would be sufficient and will help with perf).
MUL_MAT HMX rework
- Resurrect pipelined vs non-pipelined modes but with full reuse of infra
- Funnel all data types through the common path and reuse the quant/dequant helpers
- F32 MUL_MAT is handled via this path and we dynamically quantize F32 into F16
- Reworked F16 batched mode to reuse infra from 2D
MUL_MAT_ID support for HMX
- This is just a first pass with existing features like the expert sorting and some new ones like expert coalescing
FLASH_ATTN_EXT HMX pipeline improvements similar to MUL_MAT
GATED_DELTA_NET general optimizations
- Better DMA/VTCM usage, vectorize everything with HVX, etc

Additional information

Details

| model              |    test | master-06-01 t/s | this PR      t/s |
| ------------------ | ------: | ---------------: | ---------------: |
| gemma4 E2B Q4_0    |   pp200 |    606.27 ± 5.54 |    730.15 ± 7.98 |
| gemma4 E2B Q4_0    |   pp512 |   662.53 ± 19.48 |    865.01 ± 7.86 |
| gemma4 E2B Q4_0    |   pp800 |    642.75 ± 7.70 |    818.92 ± 9.06 |
| gemma4 E2B Q4_0    |  pp1024 |    629.25 ± 5.02 |    824.50 ± 1.79 |
| gemma4 E2B Q4_0    |  pp1500 |    598.96 ± 3.55 |    779.82 ± 5.89 |
| gemma4 E2B Q4_0    |  pp2048 |    578.85 ± 5.87 |    691.49 ± 4.16 |
| gemma4 E2B Q4_0    |   tg128 |     30.64 ± 0.17 |     31.25 ± 0.28 |
| qwen35 0.8B Q4_0   |   pp200 |   1087.45 ± 2.66 |   1131.51 ± 2.20 |
| qwen35 0.8B Q4_0   |   pp512 |   1287.14 ± 5.32 |   1382.64 ± 3.39 |
| qwen35 0.8B Q4_0   |   pp800 |   1320.10 ± 6.23 |   1411.01 ± 5.71 |
| qwen35 0.8B Q4_0   |  pp1024 |   1358.43 ± 6.31 |   1468.63 ± 8.16 |
| qwen35 0.8B Q4_0   |  pp1500 |   1325.45 ± 5.38 |   1432.14 ± 3.68 |
| qwen35 0.8B Q4_0   |  pp2048 |   1338.11 ± 2.13 |   1451.04 ± 3.57 |
| qwen35 0.8B Q4_0   |   tg128 |     46.53 ± 2.22 |     46.51 ± 2.24 |
| qwen35 2B Q4_0     |   pp200 |    856.91 ± 6.00 |    897.92 ± 8.44 |
| qwen35 2B Q4_0     |   pp512 |   1048.72 ± 4.45 |   1138.10 ± 1.71 |
| qwen35 2B Q4_0     |   pp800 |   1066.56 ± 1.05 |   1180.07 ± 3.48 |
| qwen35 2B Q4_0     |  pp1024 |   1008.94 ± 8.17 |   1116.61 ± 5.93 |
| qwen35 2B Q4_0     |  pp1500 |    990.42 ± 7.04 |   1114.71 ± 5.36 |
| qwen35 2B Q4_0     |  pp2048 |    993.66 ± 6.11 |   1106.30 ± 5.11 |
| qwen35 2B Q4_0     |   tg128 |     27.13 ± 0.71 |     27.71 ± 0.12 |
| olmoe A1.7B Q4_0   |   pp200 |    166.27 ± 1.29 |   449.06 ± 12.92 |
| olmoe A1.7B Q4_0   |   pp512 |    174.18 ± 2.34 |    835.00 ± 5.83 |
| olmoe A1.7B Q4_0   |   pp800 |    284.00 ± 0.75 |   959.89 ± 19.53 |
| olmoe A1.7B Q4_0   |  pp1024 |    285.45 ± 0.92 |   1263.41 ± 8.82 |
| olmoe A1.7B Q4_0   |  pp1500 |    229.69 ± 4.06 |  1024.54 ± 13.66 |
| olmoe A1.7B Q4_0   |  pp2048 |    275.94 ± 0.20 |  1206.79 ± 23.22 |
| olmoe A1.7B Q4_0   |   tg128 |     32.19 ± 0.23 |      32.5 ± 0.73 |
| lfm2 1.2B Q4_0     |   pp200 |   2344.05 ± 8.88 |   2404.90 ± 7.42 |
| lfm2 1.2B Q4_0     |   pp512 |   2771.81 ± 4.27 |   2866.39 ± 9.35 |
| lfm2 1.2B Q4_0     |   pp800 |   2598.43 ± 2.91 |   2705.05 ± 2.75 |
| lfm2 1.2B Q4_0     |  pp1024 |   2489.13 ± 5.87 |   2503.42 ± 5.22 |
| lfm2 1.2B Q4_0     |  pp1500 |   2440.16 ± 2.71 |   2498.16 ± 2.37 |
| lfm2 1.2B Q4_0     |  pp2048 |   2250.42 ± 1.49 |   2320.66 ± 1.25 |
| lfm2 1.2B Q4_0     |   tg128 |     51.65 ± 2.61 |     57.81 ± 1.60 |

Generated on S26+ with

   LD_LIBRARY_PATH=/data/local/tmp/llama.cpp/lib   \
   ADSP_LIBRARY_PATH=/data/local/tmp/llama.cpp/lib \
   ./bin/llama-bench --device HTP0 -m \
       ../gguf/gemma-4-E2B-it-Q4_0.gguf,\
       ../gguf/Qwen3.5-0.8B-Q4_0.gguf,\
       ../gguf/Qwen3.5-2B-Q4_0.gguf,\
       ../gguf/OLMoE-1B-7B-0125-Instruct-Q4_0.gguf,\
       ../gguf/LFM2.5-1.2B-Instruct-Q4_0.gguf \
       -t 6 --poll 1000 -fa 1 --ubatch-size 1024 \
       -p 200,512,800,1024,1500,2048

(columns trimmed to remove noise)

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: Yes, used Antigravity CLI to reason about pipelining, review the changes and help with refactoring and bug hunting.

This fixes an odd case where fused rms-norm-mul was failing but only in qwen3.5-2B and only at searth op-bath sizes.

Decided to use Q4_0 * F32 -> F32 matmul for this. Q4_0 gets dequantized and tiled into F16, and here we quantize and tile F32 into F16. Super simple and pretty efficient.

…er types

…have but is much more generic way This update futher improves matmul performance and at the same time removes most of the redudant logic we had in different paths.

DMA prefetch/double-buff, vectorize everything with HVX, in other words -- the usual :)

max-krasnyansky · 2026-06-01T22:02:05Z

@lhez and @ggml-org/maintainers please take a look and approve

…eed up FA alibi

… the src tensors when some srcs are empty

…t quite right

…tions for latest models (ggml-org#23989) * hex-mm: initial support for F32 * F32 -> F32 matmuls * hex-rms-norm: fix src1 stride use in fused rms_norm_mul * hex-ops: clear spad pointers in the ops that clober it This fixes an odd case where fused rms-norm-mul was failing but only in qwen3.5-2B and only at searth op-bath sizes. * hmx-mm: add support for F32 * F32 -> F32 matmul_2d on HMX Decided to use Q4_0 * F32 -> F32 matmul for this. Q4_0 gets dequantized and tiled into F16, and here we quantize and tile F32 into F16. Super simple and pretty efficient. * hmx-mm: route f16 2D matmuls through the same kernel used for all other types * hmx-mm: re-introduce pipelined vs non-pipelined mode that we used to have but is much more generic way This update futher improves matmul performance and at the same time removes most of the redudant logic we had in different paths. * hmx-fa: slighlty improved pipeline simimar to matmul updates * hmx-mm: initial version of MAT_MUL_ID support for HMX * hmx-mm: fixed mxfp4 handling for MUL_MAT_ID * hex-gdn: optimize GATED_DELTA_NET DMA prefetch/double-buff, vectorize everything with HVX, in other words -- the usual :) * hmx-mm: missed one more case where we can use fastmod * hexagon: update DCVS settings for a slight perf bump * hmx-fa: use fastdiv in hmx-flash-attn * hmx-fa: precompute slope values to avoid disrupting the inner loop * hvx-utils/fa: new HVX helpers for powf and logf and using those to speed up FA alibi * hex-ops: fixed a bug in fusion logic that was messing up the order of the src tensors when some srcs are empty * hex-fa: correctly fallback to HVX if we have sinks or the dims are not quite right (cherry picked from commit 8f7f3bf)

…tions for latest models (ggml-org#23989) * hex-mm: initial support for F32 * F32 -> F32 matmuls * hex-rms-norm: fix src1 stride use in fused rms_norm_mul * hex-ops: clear spad pointers in the ops that clober it This fixes an odd case where fused rms-norm-mul was failing but only in qwen3.5-2B and only at searth op-bath sizes. * hmx-mm: add support for F32 * F32 -> F32 matmul_2d on HMX Decided to use Q4_0 * F32 -> F32 matmul for this. Q4_0 gets dequantized and tiled into F16, and here we quantize and tile F32 into F16. Super simple and pretty efficient. * hmx-mm: route f16 2D matmuls through the same kernel used for all other types * hmx-mm: re-introduce pipelined vs non-pipelined mode that we used to have but is much more generic way This update futher improves matmul performance and at the same time removes most of the redudant logic we had in different paths. * hmx-fa: slighlty improved pipeline simimar to matmul updates * hmx-mm: initial version of MAT_MUL_ID support for HMX * hmx-mm: fixed mxfp4 handling for MUL_MAT_ID * hex-gdn: optimize GATED_DELTA_NET DMA prefetch/double-buff, vectorize everything with HVX, in other words -- the usual :) * hmx-mm: missed one more case where we can use fastmod * hexagon: update DCVS settings for a slight perf bump * hmx-fa: use fastdiv in hmx-flash-attn * hmx-fa: precompute slope values to avoid disrupting the inner loop * hvx-utils/fa: new HVX helpers for powf and logf and using those to speed up FA alibi * hex-ops: fixed a bug in fusion logic that was messing up the order of the src tensors when some srcs are empty * hex-fa: correctly fallback to HVX if we have sinks or the dims are not quite right

max-krasnyansky added 10 commits June 1, 2026 10:17

hex-mm: initial support for F32 * F32 -> F32 matmuls

1aff0c6

hex-rms-norm: fix src1 stride use in fused rms_norm_mul

2179a1f

hex-ops: clear spad pointers in the ops that clober it

584fd87

This fixes an odd case where fused rms-norm-mul was failing but only in qwen3.5-2B and only at searth op-bath sizes.

hmx-mm: add support for F32 * F32 -> F32 matmul_2d on HMX

1b616fe

Decided to use Q4_0 * F32 -> F32 matmul for this. Q4_0 gets dequantized and tiled into F16, and here we quantize and tile F32 into F16. Super simple and pretty efficient.

hmx-mm: route f16 2D matmuls through the same kernel used for all oth…

ab6067d

…er types

hmx-mm: re-introduce pipelined vs non-pipelined mode that we used to …

67c3008

…have but is much more generic way This update futher improves matmul performance and at the same time removes most of the redudant logic we had in different paths.

hmx-fa: slighlty improved pipeline simimar to matmul updates

142810f

hmx-mm: initial version of MAT_MUL_ID support for HMX

6b3a438

hmx-mm: fixed mxfp4 handling for MUL_MAT_ID

ac5734e

hex-gdn: optimize GATED_DELTA_NET

8a10960

DMA prefetch/double-buff, vectorize everything with HVX, in other words -- the usual :)

github-actions Bot added ggml changes relating to the ggml tensor library for machine learning Hexagon labels Jun 1, 2026

hmx-mm: missed one more case where we can use fastmod

480eab1

max-krasnyansky marked this pull request as ready for review June 1, 2026 22:01

max-krasnyansky requested a review from a team as a code owner June 1, 2026 22:01

lhez approved these changes Jun 1, 2026

View reviewed changes

max-krasnyansky added 6 commits June 1, 2026 16:38

hexagon: update DCVS settings for a slight perf bump

73d7b65

hmx-fa: use fastdiv in hmx-flash-attn

ca7a0ac

hmx-fa: precompute slope values to avoid disrupting the inner loop

6348492

hvx-utils/fa: new HVX helpers for powf and logf and using those to sp…

c9d4497

…eed up FA alibi

hex-ops: fixed a bug in fusion logic that was messing up the order of…

636d357

… the src tensors when some srcs are empty

hex-fa: correctly fallback to HVX if we have sinks or the dims are no…

4188d3a

…t quite right

CISC approved these changes Jun 2, 2026

View reviewed changes

max-krasnyansky merged commit 8f7f3bf into ggml-org:master Jun 2, 2026
20 of 27 checks passed

Navigation Menu

Uh oh!

hexagon: MUL_MAT, MUL_MAT_ID, FLASH_ATTN and GDN cleanup and optimizations for latest models - #23989

hexagon: MUL_MAT, MUL_MAT_ID, FLASH_ATTN and GDN cleanup and optimizations for latest models #23989
max-krasnyansky merged 17 commits into
ggml-org:masterfrom
qualcomm:hexagon-f32-matmul

max-krasnyansky commented Jun 1, 2026 •

edited

Loading

Uh oh!

max-krasnyansky commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

max-krasnyansky commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

max-krasnyansky commented Jun 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

max-krasnyansky commented Jun 1, 2026 •

edited

Loading