Add QNN (Qualcomm Hexagon HTP) execution-provider profile - #370
Merged
Conversation
kunal-vaishnavi
approved these changes
Jun 22, 2026
kunal-vaishnavi
enabled auto-merge (squash)
June 22, 2026 18:23
Performance Comparison
|
shreyshah-microsoft
force-pushed
the
add-qnn-ep-profile
branch
from
June 22, 2026 18:24
0f275ea to
11719ae
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Registers a new qnn execution-provider profile in Mobius so builds can target Qualcomm Hexagon HTP via the ONNX Runtime QNN EP, and wires the ORT-GenAI integration to dispatch the provider name correctly.
Changes:
- Added a
qnnEpCapabilitiesentry (disabling fused/contrib ops so graphs lower to standard ONNX primitives). - Added ORT-GenAI provider-name mapping for
qnn→QNN.
Show a summary per file
| File | Description |
|---|---|
src/mobius/_execution_providers.py |
Registers the new qnn EP capability profile and its default provider options. |
src/mobius/integrations/ort_genai/ep_config.py |
Maps qnn to ORT-GenAI’s provider identifier (QNN) for provider_options emission. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 2
Registers an EpCapabilities entry for the "qnn" execution provider so the ORT-GenAI builder can target the Qualcomm Hexagon NPU (HTP backend, via the onnxruntime-qnn QAIRT plugin), and maps "qnn" -> "QNN" in the ORT-GenAI provider-name table. The HTP runs a statically-shaped, QDQ-quantized graph compiled to a QNN context binary and has no kernels for ORT contrib fused ops, so the profile forces decomposition to the standard ONNX primitives the QNN op builders can lower: no fused RoPE, no Skip[Simplified]LayerNorm, no PackedMHA, and no GroupQueryAttention (empty gqa_dtypes) -- the gemma4 multimodal decoder forgoes GQA anyway (bidirectional-vision overlay). past_present_share_buffer is disabled since standard-Attention KV concat cannot alias a shared buffer; fixed prefill-bucket / fixed-max-KV static shaping is applied as a separate downstream pass before context-binary generation. Validated by building gemma-4-12B (gemma4_unified) with execution_provider="qnn": the graph emits pure ai.onnx (zero com.microsoft contrib ops).
shreyshah-microsoft
force-pushed
the
add-qnn-ep-profile
branch
from
June 22, 2026 18:26
11719ae to
3e46033
Compare
justinchuby
approved these changes
Jun 22, 2026
kunal-vaishnavi
pushed a commit
that referenced
this pull request
Jun 25, 2026
## What A rewrite rule that reshapes rank-4 `RMSNormalization` (query/key norm) to rank-3 for the QNN HTP, which miscomputes RMSNormalization over the last axis of a rank-4 tensor. ## Why Gemma-4 (and Qwen3) apply RMSNormalization over the head dimension after reshaping q/k to `(batch, seq, heads, head_dim)`, so the norm runs on a rank-4 tensor. On the Hexagon HTP this computes incorrectly -- the graph finalizes but the q/k-norm output is numerically wrong (uncorrelated with CPU); rank-3 RMSNormalization runs correctly. Pairs with the QNN EP profile (#370): that targets the HTP, this makes the q/k-norm graph compute correctly there. ## How `ReshapeRank4RMSNorm` reshapes `(B, S, H, Dh)` -> `(B, S*H, Dh)`, applies the identical RMSNormalization over the last axis, and reshapes back. Both reshapes use constant shapes (`[0, -1, head_dim]`, `[0, -1, heads, head_dim]`) so no `Shape` op is introduced and the graph stays static for the HTP. Numerically exact -- the normalized axis is unchanged. Gated on a new `supports_rank4_rmsnorm` capability (default `True`; `False` only for `qnn`), applied in the lowering stage alongside SeparateRoPE. ## Test `_reshape_rmsnorm_test.py`: the rule reshapes a rank-4 RMSNorm to rank-3 (2 Reshapes added) and the rewritten graph matches the original on CPU (where rank-4 RMSNorm is correct); a rank-3 RMSNorm is left untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Registers a
qnnexecution-provider profile so the ORT-GenAI builder can target the Qualcomm Hexagon NPU (HTP backend, via the onnxruntime-qnn QAIRT plugin):_execution_providers.py: newEpCapabilities(name="qnn", ...).integrations/ort_genai/ep_config.py: mapqnn→QNNfor ORT-GenAI provider dispatch.Why
Bringing up gemma-4-12B (
gemma4_unified) on Qualcomm Copilot+ devices (Snapdragon X Elite / X2 Elite). The HTP runs a statically-shaped, QDQ-quantized graph compiled to a QNN context binary and has no kernels for ORT contrib fused ops, so the build must emit standard ONNX primitives the QNN op builders can lower.How
The profile reuses the existing rewrite infrastructure rather than adding new passes:
supports_fused_rope=False→SeparateRoPE/UnpackQKVsupports_skip_layer_norm=False→ skip-norm inlined to standard opssupports_packed_multi_head_attention=False→ PackedMHA expandedgqa_dtypes=frozenset()→ no GroupQueryAttention (the gemma-4 multimodal decoder forgoes GQA anyway — bidirectional-vision overlay); standard Attention is emitted and static-shaped downstreamsupports_past_present_share_buffer=False(standard-Attention KV concat can't alias a shared buffer)provider_options= onnxruntime-qnn HTP launch defaults;soc_model+ EP-context binary path are device-specific, set at build/validation time.Validation
Building gemma-4-12B with
execution_provider="qnn"emits pure ai.onnx (zerocom.microsoftcontrib ops).