EP-aware model building with EpCapabilities and modular architecture - #103
Merged
Conversation
Performance Comparison
|
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
🏗️ Architecture Diff
bert (feature-extraction) / model — 42 change(s)Op summary: 61 → 49 nodes --- base
+++ head
@@ -11,51 +11,39 @@
Add
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
Add
LayerNormalizationAdded nodes:
Removed nodes:
Connectivity changes:
Interface changes:
falcon / model — 41 change(s)Op summary: 66 → 53 nodes --- base
+++ head
@@ -16,51 +16,38 @@
Cast
Unsqueeze
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Connectivity changes:
Interface changes:
gemma2 / model — 53 change(s)Op summary: 107 → 92 nodes --- base
+++ head
@@ -32,76 +32,61 @@
Unsqueeze
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Div
Tanh
MulAdded nodes:
Removed nodes:
Connectivity changes:
Interface changes:
gpt2 / model — 40 change(s)Op summary: 53 → 40 nodes --- base
+++ head
@@ -9,45 +9,32 @@
Concat
Expand
LayerNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
Add
LayerNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
Add
LayerNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
llama / model — 46 change(s)Op summary: 61 → 46 nodes --- base
+++ head
@@ -9,53 +9,38 @@
Concat
Expand
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
llama (static-cache) / model — 46 change(s)Op summary: 58 → 43 nodes --- base
+++ head
@@ -2,57 +2,42 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
mamba (ssm-text-generation) / model — 43 change(s)Op summary: 105 → 94 nodes --- base
+++ head
@@ -1,7 +1,6 @@
Gather
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Constant
Split
Transpose
@@ -16,12 +15,10 @@
Sigmoid
Mul
Transpose
-Transpose
-MatMul
+FusedMatMul
Constant
Split
-Transpose
-MatMul
+FusedMatMul
Add
Softplus
Exp
@@ -49,12 +46,9 @@
Sigmoid
Mul
Mul
-Transpose
-MatMul
-Add
-RMSNormalization
-Transpose
-MatMul
+FusedMatMul
+SkipSimplifiedLayerNormalization
+FusedMatMul
Split
Transpose
Concat
@@ -65,11 +59,9 @@
Sigmoid
Mul
Transpose
-Transpose
-MatMul
+FusedMatMul
Split
-Transpose
-MatMul
+FusedMatMul
Add
Softplus
Exp
@@ -97,9 +89,6 @@
Sigmoid
Mul
Mul
-Transpose
-MatMul
-Add
-RMSNormalization
-Transpose
-MatMul
+FusedMatMul
+SkipSimplifiedLayerNormalization
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
phi3 / model — 46 change(s)Op summary: 61 → 46 nodes --- base
+++ head
@@ -9,53 +9,38 @@
Concat
Expand
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
phi3 (static-cache) / model — 46 change(s)Op summary: 58 → 43 nodes --- base
+++ head
@@ -2,57 +2,42 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
qwen / model — 46 change(s)Op summary: 61 → 46 nodes --- base
+++ head
@@ -9,53 +9,38 @@
Concat
Expand
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
qwen (static-cache) / model — 46 change(s)Op summary: 58 → 43 nodes --- base
+++ head
@@ -2,57 +2,42 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
qwen2 / model — 46 change(s)Op summary: 61 → 46 nodes --- base
+++ head
@@ -9,53 +9,38 @@
Concat
Expand
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
qwen2 (static-cache) / model — 46 change(s)Op summary: 58 → 43 nodes --- base
+++ head
@@ -2,57 +2,42 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
qwen2_moe / model — 136 change(s)Op summary: 224 → 183 nodes --- base
+++ head
@@ -16,20 +16,16 @@
Cast
Unsqueeze
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -39,30 +35,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -70,15 +60,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -86,15 +73,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -102,36 +86,28 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Sigmoid
Mul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -141,30 +117,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -172,15 +142,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -188,15 +155,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -204,21 +168,16 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Sigmoid
Mul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Interface changes:
qwen2_moe (static-cache) / model — 136 change(s)Op summary: 214 → 173 nodes --- base
+++ head
@@ -2,22 +2,18 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -27,30 +23,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -58,15 +48,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -74,15 +61,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -90,38 +74,30 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Sigmoid
Mul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
-Transpose
-MatMul
+FusedMatMul
Add
RotaryEmbedding
RotaryEmbedding
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -131,30 +107,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -162,15 +132,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -178,15 +145,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -194,21 +158,16 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Sigmoid
Mul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Interface changes:
qwen3 / model — 48 change(s)Op summary: 73 → 58 nodes --- base
+++ head
@@ -9,12 +9,9 @@
Concat
Expand
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -24,27 +21,20 @@
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -54,20 +44,15 @@
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Connectivity changes:
Interface changes:
qwen3 (static-cache) / model — 49 change(s)Op summary: 70 → 55 nodes --- base
+++ head
@@ -2,12 +2,9 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -19,27 +16,20 @@
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -51,20 +41,15 @@
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Connectivity changes:
Interface changes:
qwen3_5_moe (hybrid-text-generation) / model — 332 change(s)Op summary: 275 → 233 nodes --- base
+++ head
@@ -18,14 +18,10 @@
Add
RMSNormalization
Shape
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Transpose
Constant
CastLike
@@ -70,8 +66,7 @@
Constant
Concat
Reshape
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
@@ -80,95 +75,76 @@
Constant
TopK
Softmax
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Transpose
-MatMul
-Sigmoid
-Mul
-Add
-Add
-Add
-RMSNormalization
-Transpose
-MatMul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+FusedMatMul
+Sigmoid
+Mul
+Add
+Add
+Add
+RMSNormalization
+FusedMatMul
Reshape
Split
Reshape
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Reshape
Add
RMSNormalization
@@ -181,8 +157,7 @@
Attention
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
@@ -191,85 +166,68 @@
Constant
TopK
Softmax
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Transpose
-MatMul
-Sigmoid
-Mul
-Add
-Add
-Add
-RMSNormalization
-Transpose
-MatMul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+FusedMatMul
+Sigmoid
+Mul
+Add
+Add
+Add
+RMSNormalization
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
qwen3_5_text (hybrid-text-generation) / model — 56 change(s)Op summary: 129 → 113 nodes --- base
+++ head
@@ -18,14 +18,10 @@
Add
RMSNormalization
Shape
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Transpose
Constant
CastLike
@@ -70,32 +66,25 @@
Constant
Concat
Reshape
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Reshape
Split
Reshape
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Reshape
Add
RMSNormalization
@@ -108,22 +97,17 @@
Attention
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
qwen3_5_vl (hybrid-qwen-vl) / decoder — 56 change(s)Op summary: 149 → 133 nodes --- base
+++ head
@@ -38,14 +38,10 @@
Add
RMSNormalization
Shape
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Transpose
Constant
CastLike
@@ -90,32 +86,25 @@
Constant
Concat
Reshape
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Reshape
Split
Reshape
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Reshape
Add
RMSNormalization
@@ -128,22 +117,17 @@
Attention
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
qwen3_5_vl (hybrid-qwen-vl) / vision — 107 change(s)Op summary: 240 → 228 nodes --- base
+++ head
@@ -93,8 +93,7 @@
Constant
Pad
LayerNormalization
-Transpose
-MatMul
+FusedMatMul
Add
Split
Reshape
@@ -143,31 +142,25 @@
Unsqueeze
Attention
Squeeze
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-Add
-Reshape
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-LayerNormalization
-Transpose
-MatMul
+FusedMatMul
+Add
+Add
+LayerNormalization
+FusedMatMul
+Add
+Gelu
+FusedMatMul
+Add
+Add
+Reshape
+LayerNormalization
+FusedMatMul
+Add
+Gelu
+FusedMatMul
+Add
+LayerNormalization
+FusedMatMul
Add
Split
Reshape
@@ -216,25 +209,20 @@
Unsqueeze
Attention
Squeeze
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
-Add
-LayerNormalization
-Reshape
-Transpose
-MatMul
-Add
-Gelu
-Transpose
-MatMul
-Add
+FusedMatMul
+Add
+Add
+LayerNormalization
+FusedMatMul
+Add
+Gelu
+FusedMatMul
+Add
+Add
+LayerNormalization
+Reshape
+FusedMatMul
+Add
+Gelu
+FusedMatMul
+AddAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Interface changes:
qwen3_moe / model — 115 change(s)Op summary: 202 → 169 nodes --- base
+++ head
@@ -16,12 +16,9 @@
Cast
Unsqueeze
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -31,8 +28,7 @@
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -42,30 +38,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -73,15 +63,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -89,15 +76,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -107,12 +91,9 @@
Add
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -122,8 +103,7 @@
RotaryEmbedding
RotaryEmbedding
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -133,30 +113,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -164,15 +138,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -180,15 +151,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -198,5 +166,4 @@
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Interface changes:
qwen3_moe (static-cache) / model — 114 change(s)Op summary: 192 → 159 nodes --- base
+++ head
@@ -2,12 +2,9 @@
Gather
Gather
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -19,8 +16,7 @@
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -30,30 +26,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -61,15 +51,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -77,15 +64,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -95,12 +79,9 @@
Add
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Reshape
Reshape
RMSNormalization
@@ -112,8 +93,7 @@
TensorScatter
TensorScatter
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
Transpose
@@ -123,30 +103,24 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
Mul
ReduceSum
Mul
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -154,15 +128,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -170,15 +141,12 @@
ReduceSum
Mul
Add
-Transpose
-MatMul
+FusedMatMul
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Mul
-Transpose
-MatMul
+FusedMatMul
Constant
Equal
Cast
@@ -188,5 +156,4 @@
Add
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
Interface changes:
qwen3_next (hybrid-text-generation) / model — 698 change(s)Op summary: 585 → 501 nodes --- base
+++ head
@@ -18,14 +18,10 @@
Add
RMSNormalization
Shape
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Transpose
Constant
CastLike
@@ -70,8 +66,7 @@
Constant
Concat
Reshape
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
@@ -82,99 +77,79 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Transpose
-MatMul
-Sigmoid
-Mul
-Add
-Add
-Add
-RMSNormalization
-Shape
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+FusedMatMul
+Sigmoid
+Mul
+Add
+Add
+Add
+RMSNormalization
+Shape
+FusedMatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Transpose
Constant
CastLike
@@ -219,8 +194,7 @@
Constant
Concat
Reshape
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
@@ -231,99 +205,79 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Transpose
-MatMul
-Sigmoid
-Mul
-Add
-Add
-Add
-RMSNormalization
-Shape
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+FusedMatMul
+Sigmoid
+Mul
+Add
+Add
+Add
+RMSNormalization
+Shape
+FusedMatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Transpose
Constant
CastLike
@@ -368,8 +322,7 @@
Constant
Concat
Reshape
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
@@ -380,99 +333,80 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Transpose
-MatMul
-Sigmoid
-Mul
-Add
-Add
-Add
-RMSNormalization
-Transpose
-MatMul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+FusedMatMul
+Sigmoid
+Mul
+Add
+Add
+Add
+RMSNormalization
+FusedMatMul
Reshape
Split
Reshape
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Reshape
Add
RMSNormalization
@@ -485,8 +419,7 @@
Attention
Sigmoid
Mul
-Transpose
-MatMul
+FusedMatMul
Add
Add
RMSNormalization
@@ -497,89 +430,72 @@
TopK
ReduceSum
Div
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Constant
-Equal
-Cast
-Mul
-ReduceSum
-Unsqueeze
-Mul
-Add
-Transpose
-MatMul
-Sigmoid
-Mul
-Transpose
-MatMul
-Mul
-Transpose
-MatMul
-Transpose
-MatMul
-Sigmoid
-Mul
-Add
-Add
-Add
-RMSNormalization
-Transpose
-MatMul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+Constant
+Equal
+Cast
+Mul
+ReduceSum
+Unsqueeze
+Mul
+Add
+FusedMatMul
+Sigmoid
+Mul
+FusedMatMul
+Mul
+FusedMatMul
+FusedMatMul
+Sigmoid
+Mul
+Add
+Add
+Add
+RMSNormalization
+FusedMatMulAdded nodes:
Removed nodes:
Modified attributes:
Connectivity changes:
t5 (seq2seq) / decoder — 64 change(s)Op summary: 99 → 78 nodes --- base
+++ head
@@ -37,63 +37,42 @@
Transpose
Unsqueeze
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMulAdded nodes:
Removed nodes:
Interface changes:
t5 (seq2seq) / encoder — 37 change(s)Op summary: 75 → 63 nodes --- base
+++ head
@@ -37,39 +37,27 @@
Transpose
Unsqueeze
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
+FusedMatMul
Attention
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalization
-Transpose
-MatMul
+FusedMatMul
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
RMSNormalizationAdded nodes:
Removed nodes:
Interface changes:
whisper (speech-to-text) / decoder — 87 change(s)Op summary: 85 → 57 nodes --- base
+++ head
@@ -1,85 +1,57 @@
Gather
Mul
Gather
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMulAdded nodes:
Removed nodes:
Connectivity changes:
whisper (speech-to-text) / encoder — 53 change(s)Op summary: 55 → 38 nodes --- base
+++ head
@@ -3,53 +3,36 @@
Conv
Gelu
Transpose
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Mul
-Transpose
-MatMul
-Transpose
-MatMul
+FusedMatMul
+FusedMatMul
Add
Attention
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
-Transpose
-MatMul
+SkipLayerNormalization
+FusedMatMul
Add
Gelu
-Transpose
-MatMul
+FusedMatMul
Add
-Add
-LayerNormalization
+SkipLayerNormalizationAdded nodes:
Removed nodes:
Connectivity changes:
Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds execution-provider (EP) aware model optimization to the build()/build_from_module() pipeline, and extends the model zoo with new hybrid conv+attention (LFM2) and audio-to-audio (LFM2-Audio, Moshi/PersonaPlex) architectures plus EP-specific lowering rules and config generation utilities.
Changes:
- Introduces EP-aware 4-stage optimization pipeline (cleanup → fusion → lowering → fold) with
execution_provider+ optional tracing. - Adds new models/tasks for LFM2 (ShortConv+Attention) and audio-to-audio splits, plus associated configs, registry entries, docs, and test cases.
- Adds multiple EP-specific rewrite rules (DML/WebGPU/TRT-RTX) and supporting tests, plus genai_config + EP validation infrastructure.
Reviewed changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/synthetic_parity_test.py | Adds skip reasons for Moshi/PersonaPlex synthetic parity. |
| tests/model_coverage_test.py | Updates coverage mapping for new audio architecture. |
| tests/ep_optimization_test.py | New tests validating EP-aware optimization behavior and trace output. |
| tests/build_graph_test.py | Adds graph-build coverage for audio-to-audio model splits and hybrid conv cache outputs. |
| tests/_test_configs.py | Adds LFM2 tiny configs and audio-to-audio config section. |
| testdata/cases/causal-lm/lfm2-1.2b.yaml | Adds a golden-case descriptor for LFM2 (skipped in CI). |
| testdata/cases/audio/lfm2-audio-1.5b.yaml | Adds a golden-case descriptor for LFM2-Audio (skipped). |
| src/mobius/tasks/_causal_lm.py | Adds use_concrete_dims support via new _create_dims() helper. |
| src/mobius/tasks/_base.py | Extends hybrid-cache support to single-state conv layers; adds _create_dims() helper. |
| src/mobius/tasks/_audio_to_audio.py | New multi-model audio-to-audio task + MoshiTask specialization. |
| src/mobius/tasks/init.py | Exports/registers new tasks (audio-to-audio, moshi). |
| src/mobius/rewrite_rules/_unpack_qkv.py | New DML lowering rule to split packed QKV in GQA. |
| src/mobius/rewrite_rules/_unpack_qkv_test.py | Tests for unpack-qkv lowering rule. |
| src/mobius/rewrite_rules/_skip_norm_test.py | Adjusts tests for updated fusion behavior. |
| src/mobius/rewrite_rules/_skip_layer_norm_test.py | Adjusts tests for updated fusion behavior. |
| src/mobius/rewrite_rules/_separate_rope.py | New DML lowering rule to separate fused RoPE from GQA. |
| src/mobius/rewrite_rules/_separate_rope_test.py | Tests for separate-rope lowering rule. |
| src/mobius/rewrite_rules/_packed_attention_test.py | Updates expectations in packed-attention tests under EP-aware builds. |
| src/mobius/rewrite_rules/_eliminate_shape.py | New WebGPU lowering rule eliminating Shape-on-mask patterns. |
| src/mobius/rewrite_rules/_eliminate_shape_test.py | Tests for eliminate-shape rewrite + ORT equivalence. |
| src/mobius/rewrite_rules/_decompose_skip_layer_norm.py | New TRT-RTX lowering rules to decompose Skip(Layer |
| src/mobius/rewrite_rules/_decompose_skip_layer_norm_test.py | Tests for TRT-RTX skip-norm decomposition rules. |
| src/mobius/rewrite_rules/_decompose_layer_norm.py | New TRT-RTX lowering rule to decompose SimplifiedLayerNormalization. |
| src/mobius/rewrite_rules/_decompose_layer_norm_test.py | Tests for simplified-layer-norm decomposition rule. |
| src/mobius/rewrite_rules/_decompose_if.py | New graph pass to inline If and replace with Where for DML/WebGPU. |
| src/mobius/rewrite_rules/_decompose_if_test.py | Tests for If→Where pass behavior. |
| src/mobius/rewrite_rules/_cast_int64_to_int32.py | New WebGPU lowering rule casting Gather indices to INT32. |
| src/mobius/rewrite_rules/_cast_int64_to_int32_test.py | Tests for INT64→INT32 Gather-index casting rule + ORT equivalence. |
| src/mobius/rewrite_rules/init.py | Exposes newly added rewrite rules/passes. |
| src/mobius/models/lfm2.py | Adds LFM2 hybrid ShortConv+Attention causal LM model. |
| src/mobius/models/lfm2_audio.py | Adds LFM2-Audio composite model (audio encoder/embedding/decoder/audio decoder). |
| src/mobius/models/init.py | Exports new models (Lfm2*, Moshi). |
| src/mobius/components/_short_conv.py | Adds ShortConv component used by LFM2 conv layers. |
| src/mobius/components/_short_conv_test.py | Tests for ShortConv correctness and cache behavior. |
| src/mobius/components/init.py | Exposes ShortConv in public components API. |
| src/mobius/_registry.py | Registers new architectures and test model IDs/variants. |
| src/mobius/_genai_config.py | Adds EP-specific genai_config.json generation helpers. |
| src/mobius/_genai_config_test.py | Tests for genai_config helpers. |
| src/mobius/_ep_validation.py | Adds model/EP compatibility validation deny-list. |
| src/mobius/_ep_validation_test.py | Tests for EP validation behavior. |
| src/mobius/_configs.py | Adds Lfm2Config, Lfm2AudioConfig, MoshiConfig and transformers parsing. |
| src/mobius/_builder.py | Adds EP-aware optimization plumbing, capability matrices, tracing, and new build args. |
| examples/moshi_realtime.py | Adds Moshi/PersonaPlex realtime streaming example using split ONNX models. |
| docs/model-catalog.md | Updates model catalog counts and new categories. |
| docs/_generate_models.py | Updates model docs generation category descriptions/order. |
Phase 1 — EP-aware optimization pipeline: - _builder.py: add execution_provider/dtype parameters to build() and build_from_module(); 4-stage _optimize() pipeline (cleanup → fuse → lower → fold); _get_optimization_passes() with per-EP rule dispatch; concrete dims for WebGPU graph capture; trace_optimization mode - tasks/_base.py: use_concrete_dims support, model_role propagation - tasks/_causal_lm.py: thread execution_provider through to _optimize() - tests/ep_optimization_test.py: 23 pipeline tests (L1-L5 coverage) Phase 2 — Rewrite rules (graph-structural lowering): - _decompose_if.py: If → Where for DML/WebGPU - _decompose_skip_layer_norm.py: SkipLayerNorm → Add+LayerNorm (TRT-RTX) - _decompose_layer_norm.py: SimplifiedLayerNorm → primitives (TRT-RTX) - _separate_rope.py: decompose fused RoPE for DML - _unpack_qkv.py: split packed QKV for DML - _eliminate_shape.py: Shape → ReduceSum+ReduceMax for WebGPU - _cast_int64_to_int32.py: INT64 Gather indices → INT32 for WebGPU - All wired into _builder.py for their respective EPs Phase 3 — Validation and config: - _ep_validation.py: reject incompatible (model_type, ep) combos - _genai_config.py: generate EP-specific genai_config.json sections - docs/execution_providers.md: EP support documentation Design invariant (from issue #100 discussion): - ir.Function for single-op fusions with self-contained fallback bodies - Rewrite rules for global graph transforms (control flow, type, topology) - Components never know about EPs 2282 tests pass. 23 EP-specific tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchu/ep-aware-building-clean
branch
from
April 2, 2026 23:34
d26645c to
62230e7
Compare
genai-related code belongs in the integrations directory alongside the
existing GenaiConfigGenerator. Moved:
src/mobius/_genai_config.py → integrations/ort_genai/ep_config.py
src/mobius/_genai_config_test.py → integrations/ort_genai/ep_config_test.py
Updated:
integrations/ort_genai/__init__.py — exports make_provider_options,
make_sliding_window_config, make_kv_cache_dim_name, make_genai_decoder_config
43 tests pass (20 ep_config + 23 ep_optimization).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
marked this pull request as draft
April 2, 2026 23:39
…t-instructions Prohibits use of onnx.helper, onnx.TensorProto, onnx.ModelProto, and any other protobuf construction APIs in favour of onnxscript.ir APIs. Changes: - CONTRIBUTING.md: New '### Zero protobuf operations' section under 'Coding conventions' with good/bad code examples - .github/copilot-instructions.md: New '### Protobuf prohibition' section under 'Code style' with four bullet-point rules Rule applies to all code: models, components, tasks, rewrite rules, tests, fixtures, and utilities — no exceptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…cated) onnxscript.ir is a deprecated alias. Correct reference is onnx_ir. Updated both CONTRIBUTING.md and .github/copilot-instructions.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…on bodies
Custom ops with portable ONNX fallbacks should be defined as ir.Function
objects, not decomposed via pattern-matching rewrite rules. InlinePass
expands them selectively when the target EP cannot execute the custom op.
The new pipeline:
Stage 1: Cleanup
Stage 2: Fusion (GQA, SkipLayerNorm, BiasGelu) +
InlinePass: expand unsupported custom ops via function bodies
Stage 3: Graph-structural rewrites (If→Where, Shape, int64→int32)
Stage 4: Fold
Changes:
- Add src/mobius/_op_function_bodies.py: ir.Function bodies for
SkipLayerNormalization, SkipSimplifiedLayerNormalization, and
SimplifiedLayerNormalization using standard ONNX ops (LayerNorm,
RMSNormalization). register_function_bodies() adds them to model.functions.
- _builder.py/_optimize(): call register_function_bodies() then
InlinePass(criteria=_should_inline) after fusion; criteria checks
caps.supports_skip_layer_norm / caps.supports_simplified_layer_norm
- _builder.py/_get_optimization_passes(): remove DecomposeSkipLayerNorm
and DecomposeSimplifiedLayerNorm from lower_stages (handled by InlinePass)
- Remove _decompose_skip_layer_norm.py + test (4 files deleted)
- Remove _decompose_layer_norm.py + test (4 files deleted)
- Update rewrite_rules/__init__.py: remove deleted exports
Adding a new custom op with portable fallback:
1. Add a _build_<name>_function() to _op_function_bodies.py
2. Add it to _FUNCTION_BUILDERS
Done — no rewrite rule needed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
changed the title
Replace onnx.helper.make_function + onnx_ir.serde.deserialize_function with direct ir.Graph / ir.Node / ir.Function construction, following the same pattern as functions/linear_attention.py. Key fix: RMSNormalization at opset 23 has a single output Y (not two). Using num_outputs=2 caused RemoveUnusedNodesPass to raise IndexError when checking optional_info against node.outputs during rewrite cleanup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
xadupre
reviewed
Apr 3, 2026
xadupre
reviewed
Apr 3, 2026
_builder.py (914 lines) served three distinct concerns. Split into: - _execution_providers.py (~232 lines): EpCapabilities dataclass, EpRegistry class with dict-compatible API, ep_registry singleton, register_ep() / get_ep() helpers, all 6 built-in EP definitions in _register_builtins(). Standalone — depends only on onnx_ir. - _optimizations.py (~461 lines): SymbolicShapeInferencePass, CleanupMetadataPass, _DEFAULT_PASSES, trace infra (_count_ops, _TraceEntry, etc.), _get_optimization_passes(), optimize_model() (public alias: _optimize). Imports from _execution_providers + _flags. - _builder.py (~416 lines): build() + build_from_module() + resolve_dtype() + dtype helpers only. Re-exports EpCapabilities, _EP_REGISTRY, SymbolicShapeInferencePass, CleanupMetadataPass, _count_ops, _count_all_ops for zero-breakage backward compat. _EP_REGISTRY is now a shim pointing to ep_registry so all existing callers (.get(), frozenset(), sorted()) continue to work unchanged. Consumer updates: - _ep_validation.py: imports ep_registry from _execution_providers; KNOWN_EPS derived from ep_registry.names(); adds register_unsupported_combo() - integrations/ort_genai/ep_config.py: imports ep_registry from _execution_providers - tests/ep_optimization_test.py: trace caplog targets mobius._optimizations All 2326 tests pass (2274 build_graph + src, 52 EP tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
- _execution_providers.py: replace 'Adding EP #6'/'#7' with 'Adding a new EP = adding one entry here' (numbers go stale) - _optimizations.py: add __all__ listing public symbols - _builder.py: consolidate two __all__ blocks into one annotated list separating public API from backward-compat re-exports Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
changed the title
_builder.py now exports only what it defines: build, build_from_module, resolve_dtype, DTYPE_MAP. Moved imports to canonical locations: - tests/build_graph_test.py: SymbolicShapeInferencePass <- _optimizations - tests/ep_optimization_test.py: _count_ops <- _optimizations Removed from _builder.py: - EpCapabilities re-export (use _execution_providers) - _EP_REGISTRY shim (use ep_registry from _execution_providers) - CleanupMetadataPass, SymbolicShapeInferencePass re-exports (use _optimizations) - _count_ops, _count_all_ops re-exports (use _optimizations) - Backward-compat comment block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Mobius builds graphs declaratively and already uses op.Where for all conditional computations. The DecomposeIf pass was purely defensive with zero real-world matches across 84+ models. Removing to reduce complexity. - Delete _decompose_if.py and _decompose_if_test.py - Remove supports_if field from EpCapabilities (DML, WebGPU) - Remove DecomposeIf from _get_optimization_passes() in _optimizations.py - Remove from rewrite_rules/__init__.py exports - Update docs/execution_providers.md Agent-signed-off: Architect (e882a41f) [claude-opus-4.6] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
CI failures observed with intermediate transformers versions between 5.0 and 5.3.0: - mamba2_integration_test: Mamba2Cache import failure (API moved) - synthetic_parity_test jamba_1/bamba_1: 'has_previous_state can only be called on LinearAttention layers' (HybridMambaAttentionDynamicCache API changed) Both tests pass cleanly with transformers 5.3.0. Tighten the lower bound to exclude the broken intermediate versions. Also adds an inline version comment on the Mamba2Cache import so the requirement is self-documenting at the import site. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…oding 0 GLM4/ChatGLM use interleaved=1 RoPE layout (adjacent real/imag pairs). Hardcoding rotary_interleaved=0 in the GQA fusion would silently produce incorrect RoPE inside the fused GroupQueryAttention kernel. Fix: read the 'interleaved' attribute from the matched Q RotaryEmbedding node (attn.inputs[0].producer()) and pass it as rotary_interleaved. Also add two regression tests: - test_rotary_interleaved_attribute_propagated: verifies GLM4 (interleaved=True) gets rotary_interleaved=1 on the fused GQA node - test_rotary_interleaved_default_is_zero: verifies Qwen3 (half-split RoPE) still gets rotary_interleaved=0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
… PR #44950) In transformers 5.5.0, HF PR #44950 '[Cache] Native mamba & hybrid cache' refactored cache classes: - Mamba2Cache removed from transformers.models.mamba2.modeling_mamba2; replaced by DynamicCache(config=...). Add a try/except shim in mamba2_integration_test.py that selects the right constructor for old (<5.4) vs new (>=5.4) APIs. - DynamicCache.has_previous_state() now raises ValueError when called with no layer_idx on a cache with only Attention layers (no LinearAttention/Mamba layers). Hybrid models (jamba, bamba) call this via _update_mamba_mask(). Our all-attention test configs (jamba_1, bamba_1) trigger this path. Fix: pass use_cache=False to the HF forward call in synthetic_parity_test — logits are identical with or without caching for a single-pass comparison. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
When GQA is absent because the build dtype is not supported by the
EP's gqa_dtypes, the Differences section now emits a self-explanatory
message instead of just showing the raw count difference.
Example:
GQA (GroupQueryAttention): ort-genai/dml=16, mobius/dml=0
↳ DML supports GQA for float16 only; current build uses bfloat16.
Rebuild with --dtype float16 to enable GQA.
Changes:
- Add dtype: str | None field to OpCounts (defaults None for compat)
- build_mobius() resolves and stores the effective dtype — for EP
overrides from _MOBIUS_EP_DEFAULT_DTYPE, or via _detect_model_dtype()
when the EP uses the model's native dtype
- New _gqa_dtype_explanation() helper: checks mob_col.dtype against
EP's gqa_dtypes; returns a clear rebuild hint when mismatched
- _compute_verdict() uses the helper when GQA is 0 for mobius and >0
for ORT GenAI, replacing the generic OP_EXPLANATION with the
context-aware message
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Both console and markdown outputs now display the effective build
dtype alongside each column header and in a per-model metadata line.
Console:
┌─ Model: ... EP: cuda
│ Dtypes: ort-genai/cuda: unknown | mobius/cuda: fp16
│
────────────────────────────────────────────────────────
Op ort-genai/cuda mobius/cuda (fp16)
────────────────────────────────────────────────────────
Markdown:
**Build dtypes:** ort-genai/cuda: unknown | mobius/cuda: fp16 | ...
| Op | ort-genai/cuda | mobius/cuda (fp16) | Notes |
Changes:
- Add _DTYPE_SHORT map: 'float32' → 'fp32', 'float16' → 'fp16', 'bf16'
- Add _col_display_label(col): appends '(fp16)' suffix when dtype known
- Add _dtype_summary(cols): compact per-column dtype string for headers
- _console_table(): use display labels for column headers; widen col_w
to account for longer labels
- render_console(): add '│ Dtypes: ...' line per model section
- render_markdown(): add '**Build dtypes:**' line per model section;
use display labels in markdown table column headers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Both terminal and markdown now show color-coded differences: Terminal (ANSI backgrounds, respects --no-color / isatty): - Green (\033[42m): ORT GenAI has more (optimization opportunity) - Purple (\033[45m): mobius has more (potential regression) Markdown (HTML spans, always applied): - Green (#d4edda): ORT GenAI has more - Purple (#e8d5f5): mobius has more - Neutral: plain text, no span Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…adding_mask Root cause: create_attention_bias() and create_padding_mask() used Shape(input_ids, start=1, end=2) to extract the sequence length, but the EliminateShape WebGPU rule only matches Shape ops on attention_mask. The fix moves the extraction to attention_mask at the source. Changes: - _common.py create_attention_bias(): query_length now comes from Shape(attention_mask, start=1, end=2) instead of input_ids - _common.py create_padding_mask(): q_len now comes from Shape(attention_mask, start=1, end=2) instead of input_ids (batch_size dim 0 still uses input_ids as it is not on attention_mask) - Remove redirect_input_ids_seq_len_shape() workaround that was added in 2c5581f — no longer needed since the source is fixed - Remove the pre-processing call in _optimizations.py - Remove the export from rewrite_rules/__init__.py - Replace two redirect-specific tests with a simpler test_no_input_ids_seq_len_shape_at_source that verifies the source produces no Shape(input_ids, start=1, end=2) at all Result: WebGPU builds produce 0 Shape nodes without any post-hoc graph rewriting. All 2285 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
- Add src/mobius/functions/fused_matmul.py: ir.Function body for com.microsoft::FusedMatMul that decomposes to Transpose(B,[1,0]) + MatMul(A, B_t) * alpha as a portable fallback for EPs without the fused kernel. - Register FusedMatMul in functions/__init__.py _FUNCTION_BUILDERS. Remove the module-level _cache: register_function_bodies() and get_function() now always create fresh ir.Function objects so that rewrite passes (which apply to function bodies in-place) cannot corrupt shared state across models or parallel test workers. - Add 'com.microsoft': 1 to graph opset_imports in tasks/_base.py so the ONNX checker accepts com.microsoft domain nodes. - Update _lora.py: LoRA adapter projections emit FusedMatMul(transB=1) instead of Transpose + MatMul for consistency with Linear. - Update _fused_matmul.py docstring: TransposeMatMulToFusedMatMul is now for external models only. Warn against applying it to models with FusedMatMul function bodies (self-loop risk). - Redesign _fused_matmul_test.py: tests now verify Linear emits FusedMatMul directly, fused_matmul_rules() works on inlined models, and ORT runs the FusedMatMul model. - Update component tests (_common_test, _components_test, _diffusion_test, _lora_test, _mamba_block_test, _mlp_test): replace MatMul count assertions with FusedMatMul counts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 56 out of 56 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (1)
src/mobius/components/_common.py:201
- create_attention_bias() now derives query_length from attention_mask, but in dynamic-cache decoding attention_mask has shape (batch, past+query) while input_ids/hidden_states have shape (batch, query). This makes query_length==total_length and produces an attention bias with the wrong query dimension (and Slice start always 0), which will break decode-time Attention shapes/semantics. Derive query_length from input_ids/hidden_states (dim 1) and keep total_length from attention_mask.
# Get query_length and total_length from shapes.
# Both come from attention_mask: total_length is the full sequence length, and
# query_length is derived as total_length - past_length. Using attention_mask
# for both means the EliminateShape rule on WebGPU can eliminate these Shape ops.
query_length = op.Shape(attention_mask, start=1, end=2) # 1-D [1]
total_length = op.Shape(attention_mask, start=1, end=2) # 1-D [1]
start = op.Sub(total_length, query_length)
# q_indices_2d: (batch_size, query_length)
q_indices_2d = op.Slice(all_indices, start, total_length, [1])
Lint (Task 1): - _build_context_test.py: flatten nested with to single with statement (SIM117) - _eliminate_shape_test.py: add blank line after docstring summary (D205) - examples/model_builder_comparison.py: parenthesize implicit string concatenations in _OP_CATALOG tuple literals (ISC004) - _build_context_test.py, _optimizations.py: apply ruff format patches PR #103 review comments (Task 2): - _unpack_qkv.py: remove unused module-level _counter = 0 (bot: unused global) - ep_config.py: fix make_kv_cache_dim_name() docstring — actual output is 'past_sliding_length' (sequence→sliding), not 'sliding_window_length' (Copilot: docstring/implementation mismatch) - _builder.py: add audio_encoder and speech to _MODEL_ROLE_MAP as 'encoder' role so GQA fusion is not incorrectly applied to audio sub-models (Copilot: missing audio keys default to 'decoder') - _model_package.py: move inline 'from mobius._optimizations import fold_constants_after_weights' to top-level import (justinchuby: move imports to top) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
PackQKV emits Concat(w_q, w_k, w_v) nodes that must be folded after apply_weights() loads the actual weight tensors. The pre-existing input_size_limit=8192 caused the fold pass to silently skip these nodes because individual weight tensors exceed 8192 elements (Llama-3.2-1B q_proj is 4M elements). Raise input_size_limit to _FOLD_OUTPUT_SIZE_LIMIT (256M) in fold_constants_after_weights() so packed-QKV Concat nodes are correctly evaluated and replaced with a single fused initializer. Verified: Llama-3.2-1B cuda/float16 Concat count drops 16 → 0 after apply_weights(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
… SeparateRoPE SeparateRoPE (Stage 3 lowering) creates duplicate Gather(cos_cache, position_ids) and Gather(sin_cache, position_ids) nodes — 2 per layer. Without CSE in Stage 4, DML builds accumulate 32+ extra Gather nodes (34 total vs 2 expected for 16-layer models). Adding CommonSubexpressionEliminationPass before FoldConstantsPass in the Stage 4 PassManager collapses these to 2 Gathers after lowering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
marked this pull request as ready for review
April 7, 2026 18:12
…k guard, perm validation - Bug 1 (_common.py): create_padding_mask() was reading q_len from attention_mask.dim1 (total_length) instead of input_ids.dim1. During dynamic-cache decode, q_len=1 but total_len>1, so the old code produced a mask with the wrong query dimension, violating ORT Attention's shape contract. Now reads q_len from input_ids and total_len from attention_mask. - Bug 2 & 3 (_qwen3_vl_vision.py, _qwen25_vl_vision.py): Changed domain="com.microsoft" to _domain="com.microsoft" in op.PackedMultiHeadAttention calls. The leading underscore is the onnxscript convention for setting the node's op domain; without it 'domain' is treated as a regular node attribute. - Bug 4 (_group_query_attention.py): AttentionToGQA.rewrite() would crash with a TypeError if attention_mask is not a graph input. Added an early return when attention_mask is None. - Bug 5 (_group_query_attention.py): _get_underlying_weight() accepted any Transpose node without checking its perm attribute. A Transpose with a non-standard perm would produce an incorrectly packed QKV weight. Now validates perm == [1, 0] and raises MatchFailureError otherwise. Test update: test_no_input_ids_seq_len_shape_at_source was asserting the old (buggy) behavior. Updated to verify that Shape(input_ids, 1) correctly appears for q_len and Shape(attention_mask, 1) appears for total_len. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchu/ep-aware-building-clean
branch
from
April 7, 2026 18:40
364e7a8 to
fc5c4b1
Compare
…efault EP docs, BiasGelu stage name, stale comments - Rename EpCapabilities.packed_attn_dtypes → qkv_pack_dtypes: the field gates PackQKV fusion for GQA, not PackedAttention. Update all references in _optimizations.py, _build_context_test.py, _group_query_attention.py, docs/execution_providers.md. - Fix DML qkv_pack_dtypes: set to frozenset() since supports_fused_rope=False triggers UnpackQKV which immediately undoes any packing. Avoids a wasteful pack→unpack round-trip. - Correct default EP documentation: the default EP applies standard fusions (SkipNorm, FusedMatMul, Gelu) but no EP-specific ops (no GQA, no PackQKV). The previous claim of 'no com.microsoft ops / only cleanup + constant folding' was incorrect. Update _execution_providers.py comment, _builder.py docstring, docs/execution_providers.md, and tests/ep_optimization_test.py docstring. - Rename optimization stage 'BiasGelu' → 'GeluFusion': the stage runs gelu_fusion_rules() which fuses to standard ONNX Gelu, not BiasGelu. Update stage name in _optimizations.py and both docstrings. - Fix rewrite_rules/__init__.py docstring: rules are applied automatically by optimize_model() for the relevant EP, not 'not applied by default'. - Remove stale PascalCase alias sentence from functions/__init__.py docstring: CausalConvWithState alias was removed. - Remove unreachable return None in examples/model_builder_comparison.py:1199. Agent-signed-off: Developer (a86fcdac) [claude-opus-4.6] Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…efault EP docs, BiasGelu stage name, stale comments - Rename EpCapabilities.packed_attn_dtypes → qkv_pack_dtypes: the field gates PackQKV fusion for GQA, not PackedAttention. Update all references in _optimizations.py, _build_context_test.py, _group_query_attention.py, docs/execution_providers.md. - Fix DML qkv_pack_dtypes: set to frozenset() since supports_fused_rope=False triggers UnpackQKV which immediately undoes any packing. Avoids a wasteful pack→unpack round-trip. - Correct default EP documentation: the default EP applies standard fusions (SkipNorm, FusedMatMul, Gelu) but no EP-specific ops (no GQA, no PackQKV). The previous claim of 'no com.microsoft ops / only cleanup + constant folding' was incorrect. Update _execution_providers.py comment, _builder.py docstring, docs/execution_providers.md, and tests/ep_optimization_test.py docstring. - Rename optimization stage 'BiasGelu' → 'GeluFusion': the stage runs gelu_fusion_rules() which fuses to standard ONNX Gelu, not BiasGelu. Update stage name in _optimizations.py and both docstrings. - Fix rewrite_rules/__init__.py docstring: rules are applied automatically by optimize_model() for the relevant EP, not 'not applied by default'. - Remove stale PascalCase alias sentence from functions/__init__.py docstring: CausalConvWithState alias was removed. - Remove unreachable return None in examples/model_builder_comparison.py:1199. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Update 4 remaining references where 'BiasGelu' was used as a stage/label name (not the com.microsoft::BiasGelu op name, which is unchanged): - docs/execution_providers.md: pipeline diagram, sample trace output (×2), and 'no matches' example context - tests/ep_optimization_test.py: trace test comments and assertion message Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…k(), add DML PackQKV test - EpCapabilities.__post_init__: enforce that qkv_pack_dtypes must be frozenset() when supports_fused_rope=False. UnpackQKV always fires for such EPs (e.g. DML), so packing would be immediately undone. The invariant is now validated at construction time with a clear error. - AttentionToGQA.check(): move the attention_mask graph-input guard from rewrite() to check(). Per onnxscript rewriter semantics, early-exit conditions belong in check() so rewrite() only runs when the match is valid and no graph modification will be silently skipped. - test_dml_lowers_rope_and_qkv: add assertion that PackQKV count == 0 for DML, verifying that qkv_pack_dtypes=frozenset() correctly prevents QKV packing on that EP. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Fix 1: Update 6 rewrite rule module docstrings that still said 'not applied by default' after the __init__.py was updated in a603837. Modules in the optimize_model() pipeline now correctly describe when they are applied: - _gelu_fusion.py: applied for all EPs (GeluFusion stage) - _skip_norm.py: applied when supports_skip_layer_norm=True (all except TRT-RTX) - _skip_layer_norm.py: same condition as _skip_norm.py - _group_query_attention.py: applied for decoder + matching gqa_dtypes/qkv_pack_dtypes - _separate_rope.py: applied when supports_fused_rope=False (DML) - _unpack_qkv.py: same condition as _separate_rope.py Modules legitimately not in the pipeline (_bias_gelu.py, _fused_matmul.py, _layer_norm_fusion.py, _packed_attention.py) are left unchanged. Fix 2: Correct docs/execution_providers.md line 89 which claimed GeluFusion produces com.microsoft::BiasGelu. It actually fuses decomposed GELU chains into standard ONNX Gelu (opset 20). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
…as() Commit fd85bdf changed query_length to use Shape(attention_mask, start=1, end=2) so that the EliminateShape WebGPU rule would fire on it. But attention_mask always has shape (batch, total_seq_len), so: query_length = Shape(attention_mask, 1) → total_seq_len total_length = Shape(attention_mask, 1) → total_seq_len start = Sub(total_length, query_length) → 0 start=0 means Slice selects ALL positions as queries. During decode, input_ids is (batch, 1), so query_length should be 1 and start should be total_seq_len - 1, slicing only the last row. With start=0 the returned bias has shape (batch, 1, total_seq_len, total_seq_len) instead of the correct (batch, 1, 1, total_seq_len), causing completely wrong attention scores and garbage model output. Fix: revert query_length to Shape(input_ids, start=1, end=2). - total_length stays on attention_mask (EliminateShape still fires for it). - On WebGPU (concrete dims), Shape(input_ids, 1) is constant-folded away, so no dynamic Shape op survives in the WebGPU export. Add test: test_query_length_from_input_ids_not_attention_mask verifies that a Shape node reading from input_ids is present in the graph, guarding against future regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Apr 8, 2026
- Create docs/ep_quickstart.md: 5-section practical guide (build for a
specific EP, trace mode, query ep_registry, register a custom EP,
common EP configurations table)
- Expand docs/execution_providers.md:
- New 'When to use build_context() vs optimize_model() directly'
section with summary table of API entry points
- Clarify Step 2 of 'Adding a New EP': both rewrite-rule and
generation-time (ep_capabilities()) approaches documented
- README.md: Add EP-aware optimization paragraph with code examples
and links to the quickstart and full reference docs
All new content is derived from reading the source code (PR #103,
now merged). No speculative or inaccurate information added.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
added a commit
that referenced
this pull request
Apr 8, 2026
Documents the EP-aware model building infrastructure added in PR #103. ## Changes ### New: `docs/ep_quickstart.md` Practical 5-section guide for the most common EP tasks: 1. Build for a specific EP (code + CLI examples) 2. See what each EP does (trace mode walkthrough) 3. Query which EPs are available (`ep_registry`, `get_ep()`) 4. Register a custom EP (`register_ep()`) 5. Common EP configurations at a glance (table) ### Updated: `docs/execution_providers.md` - New **"When to use `build_context()` vs `optimize_model()` directly"** section with a summary table of all API entry points (`build()`, `build_from_module()`, `optimize_model()`, `build_context()`) - Clarified Step 2 of "Adding a New EP": documents both the rewrite-rule approach (for post-build graph rewrites) and the generation-time `ep_capabilities()` approach (for component-level branching) ### Updated: `README.md` - Added EP-aware optimization paragraph with CUDA/WebGPU code examples - Links to the new quickstart and full reference docs ## Accuracy note All content is derived directly from reading the source code. Docstrings for `EpCapabilities`, `EpRegistry`, `build_context()`, `optimize_model()`, and `build()` were verified and are already comprehensive — no docstring changes needed. --------- Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Execution Provider aware model building with EpCapabilities dataclass, modular architecture, and comprehensive documentation.
Architecture
_execution_providers.py: EpCapabilities frozen dataclass + EpRegistry with 6 built-in EPs +register_ep()for out-of-tree extensions_optimizations.py: 4-stage optimization pipeline (cleanup → fusion → lowering → fold) with trace diagnostics_builder.py: Slimmed build API (build(),build_from_module())Documentation
docs/execution_providers.md: 373-line reference guideTesting
Closes #100. References #99.