gemma4 MoE: fix com.microsoft::MoE input slots + activation_type (#324)

## Problem

Per [upstream report on
microsoft/onnxruntime-genai#2062](microsoft/onnxruntime-genai#2062 (comment)),
the `justinchuby/gemma-4-26b-a4b-it-onnx` export does not load under
public ORT 1.25+:

```
MoE node has input size 4 not in range [min=5, max=8]
```

After patching the schema (inserting `fc1_experts_bias`), the next
failure is a weight shape mismatch:

```
Input 'fc1_experts_weights' is expected to have shape {128,704,2816}, got {128,1408,2816}
```

## Root cause

`Gemma4MoeBlock.forward` was emitting `com.microsoft::MoE` with:
- only 4 positional inputs (missing the optional `fc1_experts_bias` slot
at index 3, so `fc2_experts_weights` was in the wrong position); and
- `activation_type="silu"` even though `fc1_experts_weights` is laid out
as `[E, 2*inter, hidden]` (gate + up concatenated).

A surface fix (add `None` for slot 3, change `activation_type="swiglu"`)
gets past those two errors but uncovers a deeper incompatibility: the
ORT MoE kernel for `swiglu` is hardcoded for **GPT-OSS-style SwiGLU** —
not standard Gemma 4 SwiGLU. Full evidence + repro in
**microsoft/onnxruntime#28738**.

So there's currently no ORT-public SwiGLU MoE mode compatible with Gemma
4's standard SwiGLU semantics. Using the fused op silently produces
wrong outputs even when it loads.

## Fix

Always take the static-unroll fallback path (`_dispatch_moe_fallback`)
for Gemma 4 — drop the `ep_capabilities().supports_fused_moe` branch
entirely. The fallback emits standard ONNX ops (`Gather`, `MatMul`,
`Mul`, etc.) and computes the math correctly. Detailed comment in source
explains why so it doesn't get "re-optimized" back to the fused op
without an upstream kernel fix (microsoft/onnxruntime#28738).

Also fixes a latent bug in the fallback itself: `op.TopK(...)` was
unpacked without `_outputs=2`, so the path would have crashed at build
time the first time it was exercised.

## Verification

1. **Unit + L1 tests**: `tests/build_graph_test.py -k gemma4` → 15
passed.
2. **End-to-end ORT load**: rebuilt `google/gemma-4-26b-a4b-it` (`mobius
build ... --dtype f16`), the decoder (~51GB ONNX) loads cleanly under
public `onnxruntime==1.27.0`:

   ```
   ORT 1.27.0
   load OK in 959.6s (CPUExecutionProvider, ORT_DISABLE_ALL)
inputs (63): ['inputs_embeds', 'attention_mask', 'position_ids',
'past_key_values.0.key', 'past_key_values.0.value'] ...
   ```

(The 16-min load time is dominated by the 7680 expert MatMul nodes from
the fallback path — see "Followups" below.)

3. **HF Hub re-upload**: pushed the fixed fp16 build to
[`justinchuby/gemma-4-26b-a4b-it-onnx` under
`f16/default/`](https://huggingface.co/justinchuby/gemma-4-26b-a4b-it-onnx/tree/main/f16/default).
Users on the upstream issue can pull this and run with public ORT 1.27.

## Out of scope / followups

These are real but separate from the schema fix:

- **Native Gemma 4 SwiGLU kernel in ORT** — tracked upstream at
microsoft/onnxruntime#28738. Once ORT's MoE kernel supports standard
SwiGLU (concatenated layout, alpha=1.0, no clipping), mobius can
re-enable the fused path and get a fast quantizable form.
- **`OnnxKQuantQuantization` skips expert weights** — tracked at
microsoft/Olive#2489. The fallback's per-expert weights are accessed via
`op.Gather(stacked_weights, expert_idx)` (dynamic, not static
initializers), so the k-quant pass leaves them at fp16. Result: the
Q4_K_M build is only ~6% smaller than fp16 (47GB vs 51GB). The Olive
issue proposes two fix paths (pattern-aware kquant or a pre-pass that
unstacks per-expert weights).

## Related

- microsoft/onnxruntime-genai#2062 (upstream report from @tmtiwari)
- microsoft/onnxruntime#28738 (ORT MoE SwiGLU kernel hardcoded for
GPT-OSS, blocks standard SwiGLU)
- huggingface.co/justinchuby/gemma-4-26b-a4b-it-onnx (re-uploaded fp16)

Signed-off-by: justinchuby <11205048+justinchuby@users.noreply.github.com>
Co-authored-by: justinchuby <11205048+justinchuby@users.noreply.github.com>

Author: justinchuby

src/mobius/models/gemma4.py

@@ -1175,28 +1175,26 @@ def forward(
             # Norm residual before experts
             normed_flat = self.pre_feedforward_layernorm_2(op, residual_flat)  # [B*S, H]
 
-            caps = ep_capabilities()
-            if caps.supports_fused_moe:
-                # Fused MoE op: handles top-k selection + expert dispatch internally.
-                # NOTE: per_expert_scale is NOT applied in fused path (ORT op limitation).
-                # CastLike restores dtype after op.MoE (custom op, type=None on output)
-                # so downstream ops can correctly infer types and share scalar initializers.
-                # Using CastLike(target=normed_flat) preserves bf16/fp16/fp32 from the input.
-                moe_out_flat = op.CastLike(
-                    op.MoE(  # type: ignore[attr-defined]
-                        normed_flat,
-                        router_probs,
-                        self.fc1_experts_weights,
-                        self.fc2_experts_weights,
-                        activation_type="silu",
-                        k=self._top_k,
-                        normalize_routing_weights=1,
-                        _domain="com.microsoft",
-                    ),
-                    normed_flat,  # match input dtype (bf16/fp16/fp32)
-                )  # [B*S, H]
-            else:
-                moe_out_flat = self._dispatch_moe_fallback(op, normed_flat, router_probs)
+            # NOTE: We intentionally do NOT use the fused com.microsoft::MoE
+            # op here, even when ``ep_capabilities().supports_fused_moe`` is
+            # True. The ORT MoE kernel for ``activation_type="swiglu"`` is
+            # hardcoded for GPT-OSS-style SwiGLU:
+            #   * CUDA kernel (ft_moe/moe_kernel.cu) hardcodes
+            #     ``alpha=1.702, limit=7.0`` and an interleaved gate/up
+            #     layout — neither matches Gemma 4's standard SwiGLU
+            #     (alpha=1.0, no limit, concatenated layout).
+            #   * CPU kernel (moe_cpu.cc) refuses to load unless
+            #     ``swiglu_fusion=1`` (interleaved).
+            # ``activation_type="silu"`` would also be wrong because
+            # ``fc1_experts_weights`` packs gate+up along dim 1
+            # ([E, 2*inter, hidden]), causing a shape mismatch against
+            # the kernel's expected ``[E, inter, hidden]`` for silu.
+            #
+            # Until ORT exposes a Gemma-4-compatible SwiGLU mode (standard
+            # alpha=1.0, concatenated layout) we always take the static
+            # unroll fallback. See microsoft/onnxruntime-genai#2062 for
+            # the upstream report.
+            moe_out_flat = self._dispatch_moe_fallback(op, normed_flat, router_probs)
 
             moe_out = op.Reshape(moe_out_flat, op.Shape(residual))  # [B, S, H]
             moe_out = self.post_feedforward_layernorm_2(op, moe_out)
@@ -1248,7 +1246,7 @@ def _dispatch_moe_fallback(
         """
         # Top-K selection: top_weights/top_indices both [T, K]
         top_weights_raw, top_indices = op.TopK(
-            router_probs, op.Constant(value_ints=[self._top_k]), axis=-1
+            router_probs, op.Constant(value_ints=[self._top_k]), axis=-1, _outputs=2
         )
         # Arithmetic normalisation: weights sum to 1 (matches HF: top_k_weights /= top_k_weights.sum(-1, keepdim=True))
         top_weights = op.Div(