Running your own models on a phone's NPU — without the vendor SDK

A from-scratch INT8 inference path for the MediaTek MDLA 3.5 (APU 650) on a Nothing Phone 2a, built by driving the on-device Neuron compiler directly — bypassing the registration-walled NeuroPilot/Genio SDK. Proven by running stock ImageNet CNNs and a lip-reading model end-to-end on the NPU, hardware-witnessed.

Scope, stated up front and honestly: this is edge-NPU bring-up and a young but genuinely general INT8 CNN compiler — the same model-agnostic pipeline compiled four architecturally-distinct networks (MobileNetV2, ResNet-18, EfficientNet-B0, SqueezeNet) with no per-model code, extended only by adding op-legalizations. It is not a general-purpose compiler: op coverage is still growing, it's packaged as scripts not one tool, and the hardware bounds it to feed-forward CNNs — transformers are out (measured — see Limitations). The models (MobileNetV2, ResNet-18, LipNet) are stock and not novel; the contribution is the path to the silicon and the lowering rules learned by measuring real hardware.

---

TL;DR

• The wall: to compile a model for the MT6886's MDLA you normally need MediaTek's gated Genio/NeuroPilot SDK (ncc-tflite). Registration-walled, never obtained → dead end.

• The bypass: the phone ships an on-device Neuron compiler (/vendor/lib64/mt6886/libneuron_adapter_mgvi.so). Build the model with the Neuron Adapter API on the phone, force it onto the MDLA, execute. No SDK, no host compiler, no .dla file.

• Proven real, not CPU-fallback: ops are force-placed with NeuronCompilation_createForDevices({mtk-mdla}) (errors instead of falling back), op-support is gated per-device, and execution is witnessed independently via the APU's runtime-PM active_time counter (idle 0 ms vs inference +154 ms) — because the normal MET/ftrace trace path is absent on retail firmware.

• A small engine on top: an automated import → INT8 quantize → legalize → run on MDLA pipeline. Four stock, architecturally-distinct ImageNet CNNs run through one pipeline, every op forced onto the MDLA, extended only by op-legalizations (no per-model code):

  Model	Ops on MDLA	Latency	Top-1 vs FP32 (12 imgs)
  MobileNetV2 (depthwise, ReLU6)	83/83	1.6 ms/img	10/12
  ResNet-18 (plain conv, MaxPool, ReLU)	50/50	2.8 ms/img	11/12
  EfficientNet-B0 (squeeze-excite, SiLU, broadcast-mul)	256/256	3.9 ms/img	11/12
  SqueezeNet1.1 (fire modules, CONCAT)	47/47	2.3 ms/img	10/12

  (Numbers post-proofread: the int8 validator now honestly quantizes the lowered classifier and uses unbiased calibration sampling — earlier figures were ~1 image optimistic on the small 12-image set.)

  EfficientNet forced new legalizations — SiLU→LOGISTIC·MUL, sigmoid→LOGISTIC, squeeze-excite→broadcast MUL — all of which mapped on the silicon. Its accuracy was initially poor (3/12) under naive max-abs quantization; switching to percentile-clipped calibration (one general change) lifted it to 11/12 and also exposed a latent MaxPool scale bug, found by op-prefix bisection. The per-model accuracy spread was calibration, not architecture.

• The application: a lip-reading model (LipNet) reads a silent mouth video and prints the sentence, with the spatiotemporal conv front-end running on the NPU (the 3D convs decomposed into 2D convs the MDLA can map).

---

Background — the wall

The MDLA (MediaTek Deep Learning Accelerator) is a real fixed-function INT8 NPU, but MediaTek's developer path is closed: the offline compiler ncc-tflite that turns a TFLite model into a .dla lives inside the Genio / NeuroPilot SDK, which is behind a partner registration wall. Without it, the documented story is "you can't target the MDLA." The earlier version of this project stalled there, waiting on SDK access that never came.

The bypass — the compiler is already on the phone

The retail firmware ships the runtime halves of the toolchain in /vendor/lib64/mt6886/: libneuron_adapter_mgvi.so (an NNAPI-like Adapter that builds and compiles a graph on-device) and libneuron_runtime.6.so (loads/runs a finalized .dla). The Adapter is enough: dlopen it, build the model operand-by-operand with NeuronModel_addOperand/addOperation, compile it on the phone, and execute — the on-device compiler does the lowering the gated host compiler would have. (storeCompiledNetwork even emits a .dla-equivalent cache blob, so compile-once / run-many works; that blob is an Adapter cache, not a RuntimeAPI .dla, which is why the bare RuntimeAPI loader rejects it — a real ABI gotcha that cost a wrong assumption.)

Proving it actually runs on the MDLA

Three independent checks, because "it returned a number" isn't proof of placement:

1. Per-device op gating — getSupportedOperationsForDevices reports the op supported on mtk-mdla and not on mtk-gpu/mtk-dsp, so forcing it onto the MDLA is legitimate.
2. No fallback — createForDevices({mtk-mdla}) errors (UNMAPPABLE) rather than silently using the CPU.
3. Independent hardware witness — MediaTek's MDLA ftrace tracepoints exist but are gated behind a MET kernel module that isn't loaded on retail firmware (dead end). Instead I witness the APU power domain's runtime_active_time: a 4-second idle control moves it +0 ms; an inference run moves it +154 ms and flips the device suspended→active. The counter moves only when the NPU does work.

The first arithmetic proof was a hand-built INT8 conv whose output matched a C golden bit-for-bit (256/256) on a varied (non-degenerate) output — real quantized arithmetic, not just device placement.

A small INT8 inference engine

On top of the raw path, an automated pipeline turns a stock PyTorch model into something running on the MDLA:

stock torchvision model
   │  E1  torch.fx trace → flat IR; fold BatchNorm into conv; fuse ReLU/ReLU6; classify CONV vs DEPTHWISE
   ▼
   │  E2  INT8 post-training quantization (per-channel symmetric weights, per-tensor asymmetric activations,
   │      int32 bias); calibrate on real images; verify int8 == fp32 accuracy
   ▼
   │  E3  legalize → emit a flat manifest (operands + op DAG); a C runtime builds it via the Adapter,
   │      force-compiles onto mtk-mdla, runs
   ▼
predicted class on the NPU  (validated vs the FP32 reference on real ImageNet images)

Four architecturally-different stock CNNs go through the same pipeline with no per-model code — that's the line between "I hand-built a model that runs" and "an engine":

• MobileNetV2 — depthwise-separable, ReLU6, inverted-residual adds. 83/83 ops forced on the MDLA, 1.6 ms/image.
• ResNet-18 — plain convolutions, a 3×3 MaxPool, ReLU, basic-block residuals. 50/50 ops, 2.8 ms/image.
• EfficientNet-B0 — squeeze-excite blocks, SiLU, broadcast multiplies, sigmoid gates. 256/256 ops, 3.9 ms/image.
• SqueezeNet1.1 — fire modules with channel CONCAT, no BatchNorm. 47/47 ops, 2.3 ms/image.

Each new architecture was added by teaching the engine ops, not by writing per-model code: ResNet needed MaxPool + ReLU-fusion; EfficientNet needed SiLU→LOGISTIC·MUL, sigmoid→LOGISTIC, and broadcast MUL; SqueezeNet needed CONCAT (with NNAPI's all-inputs-share-one-scale constraint) and no-BatchNorm conv→ReLU fusion. All four classify real ImageNet photos on the NPU at 10–11/12 on a 12-image probe (the misses are genuine near-ties like purse/mailbag).

Lowering rules — learned by measuring the silicon, not from docs

The MDLA maps most of what a CNN needs (CONV/DEPTHWISE/pooling/ADD/MUL/RELU{,6}/LOGISTIC/TANH/HARD_SWISH/SOFTMAX/ CONCAT/PAD/RESHAPE/MEAN/RESIZE_BILINEAR/BATCH_MATMUL — probed on real hardware). The rules that actually made models run:

• SAME padding is TF-asymmetric on the MDLA, but PyTorch is symmetric. Matching a PyTorch reference requires an explicit symmetric PAD + a VALID conv for stride-2 layers. (This was the headline lip-reading bug: a symmetric reference dropped to 0.83 cosine on the MDLA until forced symmetric → 0.997.)
• The dense classifier must become a 1×1 CONV_2D. The MDLA's FULLY_CONNECTED rejects the real classifier (1280→1000, per-channel weights); lowering the head to a 1×1 conv over the global-avg-pool output fixes it.
• getSupportedOperationsForDevices is all-or-nothing. One unsupported op reports the whole graph as 0/N, so you debug by op-prefix bisection, not by reading the (uniformly false) support list. This is exactly how the FC problem was isolated: K=82 ops mapped, K=83 (adding the FC) → 0/83.
• BatchNorm folds offline into conv weights; global-average-pool canonicalizes to MEAN; the INT8 I/O boundary (QUANTIZE/DEQUANTIZE, not MDLA-resident) is handled by an int8 I/O contract.

The application — lip-reading on the NPU

The driving use case: take a silent video of a mouth and print the sentence, with the heavy visual model on the NPU.

• Front-end on the MDLA. Visual-speech models start with a 3D (spatiotemporal) conv stem, which the MDLA can't map (no CONV_3D). I decompose each Conv3d into a stack of Conv2d by treating the 3 temporal taps as input channels — bit-exact to the original — so the front-end runs on the NPU. Two front-ends were ported: the auto-avsr ResNet-18 visual trunk (feature-matched to PyTorch at cosine 0.9969, 511 fps) and LipNet's STCNN.
• End-to-end text. For LipNet, the STCNN runs on the MDLA and a hand-written C tail (2× bidirectional GRU + greedy CTC) decodes on the CPU. On a held-out GRID clip it prints "SET WHITE WITH P TWO SOON" — the correct sentence — with the NPU portion hardware-witnessed at 1.16 ms/frame.
• Honest accuracy. On a seen speaker (sentences held out) it's 18/18 exact, 0% WER. On a genuinely unseen speaker (the real-world number, N=100) it's ~21% WER — and crucially the INT8/MDLA path is within 0.2% WER of the FP32 model: the quantized NPU port tracks the full-precision model faithfully, right answers and wrong.

Honest scope & limitations

What it is: a working INT8, static-shape, feed-forward CNN / keyword-spotting inference engine for the MT6886 MDLA, reached without the vendor SDK, validated on real hardware against FP32 references.

What it is not:

• Not transformer-capable. LayerNorm needs 1/sqrt(var+eps), and the MDLA has no sqrt/rsqrt/exp/log anywhere (probed). The only path is a Newton-Raphson approximation from MUL/SUB/ADD — and I measured that it fails: on real ViT-B/16 LayerNorm variances (an 8,600× range), per-tensor INT8 quantization collapses 9.2% of variances to zero (perfect-sqrt p90 error still 26%), and float NR with a constant seed diverges on 7.4%. Transformers would need higher-precision/offloaded normalization, not the MDLA. This is a genuine hardware op-set limit, stated with numbers.
• A general CNN compiler, but young — not a finished one. The four stages (fx import, INT8 quantize, legalize/emit, manifest runtime) are model-agnostic: each of the three architectures was added by teaching the engine op legalizations (MaxPool, ReLU-fusion, SiLU→logistic·mul, squeeze-excite broadcast-mul, CONCAT with shared-scale unification), not per-model code — the general-compiler property, now demonstrated across four distinct families. What's missing to call it "done": still-incremental op coverage (ShuffleNet channel-shuffle / grouped conv aren't wired yet — and channel-shuffle is MDLA-unsupported), packaging into one pass/CLI instead of three scripts, and the calibration/scale-consistency edge cases that surface as you add ops (a real one — a MaxPool scale bug — was caught by op-prefix bisection here). It generalizes within INT8 feed-forward CNNs; the rsqrt wall bounds it there.
• Not novel models. MobileNetV2/ResNet-18/LipNet are off-the-shelf. The engineering is the port and the lowering, not the networks.
• One device. Everything is the MT6886 / APU 650 / MDLA 3.5 on a Nothing Phone 2a.

Reproduce

# host: build the manifest from a stock model
python lipnet/e1_import.py resnet18      # fx import → IR   (or: mobilenet_v2)
python lipnet/e2_quantize.py             # INT8 PTQ, verify int8==fp32 top-1
python lipnet/e3_emit.py                 # legalize → graph_mnet.bin + weights_mnet.bin
# phone (root via KernelSU): build the runtime, force onto the MDLA, run
aarch64-linux-android30-clang -O2 -I. neuron_engine.c -o neuron_engine -ldl -lm
LD_LIBRARY_PATH=/vendor/lib64/mt6886:/vendor/lib64 ./neuron_engine     # prints class per image

neuron_ops_full.c prints the full measured op-support matrix; neuron_engine N builds only the first N ops (the bisection tool); ENGINE-READINESS.md is an adversarially-verified analysis of which model families this can and can't run.

Stack, licenses, ethics

• Hardware: Nothing Phone 2a — MediaTek MT6886, APU 650, MDLA 3.5; rooted (KernelSU-Next), Termux SSH.
• Toolchain: Android NDK r27c (aarch64), Python/PyTorch + torchvision on the host for import/PTQ.
• Models: LipNet (MIT), auto-avsr (Apache-2.0), torchvision MobileNetV2/ResNet-18 (BSD); GRID corpus is free.
• Ethics/legal: this is my own device, for research and education. The harness dlopens the vendor libraries already on the phone — it does not redistribute any MediaTek binaries. No SDK was obtained or bypassed in a licensing sense; the on-device compiler is simply driven through its public NNAPI-shaped entry points.

Status

Working and verified on-device. Open next steps: ship the lowering as a single reusable pass, add a third model family (EfficientNet-lite / a segmentation net), and improve PTQ calibration. Transformers are out of scope by the hardware (the rsqrt wall above).