From 07c71cf919312d93dd20417219efccdecb55f4bf Mon Sep 17 00:00:00 2001 From: Chen Lai Date: Fri, 19 Apr 2024 13:40:24 -0700 Subject: [PATCH] Docs for lower smaller models to mps/coreml/qnn (#3146) Summary: Pull Request resolved: https://github.com/pytorch/executorch/pull/3146 ghstack-source-id: 223235858 Reviewed By: mcr229, kirklandsign Differential Revision: D56340028 fbshipit-source-id: ef06142546ac54105ae87007cd82369917a22b3e (cherry picked from commit d47f9fe5a3bf19d075a27865c13b6ffd2ee208da) --- examples/models/llama2/README.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/examples/models/llama2/README.md b/examples/models/llama2/README.md index 07a0534ff2c..6bc7dd0fa0b 100644 --- a/examples/models/llama2/README.md +++ b/examples/models/llama2/README.md @@ -238,6 +238,16 @@ Please refer to [this tutorial](https://pytorch.org/executorch/main/llm/llama-de ### Android Please refer to [this tutorial](https://pytorch.org/executorch/main/llm/llama-demo-android.html) to for full instructions on building the Android LLAMA Demo App. +## Optional: Smaller models delegated to other backends +Currently we supported lowering the stories model to other backends, including, CoreML, MPS and QNN. Please refer to the instruction +for each backend ([CoreML](https://pytorch.org/executorch/main/build-run-coreml.html), [MPS](https://pytorch.org/executorch/main/build-run-mps.html), [QNN](https://pytorch.org/executorch/main/build-run-qualcomm.html)) before trying to lower them. After the backend library is installed, the script to export a lowered model is + +- Lower to CoreML: `python -m examples.models.llama2.export_llama -kv --coreml -c stories110M.pt -p params.json` +- MPS: `python -m examples.models.llama2.export_llama -kv --mps -c stories110M.pt -p params.json` +- QNN: `python -m examples.models.llama2.export_llama -kv --qnn -c stories110M.pt -p params.json` + +The iOS LLAMA app supports the CoreML and MPS model and the Android LLAMA app supports the QNN model. On Android, it also allow to cross compiler the llama runner binary, push to the device and run. + # What is coming next? ## Quantization - Enabling FP16 model to leverage smaller groupsize for 4-bit quantization.