Add CoreML Quantize

[YifanShenSZ]

Motivation

Short term: TorchAO int4 quantization yields float zero point, but CoreML does not have good support for it yet. We will need CoreML int4 quantization for now.

Intermediate term: Before torch implements all CoreML-supported quantizations (e.g. palettization, sparcification, joint compression...), it will be great to have a way to use/experiment those CoreML quantizations.

Solution

In CoreML preprocess, we add CoreML quantization config as a compile spec

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/5228

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure

As of commit 554382a with merge base 7e374d7 ():

NEW FAILURES - The following jobs have failed:

• Apple / test-demo-ios / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-models-macos (cmake, llama2, portable, macos-m1-stable, 90) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• Apple / upload-frameworks-ios (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[YifanShenSZ]

@cccclai 🙏

[cccclai]

Hey could you share the command to run the script? The arg list is getting longer and it's hard to guess...

[YifanShenSZ]

Hey could you share the command to run the script? The arg list is getting longer and it's hard to guess...

Sure

python -m examples.models.llama2.export_llama -c Meta-Llama-3-8B/consolidated.00.pth -p Meta-Llama-3-8B/params.json --disable_dynamic_shape -kv --coreml --coreml-quantize b4w --coreml-enable-state

This is not the final command, though: we are adding fused sdpa

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hey could you rebase? Run into land race and the other PR touched the same file merge first...

[YifanShenSZ]

Rebased ✅

GitHub is not showing conflict yet, though. Is the conflict change in Meta internal only for now? (And I need to wait until it gets exported?)

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai merged this pull request in 4da3c5d.