Add quantized op support to llama runner by larryliu0820 · Pull Request #3062 · pytorch/executorch

larryliu0820 · 2024-04-16T17:32:47Z

Stack from ghstack (oldest at bottom):

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

Differential Revision: D56197863

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

pytorch-bot · 2024-04-16T17:32:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3062

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 41abbb5 with merge base 458d743 ():

NEW FAILURE - The following job has failed:

Apple / build-frameworks-ios / macos-job (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 5110c26 Pull Request resolved: #3062

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 15fec06 Pull Request resolved: #3062

larryliu0820 · 2024-04-16T17:41:32Z

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 008a4ba Pull Request resolved: #3062

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 5ebb8b6 Pull Request resolved: #3062

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 39697a8 Pull Request resolved: #3062

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 6206bce Pull Request resolved: #3062

kimishpatel · 2024-04-17T00:02:12Z

        -DEXECUTORCH_BUILD_OPTIMIZED=ON \
        -DEXECUTORCH_BUILD_XNNPACK="$XNNPACK" \
-        -DEXECUTORCH_BUILD_OPTIMIZED=ON \
+        -DEXECUTORCH_BUILD_QUANTIZED="$QE" \


Can we not always build? haveing so many options for build feels like additional burden for users. Maybe do default opt-in?

mikekgfb

Thanks!

kimishpatel · 2024-04-17T00:04:12Z

-  EXPORT_ARGS="${EXPORT_ARGS} -kv --use_sdpa_with_kv_cache -X -qmode 8da4w -G 128"
+EXPORT_ARGS="-c stories110M.pt -p ${PARAMS} -d ${DTYPE} -n ${EXPORTED_MODEL_NAME} -kv"
+if [[ "${XNNPACK}" == "ON" ]]; then
+  EXPORT_ARGS="${EXPORT_ARGS} -X -qmode 8da4w -G 128"


nit: does += operator work?

kimishpatel · 2024-04-17T00:07:15Z

 if(EXECUTORCH_USE_TIKTOKEN)
  # find RE2 for tokenizer
  set(ABSL_ENABLE_INSTALL ON)
+  set(ABSL_PROPAGATE_CXX_STD ON)


oh we depend on abseil for tiktoken?

Yeah, tiktoken -> re2 -> abseil

lucylq · 2024-04-17T00:12:31Z

 if(EXECUTORCH_USE_TIKTOKEN)
  # find RE2 for tokenizer
  set(ABSL_ENABLE_INSTALL ON)
+  set(ABSL_PROPAGATE_CXX_STD ON)


no tests using this path yet right?

lucylq · 2024-04-17T00:12:42Z

+  EXPORT_ARGS="${EXPORT_ARGS} --use_sdpa_with_kv_cache"
+fi
+if [[ "${QE}" == "ON" ]]; then
+  EXPORT_ARGS="${EXPORT_ARGS} --embedding-quantize 8,1024"


thanks for adding tests!

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: d1dd78a Pull Request resolved: #3062

larryliu0820 · 2024-04-17T00:54:36Z

@larryliu0820 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot · 2024-04-17T02:08:10Z

This pull request has been merged in 1f4b631.

Summary: As titled. Got too excited in #3062 and removed `EXECUTORCH_BUILD_QUANTIZED`. Looking at the CI job failure of `build-apple-framework` probably worth adding it back. Test Plan: See that CI job pass Reviewers: Subscribers: Tasks: Tags:

Summary: As titled. Got too excited in #3062 and removed `EXECUTORCH_BUILD_QUANTIZED`. Looking at the CI job failure of `build-apple-framework` probably worth adding it back. Test Plan: See that CI job pass Reviewed By: shoumikhin Differential Revision: D56281923 Pulled By: larryliu0820

Summary: As titled. Got too excited in #3062 and removed `EXECUTORCH_BUILD_QUANTIZED`. Looking at the CI job failure of `build-apple-framework` probably worth adding it back. Pull Request resolved: #3115 Test Plan: See that CI job pass Reviewed By: shoumikhin Differential Revision: D56281923 Pulled By: larryliu0820 fbshipit-source-id: e6ad411f763ff8e11d4fb1e0bc7037eb2cf69357

Add quantized op support to llama runner

f6b2fa6

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

larryliu0820 mentioned this pull request Apr 16, 2024

Fix test_llama_runner by hiding tiktoken #3055

Closed

larryliu0820 added a commit that referenced this pull request Apr 16, 2024

Add quantized op support to llama runner

35caa4e

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 5110c26 Pull Request resolved: #3062

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 16, 2024

Update on "Add quantized op support to llama runner"

a0e1193

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: [ghstack-poisoned]

larryliu0820 added a commit that referenced this pull request Apr 16, 2024

Add quantized op support to llama runner

48e60d7

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 15fec06 Pull Request resolved: #3062

Update on "Add quantized op support to llama runner"

6670a16

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

larryliu0820 added a commit that referenced this pull request Apr 16, 2024

Add quantized op support to llama runner

df0dd79

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 008a4ba Pull Request resolved: #3062

Update on "Add quantized op support to llama runner"

020dc4e

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

larryliu0820 added a commit that referenced this pull request Apr 16, 2024

Add quantized op support to llama runner

69551ba

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 5ebb8b6 Pull Request resolved: #3062

Update on "Add quantized op support to llama runner"

38cfb8d

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

larryliu0820 added a commit that referenced this pull request Apr 16, 2024

Add quantized op support to llama runner

55e8932

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 39697a8 Pull Request resolved: #3062

Update on "Add quantized op support to llama runner"

3291cb7

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

larryliu0820 added a commit that referenced this pull request Apr 16, 2024

Add quantized op support to llama runner

f4f4254

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: 6206bce Pull Request resolved: #3062

kimishpatel reviewed Apr 17, 2024

View reviewed changes

mikekgfb approved these changes Apr 17, 2024

View reviewed changes

kimishpatel reviewed Apr 17, 2024

View reviewed changes

Comment thread .ci/scripts/test_quantized_aot_lib.sh

kimishpatel reviewed Apr 17, 2024

View reviewed changes

lucylq approved these changes Apr 17, 2024

View reviewed changes

Update on "Add quantized op support to llama runner"

41abbb5

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: Differential Revision: [D56197863](https://our.internmc.facebook.com/intern/diff/D56197863) [ghstack-poisoned]

larryliu0820 added a commit that referenced this pull request Apr 17, 2024

Add quantized op support to llama runner

68c9735

Summary: Test Plan: Reviewers: Subscribers: Tasks: Tags: ghstack-source-id: d1dd78a Pull Request resolved: #3062

facebook-github-bot closed this in 1f4b631 Apr 17, 2024

facebook-github-bot added the Merged label Apr 17, 2024

larryliu0820 mentioned this pull request Apr 18, 2024

Add quantized cmake option back to fix build-apple-framework #3115

Closed

mergennachin mentioned this pull request Apr 26, 2024

disclaimer #3376

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add quantized op support to llama runner#3062

Add quantized op support to llama runner#3062
larryliu0820 wants to merge 7 commits intogh/larryliu0820/27/basefrom
gh/larryliu0820/27/head

larryliu0820 commented Apr 16, 2024 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Apr 16, 2024 •

edited

Loading

Uh oh!

larryliu0820 commented Apr 16, 2024

Uh oh!

kimishpatel Apr 17, 2024

Uh oh!

mikekgfb left a comment

Uh oh!

kimishpatel Apr 17, 2024

Uh oh!

Uh oh!

kimishpatel Apr 17, 2024

Uh oh!

larryliu0820 Apr 17, 2024

Uh oh!

lucylq Apr 17, 2024

Uh oh!

larryliu0820 Apr 17, 2024

Uh oh!

lucylq Apr 17, 2024

Uh oh!

larryliu0820 commented Apr 17, 2024

Uh oh!

facebook-github-bot commented Apr 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

larryliu0820 commented Apr 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot Bot commented Apr 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/3062

❌ 1 New Failure

Uh oh!

larryliu0820 commented Apr 16, 2024

Uh oh!

kimishpatel Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

mikekgfb left a comment

Choose a reason for hiding this comment

Uh oh!

kimishpatel Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kimishpatel Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

larryliu0820 Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

lucylq Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

larryliu0820 Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

lucylq Apr 17, 2024

Choose a reason for hiding this comment

Uh oh!

larryliu0820 commented Apr 17, 2024

Uh oh!

facebook-github-bot commented Apr 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

larryliu0820 commented Apr 16, 2024 •

edited

Loading

pytorch-bot Bot commented Apr 16, 2024 •

edited

Loading