Update x64 template kernel library for 'sqnbitgemm' by luoyu-intel · Pull Request #19016 · microsoft/onnxruntime

luoyu-intel · 2024-01-05T03:25:11Z

Description

Make JBLAS codes an external module of ORT.
Move q4 gemm code to contrib_ops.
Update template kernel library to v0.1 release.

Motivation and Context

We found that the current LLM model performance is far below our expectations. Here is some performance data collected on Mistral-7B model with Xeon-8480:

8 threads	prompt length=32 past_len=32	prompt length=1 past_len=32
ORT-main	1220ms	263ms
Neural-speed	564ms	87ms
ORT-this PR	597ms	120ms

Although Neural-speed and ORT-this PR use the same int4 kernel code, there is a 33ms(87ms vs. 120ms) latency gap between the two frameworks.
Through some statistics analysis, the summary latency of MatMulNBits is 86.7ms
The summary latency of all int4 GEMMs in Neural-speed is 84.8ms. So other OPs introduce an extra 30ms latency.

The performance of MatMulNBits in this PR meets our expectations.

Remain Issues

For hybrid CPUs, like core 12900K, the ONNXRuntime thread pool uses TaskGranularityFactor to scale its number of threads. This is not expected in our code design. It may slow down the hybrid CPU performance by 30~40%.
Prepack uses a single thread which is very slow to init a session.
MatMulNBits with zero points will fall through to COMP_FP32 even accuracy_level=4. Our COMP_INT8 IGemmCore with zero points process is not optimized for now. It will be updated in the future. So, for an int4 model with zero points, whether the accuracy_level is 0 or 4 will be no difference.

yufenglee · 2024-01-08T21:16:28Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-08T21:16:39Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-08T21:16:58Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-08T21:17:18Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee · 2024-01-09T17:16:44Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-09T17:16:51Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-09T17:17:16Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-09T17:17:33Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee · 2024-01-10T00:26:36Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-10T00:26:50Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-10T00:27:06Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-10T00:27:31Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee · 2024-01-10T15:58:15Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-10T15:58:23Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-10T15:58:47Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-10T15:59:07Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee · 2024-01-11T15:18:53Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-11T15:19:03Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-11T15:19:24Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-11T15:19:44Z

Azure Pipelines successfully started running 10 pipeline(s).

luoyu-intel · 2024-01-17T11:51:45Z

/azp run Linux CPU CI Pipeline

azure-pipelines · 2024-01-17T11:51:51Z

Commenter does not have sufficient privileges for PR 19016 in repo microsoft/onnxruntime

yufenglee · 2024-01-17T16:00:03Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-17T16:00:11Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-17T16:01:11Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-17T16:01:25Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee · 2024-01-18T02:15:40Z

/azp run Windows GPU TensorRT CI Pipeline, Windows x64 QNN CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

yufenglee · 2024-01-18T02:15:49Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, Linux QNN CI Pipeline, MacOS CI Pipeline, Windows ARM64 QNN CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2024-01-18T02:16:09Z

Azure Pipelines successfully started running 6 pipeline(s).

azure-pipelines · 2024-01-18T02:16:27Z

Azure Pipelines successfully started running 10 pipeline(s).

yufenglee

This reverts commit 459c750.

### Description  1. Make JBLAS codes an external module of ORT. 2. Move q4 gemm code to contrib_ops. 3. Update template kernel library to v0.1 release. ### Motivation and Context  We found that the current LLM model performance is far below our expectations. Here is some performance data collected on Mistral-7B model with Xeon-8480: 8 threads | prompt length=32 past_len=32 | prompt length=1 past_len=32 -- | -- | -- ORT-main | 1220ms | 263ms Neural-speed | 564ms | 87ms ORT-this PR|597ms|120ms Although `Neural-speed` and `ORT-this PR` use the same int4 kernel code, there is a 33ms(87ms vs. 120ms) latency gap between the two frameworks. Through some statistics analysis, the summary latency of `MatMulNBits` is 86.7ms The summary latency of all int4 GEMMs in `Neural-speed` is 84.8ms. So other OPs introduce an extra 30ms latency. The performance of MatMulNBits in this PR meets our expectations. ### Remain Issues 1. For hybrid CPUs, like core 12900K, the ONNXRuntime thread pool uses TaskGranularityFactor to scale its number of threads. This is not expected in our code design. It may slow down the hybrid CPU performance by 30~40%. 2. Prepack uses a single thread which is very slow to init a session. 3. MatMulNBits with zero points will fall through to COMP_FP32 even accuracy_level=4. Our COMP_INT8 IGemmCore with zero points process is not optimized for now. It will be updated in the future. So, for an int4 model with zero points, whether the accuracy_level is 0 or 4 will be no difference.

### Description  Revert PR#19016 #19016 Revert PR#17669 #17669

### Description  Revert PR#19016 microsoft/onnxruntime#19016 Revert PR#17669 microsoft/onnxruntime#17669

snnn · 2025-09-05T21:22:57Z

This PR has been cherry-picked into the rel-1.17.0 branch in PR #19243. Removing the release:1.17.0 label.

### Description  Revert PR#19016 microsoft/onnxruntime#19016 Revert PR#17669 microsoft/onnxruntime#17669

luoyu-intel marked this pull request as ready for review January 5, 2024 08:23

luoyu-intel requested a review from a team as a code owner January 5, 2024 08:23

snnn reviewed Jan 5, 2024

View reviewed changes

Comment thread cmake/external/neural_speed.cmake Outdated

edgchen1 reviewed Jan 6, 2024

View reviewed changes

Comment thread cmake/external/neural_speed.cmake Outdated

github-advanced-security AI found potential problems Jan 9, 2024

View reviewed changes

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.cc Fixed

edgchen1 reviewed Jan 9, 2024

View reviewed changes

Comment thread cmake/onnxruntime_providers_cpu.cmake Outdated

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.cc Outdated

edgchen1 reviewed Jan 10, 2024

View reviewed changes

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.h Outdated

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.cc Outdated

snnn reviewed Jan 11, 2024

View reviewed changes

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.cc Outdated

snnn reviewed Jan 11, 2024

View reviewed changes

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.h Outdated

snnn reviewed Jan 11, 2024

View reviewed changes

Comment thread onnxruntime/contrib_ops/cpu/quantization/bestla_gemm.h Outdated

Merge branch 'microsoft:main' into main

9233faf

yufenglee added the release:1.17 label Jan 17, 2024

snnn reviewed Jan 17, 2024

View reviewed changes

Comment thread cmake/deps.txt Outdated

snnn reviewed Jan 17, 2024

View reviewed changes

Comment thread cmake/external/neural_speed.cmake Outdated

luoyu-intel added 2 commits January 18, 2024 09:45

use URL directly

8c2a8c5

Merge branch 'microsoft:main' into main

912c4b9

yufenglee approved these changes Jan 18, 2024

View reviewed changes

yufenglee merged commit 459c750 into microsoft:main Jan 18, 2024

jywu-mysoft added release:1.17.0 and removed release:1.17 labels Jan 18, 2024

mszhanyi added a commit that referenced this pull request Jan 23, 2024

Revert "Update x64 template kernel library for 'sqnbitgemm' (#19016)"

0dfd2fa

This reverts commit 459c750.

edgchen1 mentioned this pull request Jan 24, 2024

Update abseil to a release tag and register neural_speed #19255

Merged

luoyu-intel mentioned this pull request Feb 2, 2024

Revert NeuralSpeed code for x64 MatMulNBits #19382

Merged

snnn pushed a commit that referenced this pull request Feb 7, 2024

Revert NeuralSpeed code for x64 MatMulNBits (#19382)

0d10c7f

### Description  Revert PR#19016 #19016 Revert PR#17669 #17669

skottmckay pushed a commit that referenced this pull request Feb 15, 2024

Revert NeuralSpeed code for x64 MatMulNBits (#19382)

c10d6a4

### Description  Revert PR#19016 #19016 Revert PR#17669 #17669

snnn removed the release:1.17.0 label Sep 5, 2025

Uh oh!

Conversation

luoyu-intel commented Jan 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Remain Issues

Uh oh!

Uh oh!

Uh oh!

yufenglee commented Jan 8, 2024

Uh oh!

yufenglee commented Jan 8, 2024

Uh oh!

azure-pipelines Bot commented Jan 8, 2024

Uh oh!

azure-pipelines Bot commented Jan 8, 2024

Uh oh!

yufenglee commented Jan 9, 2024

Uh oh!

yufenglee commented Jan 9, 2024

Uh oh!

azure-pipelines Bot commented Jan 9, 2024

Uh oh!

azure-pipelines Bot commented Jan 9, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yufenglee commented Jan 10, 2024

Uh oh!

yufenglee commented Jan 10, 2024

Uh oh!

azure-pipelines Bot commented Jan 10, 2024

Uh oh!

azure-pipelines Bot commented Jan 10, 2024

Uh oh!

yufenglee commented Jan 10, 2024

Uh oh!

yufenglee commented Jan 10, 2024

Uh oh!

azure-pipelines Bot commented Jan 10, 2024

Uh oh!

azure-pipelines Bot commented Jan 10, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yufenglee commented Jan 11, 2024

Uh oh!

yufenglee commented Jan 11, 2024

Uh oh!

azure-pipelines Bot commented Jan 11, 2024

Uh oh!

azure-pipelines Bot commented Jan 11, 2024

Uh oh!

luoyu-intel commented Jan 17, 2024

Uh oh!

azure-pipelines Bot commented Jan 17, 2024

Uh oh!

yufenglee commented Jan 17, 2024

Uh oh!

yufenglee commented Jan 17, 2024

Uh oh!

azure-pipelines Bot commented Jan 17, 2024

Uh oh!

azure-pipelines Bot commented Jan 17, 2024

Uh oh!

Uh oh!

Uh oh!

yufenglee commented Jan 18, 2024

Uh oh!

yufenglee commented Jan 18, 2024

Uh oh!

azure-pipelines Bot commented Jan 18, 2024

Uh oh!

azure-pipelines Bot commented Jan 18, 2024

Uh oh!

luoyu-intel commented Jan 5, 2024 •

edited

Loading