Skip to content

aitk

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

aitk

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
.gitignore
.gitignore
 
 
README.md
README.md
 
 
_copy.json.config
_copy.json.config
 
 
bert-base-multilingual-cased_context_ov_static.json
bert-base-multilingual-cased_context_ov_static.json
 
 
bert-base-multilingual-cased_context_ov_static.json.config
bert-base-multilingual-cased_context_ov_static.json.config
 
 
bert-base-multilingual-cased_dml.json
bert-base-multilingual-cased_dml.json
 
 
bert-base-multilingual-cased_dml.json.config
bert-base-multilingual-cased_dml.json.config
 
 
bert-base-multilingual-cased_migraphx.json
bert-base-multilingual-cased_migraphx.json
 
 
bert-base-multilingual-cased_migraphx.json.config
bert-base-multilingual-cased_migraphx.json.config
 
 
bert-base-multilingual-cased_qdq_amd.json
bert-base-multilingual-cased_qdq_amd.json
 
 
bert-base-multilingual-cased_qdq_amd.json.config
bert-base-multilingual-cased_qdq_amd.json.config
 
 
bert-base-multilingual-cased_qdq_qnn.json
bert-base-multilingual-cased_qdq_qnn.json
 
 
bert-base-multilingual-cased_qdq_qnn.json.config
bert-base-multilingual-cased_qdq_qnn.json.config
 
 
bert-base-multilingual-cased_qnn_gpu.json
bert-base-multilingual-cased_qnn_gpu.json
 
 
bert-base-multilingual-cased_qnn_gpu.json.config
bert-base-multilingual-cased_qnn_gpu.json.config
 
 
bert-base-multilingual-cased_trtrtx.json
bert-base-multilingual-cased_trtrtx.json
 
 
bert-base-multilingual-cased_trtrtx.json.config
bert-base-multilingual-cased_trtrtx.json.config
 
 
inference_sample.ipynb
inference_sample.ipynb
 
 
info.yml
info.yml
 
 
model_project.config
model_project.config
 
 
requirements.txt
requirements.txt
 
 
user_script.py
user_script.py
 
 
winml.py
winml.py
 
 

README.md

BERT Optimization

This folder contains examples of BERT optimization using different workflows.

  • QDQ for Qualcomm NPU / AMD NPU
  • OpenVINO for Intel® CPU/GPU/NPU
  • Float downcasting for NVIDIA TRT for RTX GPU / DML for general GPU

QDQ for Qualcomm NPU / AMD NPU

This workflow quantizes the model. It performs the pipeline:

  • HF Model-> ONNX Model ->Quantized Onnx Model

Intel® Workflows

This workflow performs quantization with OpenVINO NNCF. It performs the optimization pipeline:

  • HuggingFace Model -> OpenVINO Model -> Quantized OpenVINO model -> Quantized encapsulated ONNX OpenVINO IR model

Latency / Throughput

Model Version Latency (ms/sample) Throughput (token per second) Dataset
PyTorch FP32 1162 0.81 facebook/xnli
ONNX INT8 (QDQ) 590 1.75 facebook/xnli

Note: Latency can vary significantly depending on the hardware and system environment. The values provided here are for reference only and may not reflect performance on all devices.