From eba6f4cdedcfe648e64947ac695c8342a816af40 Mon Sep 17 00:00:00 2001 From: Tianlei Wu Date: Sat, 16 May 2020 13:26:34 -0700 Subject: [PATCH 1/4] Use shorter name for tools --- .../python/tools/transformers/README.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/onnxruntime/python/tools/transformers/README.md b/onnxruntime/python/tools/transformers/README.md index 6c1a521ce44ab..1189e76200e6b 100644 --- a/onnxruntime/python/tools/transformers/README.md +++ b/onnxruntime/python/tools/transformers/README.md @@ -18,7 +18,16 @@ pip install onnxruntime-tools After it is installed, you can use command like the following to optimize model: ```console -python -m onnxruntime_tools.transformers.optimizer --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 +python -m onnxruntime_tools.optimizer --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 +``` + +In your python code, you can use it like the following: + +```python +from onnxruntime_tools import optimizer +optimized_model = optimizer.optimize_model("gpt2.onnx", model_type='gpt2', num_heads=12, hidden_size=768) +optimized_model.convert_model_float32_to_float16() +optimized_model.save_model_to_file("gpt2_fp16.onnx") ``` If you want to use the latest script, you can get script files from [here](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/). Then run it like the following: @@ -36,7 +45,7 @@ For tf2onnx, please refer to [its BERT tutorial](https://github.com/onnx/tensorf Example of using the script optimizer.py to optimize a BERT-large model to run in V100 GPU: ```console -python -m onnxruntime_tools.transformers.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 +python -m onnxruntime_tools.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 ``` ### Options @@ -82,7 +91,7 @@ The benchmark script requires PyTorch be installed. You can run benchmark script to see the inference speed of OnnxRuntime. Here is an example to run benchmark on a pretrained model bert-base-cased on GPU. ```console -python -m onnxruntime_tools.transformers.benchmark -m bert-base-cased -g +python -m onnxruntime_tools.benchmark -m bert-base-cased -g ``` ## Model Verification @@ -92,7 +101,7 @@ If your model has three inputs (like input_ids, token_type_ids and attention_mas Example of verifying models optimized for CPU: ```console -python -m onnxruntime_tools.transformers.compare_bert_results --baseline_model original_model.onnx --optimized_model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 +python -m onnxruntime_tools.compare_bert_results --baseline_model original_model.onnx --optimized_model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 ``` For GPU, please append --use_gpu to the command. @@ -102,7 +111,7 @@ For GPU, please append --use_gpu to the command. bert_perf_test.py can be used to check the model inference performance. Below are examples: ```console -python -m onnxruntime_tools.transformers.bert_perf_test --model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 --test_times 10 --inclusive +python -m onnxruntime_tools.bert_perf_test --model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 --test_times 10 --inclusive ``` For GPU, please append --use_gpu to the command. From e6ed8f1559635225c4347f6a9ba7dfe658202de4 Mon Sep 17 00:00:00 2001 From: Tianlei Wu Date: Sat, 16 May 2020 13:44:57 -0700 Subject: [PATCH 2/4] revert to longer name --- onnxruntime/python/tools/transformers/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/onnxruntime/python/tools/transformers/README.md b/onnxruntime/python/tools/transformers/README.md index 1189e76200e6b..98ddca9bcce34 100644 --- a/onnxruntime/python/tools/transformers/README.md +++ b/onnxruntime/python/tools/transformers/README.md @@ -18,13 +18,13 @@ pip install onnxruntime-tools After it is installed, you can use command like the following to optimize model: ```console -python -m onnxruntime_tools.optimizer --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 +python -m onnxruntime_tools.transformers.optimizer --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 ``` In your python code, you can use it like the following: ```python -from onnxruntime_tools import optimizer +from onnxruntime_tools.transformers import optimizer optimized_model = optimizer.optimize_model("gpt2.onnx", model_type='gpt2', num_heads=12, hidden_size=768) optimized_model.convert_model_float32_to_float16() optimized_model.save_model_to_file("gpt2_fp16.onnx") @@ -45,7 +45,7 @@ For tf2onnx, please refer to [its BERT tutorial](https://github.com/onnx/tensorf Example of using the script optimizer.py to optimize a BERT-large model to run in V100 GPU: ```console -python -m onnxruntime_tools.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 +python -m onnxruntime_tools.transformers.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 ``` ### Options @@ -91,7 +91,7 @@ The benchmark script requires PyTorch be installed. You can run benchmark script to see the inference speed of OnnxRuntime. Here is an example to run benchmark on a pretrained model bert-base-cased on GPU. ```console -python -m onnxruntime_tools.benchmark -m bert-base-cased -g +python -m onnxruntime_tools.transformers.benchmark -m bert-base-cased -g ``` ## Model Verification @@ -101,7 +101,7 @@ If your model has three inputs (like input_ids, token_type_ids and attention_mas Example of verifying models optimized for CPU: ```console -python -m onnxruntime_tools.compare_bert_results --baseline_model original_model.onnx --optimized_model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 +python -m onnxruntime_tools.transformers.compare_bert_results --baseline_model original_model.onnx --optimized_model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 ``` For GPU, please append --use_gpu to the command. @@ -111,7 +111,7 @@ For GPU, please append --use_gpu to the command. bert_perf_test.py can be used to check the model inference performance. Below are examples: ```console -python -m onnxruntime_tools.bert_perf_test --model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 --test_times 10 --inclusive +python -m onnxruntime_tools.transformers.bert_perf_test --model optimized_model_cpu.onnx --batch_size 1 --sequence_length 128 --samples 100 --test_times 10 --inclusive ``` For GPU, please append --use_gpu to the command. From 308e93120cb0c02c4546d8a01091a77b6cd4df69 Mon Sep 17 00:00:00 2001 From: Tianlei Wu Date: Sat, 16 May 2020 18:27:30 -0700 Subject: [PATCH 3/4] Use optimizer_cli --- .../python/tools/transformers/README.md | 30 +++++++++++++------ 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/onnxruntime/python/tools/transformers/README.md b/onnxruntime/python/tools/transformers/README.md index 98ddca9bcce34..dc9a55e1dab4a 100644 --- a/onnxruntime/python/tools/transformers/README.md +++ b/onnxruntime/python/tools/transformers/README.md @@ -16,20 +16,20 @@ This tool can be installed using pip as follows: pip install onnxruntime-tools ``` -After it is installed, you can use command like the following to optimize model: -```console -python -m onnxruntime_tools.transformers.optimizer --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 -``` - In your python code, you can use it like the following: ```python -from onnxruntime_tools.transformers import optimizer +from onnxruntime_tools import optimizer optimized_model = optimizer.optimize_model("gpt2.onnx", model_type='gpt2', num_heads=12, hidden_size=768) optimized_model.convert_model_float32_to_float16() optimized_model.save_model_to_file("gpt2_fp16.onnx") ``` +You can also use command like the following to optimize model: +```console +python -m onnxruntime_tools.optimizer_cli --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 +``` + If you want to use the latest script, you can get script files from [here](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers/). Then run it like the following: ```console python optimizer.py --input gpt2.onnx --output gpt2_opt.onnx --model_type gpt2 @@ -45,7 +45,7 @@ For tf2onnx, please refer to [its BERT tutorial](https://github.com/onnx/tensorf Example of using the script optimizer.py to optimize a BERT-large model to run in V100 GPU: ```console -python -m onnxruntime_tools.transformers.optimizer --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 +python -m onnxruntime_tools.optimizer_cli --input bert_large.onnx --output bert_large_fp16.onnx --num_heads 16 --hidden_size 1024 --float16 ``` ### Options @@ -88,11 +88,23 @@ For GPT2 models, current optimization does not support past state (both inputs a The benchmark script requires PyTorch be installed. -You can run benchmark script to see the inference speed of OnnxRuntime. Here is an example to run benchmark on a pretrained model bert-base-cased on GPU. +You can run benchmark script to see the inference speed of OnnxRuntime. Here is an example to run benchmark on pretrained model bert-base-cased on GPU. ```console -python -m onnxruntime_tools.transformers.benchmark -m bert-base-cased -g +python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -o -v -b 0 +python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -o +python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -e torch +python -m onnxruntime_tools.transformers.benchmark -g -m bert-base-cased -e torchscript ``` +The first command will generate ONNX models (both before and after optimizations), but not run performance tests since we set batch size to 0. The other three commands will run performance test on three engines: OnnxRuntime, PyTorch and PyTorch+TorchScript. + +If you remove -o parameter, optimizer is not used in benchmark. + +If your GPU (like V100 or T4) has TensorCore, you can append --fp16 to the above commands to enable mixed precision using float16. + +If you want to benchmark on CPU, you can remove -g option in the commands. + +Note that our current benchmark on GPT2 model has disabled past state from inputs and outputs. ## Model Verification From 8efd3d9eeced6729a33015b1053a867236ef9263 Mon Sep 17 00:00:00 2001 From: Tianlei Wu Date: Sat, 16 May 2020 18:35:18 -0700 Subject: [PATCH 4/4] Add comments about -i parameter --- onnxruntime/python/tools/transformers/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/onnxruntime/python/tools/transformers/README.md b/onnxruntime/python/tools/transformers/README.md index dc9a55e1dab4a..53bf5e6968ee2 100644 --- a/onnxruntime/python/tools/transformers/README.md +++ b/onnxruntime/python/tools/transformers/README.md @@ -106,6 +106,8 @@ If you want to benchmark on CPU, you can remove -g option in the commands. Note that our current benchmark on GPT2 model has disabled past state from inputs and outputs. +By default, ONNX model has only one input (input_ids). You can use -i parameter to test models with more inputs. For example, we can add "-i 3" to command line to test a bert model with 3 inputs (input_ids, token_type_ids and attention_mask). The performance result might be different. This option only supports OnnxRuntime right now. + ## Model Verification If your model has three inputs (like input_ids, token_type_ids and attention_mask), a script compare_bert_results.py can be used to do a quick verification. The tool will generate some fake input data, and compare results from both the original and optimized models. If outputs are all close, it is safe to use the optimized model.