From 26c41fc28da617ea5dffb2773d9f1bf9e2d7d49d Mon Sep 17 00:00:00 2001 From: Mengtao Yuan Date: Thu, 18 Apr 2024 20:14:29 -0700 Subject: [PATCH] Update Llama3 perplexity numbers in README.md Update Llama3 perplexity numbers in README.md, with 4-bit quantization with different group sizes. --- examples/models/llama2/README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/examples/models/llama2/README.md b/examples/models/llama2/README.md index 2244917d1e4..c55f53d4d56 100644 --- a/examples/models/llama2/README.md +++ b/examples/models/llama2/README.md @@ -24,11 +24,12 @@ For Llama3, we can use the same process. Note that it's only supported in the Ex ## Quantization: We employed 4-bit groupwise per token dynamic quantization of all the linear layers of the model. Dynamic quantization refers to quantizating activations dynamically, such that quantization parameters for activations are calculated, from min/max range, at runtime. Here we quantized activations with 8bits (signed integer). Furthermore, weights are statically quantized. In our case weights were per-channel groupwise quantized with 4bit signed integer. For more information refer to this [page](https://github.com/pytorch-labs/ao/). -We evaluated WikiText perplexity using [LM Eval](https://github.com/EleutherAI/lm-evaluation-harness). Below are the results for two different groupsizes. +We evaluated WikiText perplexity using [LM Eval](https://github.com/EleutherAI/lm-evaluation-harness). Below are the results for two different groupsizes, with max_seq_len 2048, and 1000 samples. -|Llama 2 | Baseline (FP32) | Groupwise 4-bit (128) | Groupwise 4-bit (256) +|Model | Baseline (FP32) | Groupwise 4-bit (128) | Groupwise 4-bit (256) |--------|-----------------| ---------------------- | --------------- -|Wikitext Perplexity | 9.16 | 10.2 | 10.7 +|Llama 2 7B | 9.2 | 10.2 | 10.7 +|Llama 3 8B | 7.9 | 9.4 | 9.7 Note that groupsize less than 128 was not enabled, since such model were still too large. This is because our current efforts have focused on enabling FP32 and support for FP16 is under way. What this implies for model size is that 1) embedding table is in FP32 and 2) quantized weights scales are FP32.