GPTQ quantization(3 or 4 bit quantization) support for LLaMa

[qwopqwop200]

GPTQ is currently the SOTA one shot quantization method for LLMs.
GPTQ supports amazingly low 3-bit and 4-bit weight quantization. And it can be applied to LLaMa.
I've actually confirmed that this works well in LLaMa 7b.
I haven't tested the memory usage(n-bit cuda kernel), but I think it should work.

Model(LLaMa-7B)	Bits	group-size	Wikitext2	PTB	C4
FP16	16	-	5.67	8.79	7.05
RTN	4	-	6.28	9.68	7.70
GPTQ	4	64	6.16	9.66	7.52
RTN	3	-	25.66	61.25	28.19
GPTQ	3	64	12.24	16.77	9.55

code: https://github.com/qwopqwop200/GPTQ-for-LLaMa

[oobabooga]

That's very interesting and promising @qwopqwop200. Do you think that this can be generalized to any model through some wrapper like this?

model  = AutoModelForCausalLM.from_pretrained(...)
model = convert_to_4bit(model)

output_ids = model.generate(input_ids)

[qwopqwop200]

I think it's difficult if the implementation of the model is not constant.
For example, OPT and bloom are mostly similar, but the architecture is different in some parts.
For example, in positional embedding, opt uses LearnedPositionalEmbedding, while Bloom uses ALiBi.
Due to these differences, some parts of the code may be different.
However, most of the code is the same. If you cope with these differences, I think you can be compatible with most(not all) Transformer architectures.

[oobabooga]

Thanks for the clarifications. If my 2 brain cells did the math right, 4-bit would allow llama-30b to be loaded with about 20GB VRAM. Having that in the web UI would be very nice.

[MetaIX]

I would love to see this.. imagine the possibilities. Also, does this work on windows?

Kept getting this error.

I assume this might be because I couldn't properly install the CUDA extension, as I was also met with this error.

[qwopqwop200]

I would love to see this.. imagine the possibilities. Also, does this work on windows?

Kept getting this error.

I assume this might be because I couldn't properly install the CUDA extension, as I was also met with this error.

I am currently experimenting on windows 11 and installed cuda kernel..
If you can't install it on Windows, you can also use wsl2.

[oobabooga]

Another question: I see no mention of temperature, top_p, top_k, etc in the code. Is it possible to use those parameters somehow?

[qwopqwop200]

My code is based on GPTQ and GPTQ only supports benchmark code for simplicity.
Therefore, you need to create a separate code for inference.
like this code

[musicurgy]

Already writing implementations for 4-bit, love it. How fast is the inference time when running llama 30B 4-bit on a 3090?

[oobabooga]

To be honest, it is not clear to me how to implement this because there is no inference code with some examples to follow. Also, without temperature, repetition_penalty, top_p and top_k (specifically those 4 parameters), the results would not be good. Maybe someone can help?

It seems like bitsandbytes will have int4 support soon huggingface/transformers#21955 (comment), but that will probably not be equivalent to GPTQ. Figure 1 in the paper shows a comparison between naive 4-bit quantization (which they call RTN, "round-to-nearest") and their approach, and it is clear that the difference is huge: https://arxiv.org/pdf/2210.17323.pdf

[aeryncaen]

I'm working on converting all the llama variants to 3-bit, keep an eye on the decapoda-research. I'll update here when they're available.

[oobabooga]

Super, @zoidbb!