Can i quantize a 4Bit model more?

[Shreyas-ITB]

Hi i want to quantize a model which is already quantized to 4bit q4_1 but i want to make it compute faster so i wanted to ask what is the command to quantize the quantized module. I tried once with the command that is in the readme file but that didnt work. so can anyone help me?

[howard0su]

what command did you run? I think you can convert ggml to pth, then pth to ggml with q4_0.

[sgoll]

At the moment, I believe only 4-bit quantization has been implemented and is natively supported. You can find discussions about possibly supporting 2-bit quantization here (and 3-bit as a side note):

• Any interest in 2-bit quantizing the 65B model? #206
• 2-bit integer quantization  #456

There is a fork that has implemented 2-bit quantization with some success, see #456 (comment).

[prusnak]

The comment above explains the situation. One node: if there is a 2-bit or 3-bit quantization available, it should be always performed from the original f16 or f32 file.