📌 AutoAWQ Roadmap

[casper-hansen]

Optimization

• Fused layers of LLaMa models
• Implement GEMV kernel [NEW] GEMV kernel implementation #40
• Implement ExLlama kernels Exllama kernels support #313
• More fused layers for implemented models [NEW] GEMV kernel implementation #40
• INT8 quantization INT8 quantization support #45
• Optimize split_k_iters Create tuning section in quant_config #39

More models

• LLaMa 1/2
• MPT
• Falcon
• Bloom
• GPT-J
• OPT
• BERT Experiment with implementing AWQ for BERT models #4
• BigCode Implement BigCode models (StarCoder etc.) #5
• Support GPT-2 #49
• Support Baichuan #50
• Support Qwen #51
• Support InternLM #52
• GPT-NeoX Support for gpt-neox model #41

Ease of access

• Distribute PyPi package
• Re-add LLaVa model compatibility add llava model support #22
• Custom datasets to quantize models with Allow user to use custom calibration data for quantization #27
• Metal GPUs Implement metal kernel for GPUs on Mac #44
• ROCm GPUs AMD ROCM Support #315
• CPU implementation
• Push to hub functionality push_to_hub error #42

Software integration and quality

• Unit & integration testing Add unit/integration testing #31
• Integrate into Huggingface optimum/transformers
• Quantization config Create class QuantConfig  #8
• Model weight sharding and shard index Implement weight map and weight sharding #36

[flexchar]

Hey Casper, first of all, amazing work!

I'm actually really curious - what's the reasoning behind supporting legacy models such as GPT-2 or GPT-J/OPT that are already in?

In my perception, the latest developments mostly on MPT/Llama 2 are by orders of magnitude better than the legacy models.

[casper-hansen]

Hey Casper, first of all, amazing work!

I'm actually really curious - what's the reasoning behind supporting legacy models such as GPT-2 or GPT-J/OPT that are already in?

In my perception, the latest developments mostly on MPT/Llama 2 are by orders of magnitude better than the legacy models.

Supporting older models is on the roadmap because people still use those models and ask for them. However, I do try to focus my efforts on optimizing the newer models.

[heiqilin1985]

yi-34b 能支持吗？看数据这个模型很牛叉啊。

[casper-hansen]

yi-34b 能支持吗？看数据这个模型很牛叉啊。

Yi is now supported on the main branch

[SinanAkkoyun]

Can you please implement Phi 1.5 support? Thank you for all the amazing work!

[xTayEx]

Hi Casper, thank you for your wonderful work! I wonder if there is some tutorial for adding support for new model? I have noticed that Baichuan is on the roadmap. I would like try to add support for this model, could you please give me some pointer on how to support new model?

[casper-hansen]

@xTayEx I do not have a written guide, but here are the steps:

1. Create a model class BaichuanAWQForCausalLM
2. Add the model to the model map https://github.com/casper-hansen/AutoAWQ/blob/main/awq/models/auto.py#L6
3. Import the model here https://github.com/casper-hansen/AutoAWQ/blob/main/awq/models/__init__.py

For creating the model class, look into the llama class or other classes to see how they are defined.

[casper-hansen]

Can you please implement Phi 1.5 support? Thank you for all the amazing work!

Phi 1.5 support has been attempted, but they have a very unusual model definition. Until it's been standardized, I am not sure I will support it.

[SinanAkkoyun]

Phi 1.5 support has been attempted, but they have a very unusual model definition. Until it's been standardized, I am not sure I will support it.

Oh :( Do you mean until a new phi model comes out?
Phi 1.5 is such an amazing model for so many applications

What would roughly be the steps to implement it on our own?

[christian-ci]

Hi @casper-hansen First of all thank you for the Amazing work. From my understanding there is an AWQ TheBloke Mixtral 8x7b Base Instruct version. I tried to run inference on it and ran into issues. Would this model be supported? Also is there a way to contribute with a donation?

[casper-hansen]

We achieved most items on the roadmap, so closing this for now to focus on other things.