[Bug]: AttributeError: 'Qwen2Model' object has no attribute 'rotary_emb'

[Alex-DeepL]

Your current environment

The output of `python collect_env.py`

Your output of `python collect_env.py` here

Model Input Dumps

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = '/home/root123/workspace/model/qwen2-0-5/'

quant_path = '/home/root123/workspace/model/qwen2-0-5-awq-4/'

quant_config = { "zero_point": True, "q_group_size": 128, "w_bit": 4, "version": "GEMM" }

Load model

model = AutoAWQForCausalLM.from_pretrained(
model_path, **{"low_cpu_mem_usage": True, "use_cache": False}
)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

Quantize

model.quantize(tokenizer, quant_config=quant_config)

Save quantized model

model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized at "{quant_path}"')

🐛 Describe the bug

在进行awq模型量化时报错

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DaBossCoda]

im getting same thing but i think this is a bug AutoAwq

[Alex-DeepL]

bro,How did you solve this problem?

…

------------------ 原始邮件 ------------------ 发件人: ***@***.***>; 发送时间: 2024年11月30日(星期六) 中午11:18 收件人: ***@***.***>; 抄送: ***@***.***>; ***@***.***>; 主题: Re: [vllm-project/vllm] [Bug]: AttributeError: 'Qwen2Model' object has no attribute 'rotary_emb' (Issue #10773) im getting same thing but i think this is a bug AutoAwq — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[ArlanCooper]

+1

[casper-hansen]

This is a bug in the weight saving method provided by Huggingface. This is being tracked in Transformers below and will be fixed in the future - until then, recommend to use v0.2.6 of AutoAWQ or patch your model like this: casper-hansen/AutoAWQ#665 (comment)

huggingface/transformers#35080

[casper-hansen]

This issue can now be closed as it has been resolved upstream in huggingface_hub and now autoawq.