[Feature]: Support Internlm2 Lora loading

[newportchen]

I tried to modify the source code to support Lora loading of the internlm2 model, load lora is fine, but inference result is not correct.

the specific modifications include:
1. add supported_lora_modules:

models/internlm2.py:
`class InternLM2ForCausalLM(nn.Module):
packed_modules_mapping = {
"wqkv":["wqkv"],
"gate_up_proj": [
"w1",
"w3",
],
}

# LoRA specific attributes
supported_lora_modules = [
    "wqkv",
    "wo",
    "gate_up_proj",
    "w2",
]
embedding_modules = {}
embedding_padding_modules = []`

2. add vocab_size 92544 support:
bgmv_config.h
f(in_T, out_T, W_T, narrow, 92544) \

I don't know where the problem is，some one can help me?

Alternatives

No response

Additional context

No response

[newportchen]

@jeejeelee

[jeejeelee]

@newportchen Your approach seems promising,I'm not familiar with internLM2, not sure about the reason for loadling wqkv

[hxhcreate]

When will InternLM2ForCausalLM support lora loading? eager waiting for that

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!