Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Fixes & updates on lora, qlora scripts and hf_decoder_model #787

Merged
merged 15 commits into from
Apr 24, 2024
Merged

Fixes & updates on lora, qlora scripts and hf_decoder_model #787

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f902549
Argument order change, grouping relative args
wheresmyhair Apr 23, 2024
9e78b26
Add conversation template
wheresmyhair Apr 23, 2024
1ebcfa9
Add conversation template to dataset fingerprint
wheresmyhair Apr 23, 2024
806ce87
Add conversation template to dataset fingerprint
wheresmyhair Apr 23, 2024
b6ef108
Qlora model loading args bug fix
wheresmyhair Apr 23, 2024
fca0121
Update default dataset and conversation template for lora and qlora s…
wheresmyhair Apr 23, 2024
434a932
Argument order change, grouping relative args
wheresmyhair Apr 23, 2024
28d968c
Add conversation template
wheresmyhair Apr 23, 2024
1bcbf7f
Add conversation template to dataset fingerprint
wheresmyhair Apr 23, 2024
62cd498
Add conversation template to dataset fingerprint
wheresmyhair Apr 23, 2024
59e0cd5
Qlora model loading args bug fix
wheresmyhair Apr 23, 2024
134a4ab
Update default dataset and conversation template for lora and qlora s…
wheresmyhair Apr 23, 2024
77b227a
Merge branch 'yizhenjia-conversation-template-dev' of https://github.…
wheresmyhair Apr 24, 2024
0b5fb62
Revert "Argument order change, grouping relative args"
wheresmyhair Apr 24, 2024
8826fa7
Revert finetune with lisa shell
wheresmyhair Apr 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion scripts/run_finetune_with_lora.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@

# Parses arguments
model_name_or_path=gpt2
dataset_path=data/alpaca/train
dataset_path=data/alpaca/train_conversation
conversation_template=llama2
output_dir=output_models/finetune
deepspeed_args="--master_port=11000"

Expand All @@ -18,6 +19,10 @@ while [[ $# -ge 1 ]]; do
dataset_path="$2"
shift
;;
--conversation_template)
conversation_template="$2"
shift
;;
-o|--output_lora_path)
output_dir="$2"
shift
Expand All @@ -43,6 +48,7 @@ deepspeed ${deepspeed_args} \
examples/finetune.py \
--model_name_or_path ${model_name_or_path} \
--dataset_path ${dataset_path} \
--conversation_template ${conversation_template} \
--output_dir ${output_dir} --overwrite_output_dir \
--num_train_epochs 0.01 \
--learning_rate 1e-4 \
Expand Down
8 changes: 7 additions & 1 deletion scripts/run_finetune_with_lora_save_aggregated_weights.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@

# Parses arguments
model_name_or_path=gpt2
dataset_path=data/alpaca/train
dataset_path=data/alpaca/train_conversation
conversation_template=llama2
output_dir=output_models/finetune
deepspeed_args="--master_port=11000"

Expand All @@ -18,6 +19,10 @@ while [[ $# -ge 1 ]]; do
dataset_path="$2"
shift
;;
--conversation_template)
conversation_template="$2"
shift
;;
-o|--output_model_path)
output_dir="$2"
shift
Expand All @@ -43,6 +48,7 @@ deepspeed ${deepspeed_args} \
examples/finetune.py \
--model_name_or_path ${model_name_or_path} \
--dataset_path ${dataset_path} \
--conversation_template ${conversation_template} \
--output_dir ${output_dir} --overwrite_output_dir \
--num_train_epochs 0.01 \
--learning_rate 1e-4 \
Expand Down
8 changes: 7 additions & 1 deletion scripts/run_finetune_with_qlora_save_aggregated_weights.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@

# Parses arguments
model_name_or_path=meta-llama/Llama-2-13b-hf
dataset_path=/home/paperspace/LMFlow/alpaca/train
dataset_path=data/alpaca/train_conversation
conversation_template=llama2
output_dir=output_models/finetune
deepspeed_args="--master_port=11000"

Expand All @@ -18,6 +19,10 @@ while [[ $# -ge 1 ]]; do
dataset_path="$2"
shift
;;
--conversation_template)
conversation_template="$2"
shift
;;
-o|--output_model_path)
output_dir="$2"
shift
Expand All @@ -43,6 +48,7 @@ deepspeed ${deepspeed_args} \
examples/finetune.py \
--model_name_or_path ${model_name_or_path} \
--dataset_path ${dataset_path} \
--conversation_template ${conversation_template} \
--output_dir ${output_dir} --overwrite_output_dir \
--num_train_epochs 0.01 \
--learning_rate 1e-4 \
Expand Down
3 changes: 2 additions & 1 deletion src/lmflow/models/hf_decoder_model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,6 +631,7 @@ def tokenize_function(examples):
(
fingerprint
+ str(self.tokenizer)
+ str(conversation_template) if dataset_type == "conversation" else ""
+ f'###disable_group_texts={data_args.disable_group_texts}'
+ f'###block_size={data_args.block_size}'
).encode("utf-8")
Expand Down Expand Up @@ -818,7 +819,7 @@ def get_peft_without_qlora(self):
torch_dtype=torch_dtype,
device_map=device_map,
trust_remote_code = self.model_args.trust_remote_code,
attn_implementation="flash_attention_2" if model_args.use_flash_attention else None,
attn_implementation="flash_attention_2" if self.model_args.use_flash_attention else None,
)

self.backend_model = PeftModel.from_pretrained(self.backend_model_full, tmpdirname)
Expand Down
Loading