Skip to content

Commit b2064ea

Browse files
Jintao-HuangJintao-Huang
authored
Add openbuddy llama2 (#47)
1 parent d121f9b commit b2064ea

File tree

12 files changed

+46
-21
lines changed

12 files changed

+46
-21
lines changed

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ Key features:
3838
1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en
3939
2. agent: [damo-agent-zh](https://modelscope.cn/datasets/damo/MSAgent-Bench/summary), damo-agent-mini-zh
4040
3. multi-modal: coco-en
41-
5. supported templates: chatml(qwen), baichuan, chatglm2, llama, openbuddy_llama, default
41+
5. supported templates: chatml(qwen), baichuan, chatglm2, llama, openbuddy-llama, default
4242

4343
# Installation
4444

‎README_CN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ SWIFT（Scalable lightWeight Infrastructure for Fine-Tuning）是一个可扩展
3636
1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en
3737
2. agent: [damo-agent-zh](https://modelscope.cn/datasets/damo/MSAgent-Bench/summary), damo-agent-mini-zh
3838
3. multi-modal: coco-en
39-
5. 支持的对话模板: chatml(qwen), baichuan, chatglm2, llama, openbuddy_llama, default
39+
5. 支持的对话模板: chatml(qwen), baichuan, chatglm2, llama, openbuddy-llama, default
4040

4141
# 安装
4242

‎examples/pytorch/llm/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,13 +16,13 @@
1616

1717
## Features
1818
1. supported SFT methods: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), full(full parameter fine-tuning)
19-
2. supported models: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, polylm-13b
19+
2. supported models: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b
2020
3. supported features: quantization, ddp, model parallelism(device map), gradient checkpointing, gradient accumulation, pushing to modelscope hub, custom datasets, multimodal and agent SFT, mutli-round chat, ...
2121
4. supported datasets:
2222
1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en
2323
2. agent: [damo-agent-zh](https://modelscope.cn/datasets/damo/MSAgent-Bench/summary), damo-agent-mini-zh
2424
3. multi-modal: coco-en
25-
5. supported templates: chatml(qwen), baichuan, chatglm2, llama, openbuddy_llama, default
25+
5. supported templates: chatml(qwen), baichuan, chatglm2, llama, openbuddy-llama, default
2626

2727
## Prepare the Environment
2828
Experimental environment: A10, 3090, A100, ... (V100 does not support bf16, quantization)

‎examples/pytorch/llm/README_CN.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,13 @@
1717

1818
## 特性
1919
1. 支持的SFT方法: [lora](https://arxiv.org/abs/2106.09685), [qlora](https://arxiv.org/abs/2305.14314), 全参数微调
20-
2. 支持的模型: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, polylm-13b
20+
2. 支持的模型: qwen-7b, [qwen-7b-chat](https://github.com/QwenLM/Qwen-7B), qwen-vl, [qwen-vl-chat](https://github.com/QwenLM/Qwen-VL), baichuan-7b, baichuan-13b, baichuan-13b-chat, chatglm2-6b, chatglm2-6b-32k, llama2-7b, llama2-7b-chat, llama2-13b, llama2-13b-chat, llama2-70b, llama2-70b-chat, openbuddy-llama2-13b, openbuddy-llama-65b, openbuddy-llama2-70b, polylm-13b
2121
3. 支持的特性: 模型量化, DDP, 模型并行(device_map), gradient checkpointing, 梯度累加, 支持推送ModelScope Hub, 自定义数据集, 多模态和Agent SFT, 多轮对话, ...
2222
4. 支持的数据集:
2323
1. NLP: alpaca-en(gpt4), alpaca-zh(gpt4), finance-en, multi-alpaca-all, code-en, instinwild-en, instinwild-zh, cot-en, cot-zh, firefly-all-zh, poetry-zh, instruct-en, gpt4all-en
2424
2. agent: [damo-agent-zh](https://modelscope.cn/datasets/damo/MSAgent-Bench/summary), damo-agent-mini-zh
2525
3. multi-modal: coco-en
26-
5. 支持的对话模板: chatml(qwen), baichuan, chatglm2, llama, openbuddy_llama, default
26+
5. 支持的对话模板: chatml(qwen), baichuan, chatglm2, llama, openbuddy-llama, default
2727

2828
## 准备实验环境
2929
实验环境: A10, 3090, A100均可. (V100不支持bf16, 量化)

‎examples/pytorch/llm/scripts/openbuddy-llama-65b/qlora/infer.sh‎ renamed to ‎examples/pytorch/llm/scripts/openbuddy-llama2-70b/qlora/infer.sh‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
1-
# 38G
1+
# 40G
22
CUDA_VISIBLE_DEVICES=0,1 \
33
python src/llm_infer.py \
4-
--model_type openbuddy-llama-65b \
4+
--model_type openbuddy-llama2-70b \
55
--sft_type lora \
6-
--ckpt_dir "runs/openbuddy-llama-65b/vx_xxx/checkpoint-xxx" \
6+
--template_type openbuddy-llama \
7+
--dtype bf16 \
8+
--ckpt_dir "runs/openbuddy-llama2-70b/vx_xxx/checkpoint-xxx" \
79
--eval_human true \
810
--quantization_bit 4 \
11+
--bnb_4bit_comp_dtype bf16 \
912
--max_new_tokens 1024 \
1013
--temperature 0.9 \
1114
--top_k 50 \

‎examples/pytorch/llm/scripts/openbuddy-llama-65b/qlora/sft.sh‎ renamed to ‎examples/pytorch/llm/scripts/openbuddy-llama2-70b/qlora/sft.sh‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,32 @@
1-
# 40G
1+
# 42G VRAM
22
CUDA_VISIBLE_DEVICES=0,1 \
33
python src/llm_sft.py \
4-
--model_type openbuddy-llama-65b \
4+
--model_type openbuddy-llama2-70b \
55
--sft_type lora \
6+
--template_type openbuddy-llama \
7+
--dtype bf16 \
68
--output_dir runs \
79
--dataset alpaca-en,alpaca-zh \
810
--dataset_sample 20000 \
911
--num_train_epochs 1 \
1012
--max_length 1024 \
1113
--quantization_bit 4 \
14+
--bnb_4bit_comp_dtype bf16 \
1215
--lora_rank 8 \
1316
--lora_alpha 32 \
1417
--lora_dropout_p 0.1 \
18+
--gradient_checkpointing true \
1519
--batch_size 1 \
20+
--weight_decay 0. \
1621
--learning_rate 1e-4 \
1722
--gradient_accumulation_steps 16 \
23+
--max_grad_norm 0.5 \
24+
--warmup_ratio 0.03 \
1825
--eval_steps 50 \
1926
--save_steps 50 \
2027
--save_total_limit 2 \
2128
--logging_steps 10 \
29+
--push_to_hub false \
30+
--hub_model_id openbuddy-llama2-70b-qlora \
31+
--hub_private_repo true \
32+
--hub_token 'your-sdk-token' \

‎examples/pytorch/llm/src/utils/model.py‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -278,13 +278,19 @@ class LoRATM(NamedTuple):
278278
'openbuddy-llama2-13b': {
279279
'model_id': 'OpenBuddy/openbuddy-llama2-13b-v8.1-fp16',
280280
'revision': 'v1.0.0',
281-
'template': 'openbuddy_llama',
281+
'template': 'openbuddy-llama',
282282
'lora_TM': LoRATM.llama2,
283283
},
284284
'openbuddy-llama-65b': {
285285
'model_id': 'OpenBuddy/openbuddy-llama-65b-v8-bf16',
286286
'revision': 'v1.0.0',
287-
'template': 'openbuddy_llama',
287+
'template': 'openbuddy-llama',
288+
'lora_TM': LoRATM.llama2,
289+
},
290+
'openbuddy-llama2-70b': {
291+
'model_id': 'OpenBuddy/openbuddy-llama2-70b-v10.1-bf16',
292+
'revision': 'v1.0.0',
293+
'template': 'openbuddy-llama',
288294
'lora_TM': LoRATM.llama2,
289295
},
290296
'polylm-13b': {

‎examples/pytorch/llm/src/utils/preprocess.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@
4242
'chat_sep': [' ', ['eos_token_id', 'bos_token_id'], '[INST] '],
4343
'suffix': [['eos_token_id']],
4444
},
45-
'openbuddy_llama': {
45+
'openbuddy-llama': {
4646
'prefix': ['{{system}}\n\n'],
4747
'prompt': ['User: {{query}}\nAssistant: '],
4848
'chat_sep': ['\n'],

‎swift/trainers/mixin.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@
2626
from swift.utils.constants import Invoke
2727
from swift.utils.logger import get_logger
2828
from .utils import (can_return_loss, find_labels, get_function,
29-
is_instance_of_ms_Model)
29+
is_instance_of_ms_model)
3030

3131
logger = get_logger()
3232

@@ -229,7 +229,7 @@ def _save(self, output_dir: Optional[str] = None, state_dict=None):
229229
output_dir = output_dir if output_dir is not None else self.args.output_dir
230230
os.makedirs(output_dir, exist_ok=True)
231231
logger.info(f'Saving model checkpoint to {output_dir}')
232-
if is_instance_of_ms_Model(self.model):
232+
if is_instance_of_ms_model(self.model):
233233
model_dir = getattr(self.model, 'model_dir', None)
234234
if model_dir is not None:
235235
src_path = os.path.join(model_dir, 'configuration.json')
@@ -263,7 +263,7 @@ def _save(self, output_dir: Optional[str] = None, state_dict=None):
263263
else:
264264
torch.save(state_dict,
265265
os.path.join(output_dir, 'pytorch_model.bin'))
266-
elif is_instance_of_ms_Model(self.model):
266+
elif is_instance_of_ms_model(self.model):
267267
PreTrainedModel.save_pretrained(
268268
self.model,
269269
output_dir,

‎swift/trainers/utils.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ def get_function(
3737
return method_or_function
3838

3939

40-
def is_instance_of_ms_Model(model: Module) -> bool:
40+
def is_instance_of_ms_model(model: Module) -> bool:
4141
"""avoid import modelscope: circular dependency problem"""
4242
for m_cls in model.__class__.__mro__:
4343
cls_name = m_cls.__name__

0 commit comments

Comments
 (0)