Support peft 0.11.0 (#953)

Author: tastelikefeet

README.md

@@ -41,6 +41,7 @@ Additionally, we are expanding capabilities for other modalities. Currently, we
 SWIFT has rich documentations for users, please check [here](https://github.com/modelscope/swift/tree/main/docs/source_en/LLM).
 
 ## 🎉 News
+- 🔥2024.05.17: Support peft=0.11.0. Meanwhile support 3 new tuners: `BOFT`, `Vera` and `Pissa`. use `--sft_type boft/vera` to use BOFT or Vera, use `--init_lora_weights pissa` with `--sft_type lora` to use Pissa.
 - 2024.05.16: Supports Llava-Next (Stronger) series models. For best practice, you can refer to [here](https://github.com/modelscope/swift/tree/main/docs/source_en/Multi-Modal/llava-best-practice.md).
 - 🔥2024.05.13: Support Yi-1.5 series models，use `--model_type yi-1_5-9b-chat` to begin!
 - 2024.05.11: Support for qlora training and quantized inference using [hqq](https://github.com/mobiusml/hqq) and [eetq](https://github.com/NetEase-FuXi/EETQ). For more information, see the [LLM Quantization Documentation](https://github.com/modelscope/swift/tree/main/docs/source_en/LLM/LLM-quantization.md).

README_CN.md

@@ -42,6 +42,7 @@ SWIFT支持近**200种LLM和MLLM**（多模态大模型）的训练、推理、
 SWIFT具有丰富的文档体系，如有使用问题请请查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/LLM).
 
 ## 🎉 新闻
+- 🔥2024.05.17: 支持peft=0.11.0. 同时支持了三个新的tuner方法： `BOFT`, `Vera` 和 `Pissa`. 使用 `--sft_type boft/vera` 开启BOFT或者Vera, 使用 `--init_lora_weights pissa` 以及 `--sft_type lora` 来使用 Pissa.
 - 2024.05.16: 支持Llava-Next (Stronger)系列模型，最佳实践可以查看[这里](https://github.com/modelscope/swift/tree/main/docs/source/Multi-Modal/llava最佳实践.md).
 - 🔥2024.05.13: 支持Yi-1.5系列模型，使用`--model_type yi-1_5-9b-chat`等开始体验
 - 2024.05.11: 支持使用[hqq](https://github.com/mobiusml/hqq)和[eetq](https://github.com/NetEase-FuXi/EETQ)进行qlora训练和量化推理，可以查看[LLM量化文档](https://github.com/modelscope/swift/tree/main/docs/source/LLM/LLM量化文档.md)

docs/source/LLM/命令行参数.md

@@ -63,6 +63,7 @@
 - `--lora_rank`: 默认为`8`. 只有当`sft_type`指定为'lora'时才生效.
 - `--lora_alpha`: 默认为`32`. 只有当`sft_type`指定为'lora'时才生效.
 - `--lora_dropout_p`: 默认为`0.05`, 只有当`sft_type`指定为'lora'时才生效.
+- `--init_lora_weights`: 初始化LoRA weights的方法, 可以指定为`true`, `false`, `guassian`, `pissa`, `pissa_niter_[number of iters]`, 默认值`true`.
 - `--lora_bias_trainable`: 默认为`'none'`, 可以选择的值: 'none', 'all'. 如果你要将bias全都设置为可训练, 你可以设置为`'all'`.
 - `--lora_modules_to_save`: 默认为`[]`. 如果你想要训练embedding, lm_head, 或者layer_norm, 你可以设置此参数, 例如: `--lora_modules_to_save EMBEDDING LN lm_head`. 如果传入`'EMBEDDING'`, 则将Embedding层添加到`lora_modules_to_save`. 如果传入`'LN'`, 则将`RMSNorm`和`LayerNorm`添加到`lora_modules_to_save`.
 - `--lora_dtype`: 默认为`'AUTO'`, 指定lora模块的dtype类型. 如果是`AUTO`则跟随原始模块的dtype类型. 你可以选择的值: 'fp16', 'bf16', 'fp32', 'AUTO'.
@@ -136,6 +137,23 @@
 
 - `--sequence_parallel_size`: 默认值`1`, 大于1时可以拆分一个sequence到多张显卡上以节省显存, 值需要设置为能被DDP数量整除
 
+### BOFT 参数
+
+- `--boft_block_size`: BOFT块尺寸, 默认值4.
+- `--boft_block_num`: BOFT块数量, 不能和`boft_block_size`同时使用.
+- `--boft_target_modules`: BOFT目标模块. 默认为`['DEFAULT']`. 如果boft_target_modules传入`'DEFAULT'` or `'AUTO'`, 则根据`model_type`查找`MODEL_MAPPING`中的boft_target_modules`(默认指定为qkv). 如果传入`'ALL'`, 则将所有的Linear层(不含head)指定为boft模块.
+- `--boft_dropout`: boft的dropout值, 默认0.0.
+- `--boft_modules_to_save`: 需要额外训练和存储的模块, 默认为`None`.
+
+### Vera参数
+
+- `--vera_rank`: Vera Attention的尺寸, 默认值256.
+- `--vera_projection_prng_key`: 是否存储Vera映射矩阵, 默认为True.
+- `--vera_target_modules`: Vera目标模块. 默认为`['DEFAULT']`. 如果vera_target_modules传入`'DEFAULT'` or `'AUTO'`, 则根据`model_type`查找`MODEL_MAPPING`中的vera_target_modules`(默认指定为qkv). 如果传入`'ALL'`, 则将所有的Linear层(不含head)指定为vera模块.
+- `--vera_dropout`: Vera的dropout值, 默认`0.0`.
+- `--vera_d_initial`: Vera的d矩阵的初始值, 默认`0.1`.
+- `--vera_modules_to_save`: 需要额外训练和存储的模块, 默认为`None`.
+
 ### LoRA+微调参数
 
 - `--lora_lr_ratio`: 默认值`None`, 建议值`10~16`, 使用lora时指定该参数即可使用lora+.

docs/source_en/LLM/Command-line-parameters.md

@@ -63,6 +63,7 @@
 - `--lora_rank`: Default is `8`. Only takes effect when `sft_type` is 'lora'.
 - `--lora_alpha`: Default is `32`. Only takes effect when `sft_type` is 'lora'.
 - `--lora_dropout_p`: Default is `0.05`, only takes effect when `sft_type` is 'lora'.
+- `--init_lora_weights`: Method to initialize LoRA weights, can be specified as `true`, `false`, `gaussian`, `pissa`, or `pissa_niter_[number of iters]`. Default value `true`.
 - `--lora_bias_trainable`: Default is `'none'`, options: 'none', 'all'. Set to `'all'` to make all biases trainable.
 - `--lora_modules_to_save`: Default is `[]`. If you want to train embedding, lm_head, or layer_norm, you can set this parameter, e.g. `--lora_modules_to_save EMBEDDING LN lm_head`. If passed `'EMBEDDING'`, Embedding layer will be added to `lora_modules_to_save`. If passed `'LN'`, `RMSNorm` and `LayerNorm` will be added to `lora_modules_to_save`.
 - `--lora_dtype`: Default is `'AUTO'`, specifies dtype for lora modules. If `AUTO`, follow dtype of original module. Options: 'fp16', 'bf16', 'fp32', 'AUTO'.
@@ -135,6 +136,23 @@
 
 - `--sequence_parallel_size`: Default value `1`, a positive value can be used to split a sequence to multiple GPU to reduce memory usage. The value should divide the GPU count.
 
+### BOFT Parameters
+
+- `--boft_block_size`: BOFT block size, default value is 4.
+- `--boft_block_num`: Number of BOFT blocks, cannot be used simultaneously with `boft_block_size`.
+- `--boft_target_modules`: BOFT target modules. Default is `['DEFAULT']`. If `boft_target_modules` is set to `'DEFAULT'` or `'AUTO'`, it will look up `boft_target_modules` in the `MODEL_MAPPING` based on `model_type` (default specified as qkv). If set to `'ALL'`, all Linear layers (excluding the head) will be designated as BOFT modules.
+- `--boft_dropout`: Dropout value for BOFT, default is 0.0.
+- `--boft_modules_to_save`: Additional modules to be trained and saved, default is `None`.
+
+### Vera Parameters
+
+- `--vera_rank`: Size of Vera Attention, default value is 256.
+- `--vera_projection_prng_key`: Whether to store the Vera projection matrix, default is True.
+- `--vera_target_modules`: Vera target modules. Default is `['DEFAULT']`. If `vera_target_modules` is set to `'DEFAULT'` or `'AUTO'`, it will look up `vera_target_modules` in the `MODEL_MAPPING` based on `model_type` (default specified as qkv). If set to `'ALL'`, all Linear layers (excluding the head) will be designated as Vera modules. Vera modules need to share a same shape.
+- `--vera_dropout`: Dropout value for Vera, default is 0.0.
+- `--vera_d_initial`: Initial value for Vera's d matrix, default is 0.1.
+- `--vera_modules_to_save`: Additional modules to be trained and saved, default is `None`.
+
 ### LoRA+ Fine-tuning Parameters
 
 - `--lora_lr_ratio`: Default `None`, recommended value `10~16`, specify this parameter when using lora to enable lora+.

requirements/framework.txt

@@ -8,7 +8,7 @@ nltk
 numpy
 optimum>=1.17.0
 pandas
-peft>=0.9.0,<0.11.0
+peft>=0.11.0,<0.12.0
 requests
 rouge
 safetensors

swift/llm/tuner.py

@@ -9,8 +9,8 @@
 
 from swift.torchacc_utils import consolidate_checkpoint
 from swift.trainers import TrainerCallback
-from swift.tuners import (AdaLoraConfig, AdapterConfig, IA3Config, LongLoRAModelType, LoraConfig, LoRAConfig,
-                          NEFTuneConfig, Swift)
+from swift.tuners import (AdaLoraConfig, AdapterConfig, BOFTConfig, IA3Config, LongLoRAModelType, LoraConfig,
+                          LoRAConfig, NEFTuneConfig, Swift, VeraConfig)
 from swift.tuners.llamapro import LLaMAProConfig
 from swift.tuners.module_mapping import MODEL_KEYS_MAPPING
 from swift.utils import activate_model_parameters, freeze_model_parameters, get_logger, use_torchacc
@@ -24,6 +24,10 @@ def handle_target_modules(model, args: SftArguments) -> None:
         target_modules = args.ia3_target_modules
         assert len(args.ia3_feedforward_modules) > 0, ('Setting ia3_target_modules to `ALL` '
                                                        'need to pass MLP linear names to `ia3_feedforward_modules`')
+    elif args.sft_type == 'vera':
+        target_modules = args.vera_target_modules
+    elif args.sft_type == 'boft':
+        target_modules = args.boft_target_modules
     else:
         target_modules = args.lora_target_modules
     if args.lora_use_embedding:
@@ -33,14 +37,43 @@ def handle_target_modules(model, args: SftArguments) -> None:
     if args.sft_type == 'ia3':
         args.ia3_target_modules = target_modules
         logger.info(f'ia3_target_modules: {args.ia3_target_modules}')
+    elif args.sft_type == 'vera':
+        args.vera_target_modules = target_modules
+        logger.info(f'vera_target_modules: {args.ia3_target_modules}')
+    elif args.sft_type == 'boft':
+        args.boft_target_modules = target_modules
+        logger.info(f'boft_target_modules: {args.boft_target_modules}')
     else:
         args.lora_target_modules = target_modules
         logger.info(f'lora_target_modules: {args.lora_target_modules}')
 
 
+def handle_same_dim_target_modules(model: torch.nn.Module, config: VeraConfig):
+    target_modules = config.target_modules
+    modules_dict = {
+        name: module.weight.shape
+        for name, module in model.named_modules()
+        if isinstance(module, torch.nn.Linear) and any([t in name for t in target_modules])
+    }  # only Linear for now
+    if len(set(modules_dict.values())) > 1:
+        v = [t for t in target_modules if 'v' in t]
+        if not v:
+            raise ValueError('Please manually pass in `vera_target_modules`, do not use `DEFAULT` or `ALL`,'
+                             'because Vera need all target linears to be the same size.')
+        v = v[0]
+        shape = [shape for name, shape in modules_dict.items() if v in name][0]
+        names = [_name for _name, _shape in modules_dict.items() if _shape == shape]
+        config.target_modules = [t for t in target_modules if any([t in name for name in names])]
+    return config
+
+
 def handle_modules_to_save(model, args: SftArguments) -> None:
     if args.sft_type == 'ia3':
         modules_to_save = args.ia3_modules_to_save
+    elif args.sft_type == 'vera':
+        modules_to_save = args.vera_modules_to_save
+    elif args.sft_type == 'boft':
+        modules_to_save = args.boft_modules_to_save
     else:
         modules_to_save = args.lora_modules_to_save
     if args.lora_m2s_use_embedding:
@@ -51,6 +84,12 @@ def handle_modules_to_save(model, args: SftArguments) -> None:
     if args.sft_type == 'ia3':
         args.ia3_modules_to_save = modules_to_save
         logger.info(f'ia3_modules_to_save: {args.ia3_modules_to_save}')
+    elif args.sft_type == 'vera':
+        args.vera_modules_to_save = modules_to_save
+        logger.info(f'vera_modules_to_save: {args.vera_modules_to_save}')
+    elif args.sft_type == 'boft':
+        args.boft_modules_to_save = modules_to_save
+        logger.info(f'boft_modules_to_save: {args.boft_modules_to_save}')
     else:
         args.lora_modules_to_save = modules_to_save
         logger.info(f'lora_modules_to_save: {args.lora_modules_to_save}')
@@ -62,6 +101,8 @@ def prepare_model(model, args: SftArguments):
         if args.resume_from_checkpoint is None:
             handle_target_modules(model, args)
             handle_modules_to_save(model, args)
+            if args.init_lora_weights and args.init_lora_weights.lower() in ('true', 'false'):
+                args.init_lora_weights = args.init_lora_weights.lower() in ('true', 'True')
             lora_kwargs = {
                 'r': args.lora_rank,
                 'target_modules': args.lora_target_modules,
@@ -72,6 +113,7 @@ def prepare_model(model, args: SftArguments):
                 'use_rslora': args.use_rslora,
                 'use_dora': args.use_dora,
                 'lorap_lr_ratio': args.lora_lr_ratio,
+                'init_lora_weights': args.init_lora_weights,
             }
             if args.sft_type in ('lora', 'longlora'):
                 if args.lora_dtype == 'AUTO':
@@ -158,6 +200,29 @@ def prepare_model(model, args: SftArguments):
                     act_layer=args.adapter_act)
                 model = Swift.prepare_model(model, adapter_config)
                 logger.info(f'adapter_config: {adapter_config}')
+            elif args.sft_type == 'vera':
+                vera_config = VeraConfig(
+                    r=args.vera_rank,
+                    target_modules=args.vera_target_modules,
+                    projection_prng_key=args.vera_projection_prng_key,
+                    vera_dropout=args.vera_dropout,
+                    d_initial=args.vera_d_initial,
+                    modules_to_save=args.vera_modules_to_save,
+                )
+                vera_config = handle_same_dim_target_modules(model, vera_config)
+                model = Swift.prepare_model(model, vera_config)
+                logger.info(f'vera_config: {vera_config}')
+            elif args.sft_type == 'boft':
+                boft_config = BOFTConfig(
+                    boft_block_size=args.boft_block_size,
+                    boft_block_num=args.boft_block_num,
+                    boft_n_butterfly_factor=args.boft_n_butterfly_factor,
+                    target_modules=args.boft_target_modules,
+                    boft_dropout=args.boft_dropout,
+                    modules_to_save=args.boft_modules_to_save,
+                )
+                model = Swift.prepare_model(model, boft_config)
+                logger.info(f'boft_config: {boft_config}')
         else:
             if use_torchacc():
                 consolidate_checkpoint(args.resume_from_checkpoint, 'adapter_model')

swift/llm/utils/argument.py

@@ -32,7 +32,7 @@
 
 
 def is_adapter(sft_type: str) -> bool:
-    return sft_type in {'lora', 'longlora', 'adalora', 'ia3', 'llamapro', 'adapter'}
+    return sft_type in {'lora', 'longlora', 'adalora', 'ia3', 'llamapro', 'adapter', 'vera', 'boft'}
 
 
 class ArgumentsBase:
@@ -404,7 +404,7 @@ class SftArguments(ArgumentsBase):
         default=None,
         metadata={'help': "Decoder Class name of model, e.g. 'QWenBlock' for QWen, 'LlamaDecoderLayer' for LLama"})
 
-    sft_type: Literal['lora', 'full', 'longlora', 'adalora', 'ia3', 'llamapro', 'adapter'] = 'lora'
+    sft_type: Literal['lora', 'full', 'longlora', 'adalora', 'ia3', 'llamapro', 'adapter', 'vera', 'boft'] = 'lora'
     freeze_parameters: float = 0.  # 0 ~ 1
     additional_trainable_parameters: List[str] = field(default_factory=list)
     tuner_backend: Literal['swift', 'peft', 'unsloth'] = 'peft'
@@ -457,6 +457,23 @@ class SftArguments(ArgumentsBase):
     lora_lr_ratio: float = None
     use_rslora: bool = False
     use_dora: bool = False
+    init_lora_weights: Literal['gaussian', 'pissa', 'pissa_niter_[number of iters]', 'loftq', 'true', 'false'] = 'true'
+
+    # BOFT
+    boft_block_size: int = 4
+    boft_block_num: int = 0
+    boft_n_butterfly_factor: int = 1
+    boft_target_modules: Optional[Union[List[str], str]] = field(default_factory=lambda: ['DEFAULT'])
+    boft_dropout: float = 0.0
+    boft_modules_to_save: List[str] = field(default_factory=list)
+
+    # Vera
+    vera_rank: int = 256
+    vera_target_modules: Optional[Union[List[str], str]] = field(default_factory=lambda: ['DEFAULT'])
+    vera_projection_prng_key: int = 0
+    vera_dropout: float = 0.0
+    vera_d_initial: float = 0.1
+    vera_modules_to_save: List[str] = field(default_factory=list)
 
     # adapter
     adapter_act: str = 'gelu'
@@ -684,6 +701,12 @@ def __post_init__(self) -> None:
             self.ia3_feedforward_modules = self._prepare_target_modules(self.ia3_feedforward_modules)
             self.ia3_target_modules = self._prepare_target_modules(self.ia3_target_modules)
             self.ia3_modules_to_save = self._prepare_modules_to_save(self.ia3_modules_to_save)
+        elif self.sft_type == 'vera':
+            self.vera_target_modules = self._prepare_target_modules(self.vera_target_modules)
+            self.vera_modules_to_save = self._prepare_modules_to_save(self.vera_modules_to_save)
+        elif self.sft_type == 'boft':
+            self.boft_target_modules = self._prepare_target_modules(self.boft_target_modules)
+            self.boft_modules_to_save = self._prepare_modules_to_save(self.boft_modules_to_save)
         else:
             self.lora_target_modules = self._prepare_target_modules(self.lora_target_modules)
             self.lora_modules_to_save = self._prepare_modules_to_save(self.lora_modules_to_save)
@@ -926,7 +949,7 @@ class InferArguments(ArgumentsBase):
     model_id_or_path: Optional[str] = None
     model_revision: Optional[str] = None
 
-    sft_type: Literal['lora', 'longlora', 'full', 'adalora', 'ia3', 'llamapro'] = 'lora'
+    sft_type: Literal['lora', 'longlora', 'full', 'adalora', 'ia3', 'llamapro', 'vera', 'boft'] = 'lora'
     template_type: str = field(
         default='AUTO', metadata={'help': f"template_type choices: {list(TEMPLATE_MAPPING.keys()) + ['AUTO']}"})
     infer_backend: Literal['AUTO', 'vllm', 'pt'] = 'AUTO'

swift/trainers/mixin.py

@@ -11,6 +11,7 @@
 
 import json
 import numpy as np
+import peft
 import safetensors
 import torch
 import transformers
@@ -250,6 +251,8 @@ def __init__(self,
             optimizers=optimizers,
             preprocess_logits_for_metrics=preprocess_logits_for_metrics,
             **kwargs)
+        if not self.label_names:
+            self.label_names = ['labels']
         if is_quantized and use_swift:
             model._hf_peft_config_loaded = _hf_peft_config_loaded
 
@@ -381,7 +384,7 @@ def _save(self, output_dir: Optional[str] = None, state_dict=None):
         from swift import SWIFT_MAPPING
         addtional_module_tuners = [
             name.lower() for name, (config, cls) in SWIFT_MAPPING.items() if cls.has_additional_modules()
-        ]
+        ] + list(peft.PEFT_TYPE_TO_CONFIG_MAPPING.keys())
         if self.tokenizer is not None and sft_args.sft_type not in addtional_module_tuners:
             self.tokenizer.save_pretrained(output_dir)
         # training_args.bin