[megatron] Support ovis2.5 (#5719)

Author: Jintao-Huang

docs/source/Instruction/命令行参数.md

@@ -171,7 +171,6 @@
 - enable_dft_loss: 是否在SFT训练中使用[DFT](https://arxiv.org/abs/2508.05629) (Dynamic Fine-Tuning) loss，默认为False。
 - enable_channel_loss: 打开channel loss，默认为`False`。你需要在数据集中准备"channel"字段，ms-swift会根据该字段分组统计loss。数据集格式参考[channel loss](../Customization/自定义数据集.md#channel-loss)。channel loss兼容packing/padding_free/loss_scale等技术。
   - 注意：该参数为"ms-swift>=3.8"新增，若要在"ms-swift<3.8"使用channel loss，请查看v3.7文档。
-  - 注意：该功能暂不兼容序列并行，待修复。
 - logging_dir: tensorboard日志路径。默认为None，即设置为`f'{self.output_dir}/runs'`。
 - predict_with_generate: 验证时使用生成式的方式，默认为False。
 - metric_for_best_model: 默认为None，即当`predict_with_generate`设置为False时，设置为'loss'，否则设置为'rouge-l'（在PPO训练时，不进行默认值设置；GRPO训练设置为'reward'）。

docs/source/Instruction/支持的模型和数据集.md

@@ -701,8 +701,8 @@
 |[AIDC-AI/Ovis2-8B](https://modelscope.cn/models/AIDC-AI/Ovis2-8B)|ovis2|ovis2|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2-8B](https://huggingface.co/AIDC-AI/Ovis2-8B)|
 |[AIDC-AI/Ovis2-16B](https://modelscope.cn/models/AIDC-AI/Ovis2-16B)|ovis2|ovis2|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2-16B](https://huggingface.co/AIDC-AI/Ovis2-16B)|
 |[AIDC-AI/Ovis2-34B](https://modelscope.cn/models/AIDC-AI/Ovis2-34B)|ovis2|ovis2|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2-34B](https://huggingface.co/AIDC-AI/Ovis2-34B)|
-|[AIDC-AI/Ovis2.5-2B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-2B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2.5-2B](https://huggingface.co/AIDC-AI/Ovis2.5-2B)|
-|[AIDC-AI/Ovis2.5-9B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-9B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2.5-9B](https://huggingface.co/AIDC-AI/Ovis2.5-9B)|
+|[AIDC-AI/Ovis2.5-2B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-2B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2714;|vision|[AIDC-AI/Ovis2.5-2B](https://huggingface.co/AIDC-AI/Ovis2.5-2B)|
+|[AIDC-AI/Ovis2.5-9B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-9B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2714;|vision|[AIDC-AI/Ovis2.5-9B](https://huggingface.co/AIDC-AI/Ovis2.5-9B)|
 |[XiaomiMiMo/MiMo-VL-7B-SFT](https://modelscope.cn/models/XiaomiMiMo/MiMo-VL-7B-SFT)|mimo_vl|mimo_vl|transformers>=4.49, qwen_vl_utils>=0.0.6, decord|&#x2718;|vision, video|[XiaomiMiMo/MiMo-VL-7B-SFT](https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-SFT)|
 |[XiaomiMiMo/MiMo-VL-7B-RL](https://modelscope.cn/models/XiaomiMiMo/MiMo-VL-7B-RL)|mimo_vl|mimo_vl|transformers>=4.49, qwen_vl_utils>=0.0.6, decord|&#x2718;|vision, video|[XiaomiMiMo/MiMo-VL-7B-RL](https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-RL)|
 |[mispeech/midashenglm-7b](https://modelscope.cn/models/mispeech/midashenglm-7b)|midashenglm|midashenglm|transformers>=4.52, soundfile|&#x2718;|audio|[mispeech/midashenglm-7b](https://huggingface.co/mispeech/midashenglm-7b)|

docs/source_en/Instruction/Command-line-parameters.md

@@ -174,7 +174,6 @@ This parameter list inherits from transformers `Seq2SeqTrainingArguments`, with
 - enable_dft_loss: Whether to use [DFT](https://arxiv.org/abs/2508.05629) (Dynamic Fine-Tuning) loss in SFT training, default is False.
 - enable_channel_loss: Enable channel loss, default is `false`. You need to prepare a "channel" field in your dataset; ms-swift will compute and aggregate the loss grouped by this field. For dataset format, please refer to [channel loss](../Customization/Custom-dataset.md#channel-loss). Channel loss is compatible with techniques such as packing, padding-free, and loss scaling.
   - Note: This parameter is newly added in "ms-swift>=3.8". If you want to use channel loss in "ms-swift<3.8", please refer to the v3.7 documentation.
-  - Note: This feature is currently not compatible with sequence parallelism and will be fixed later.
 - logging_dir: The path for TensorBoard logs. Defaults to None, which means it is set to `f'{self.output_dir}/runs'`.
 - predict_with_generate: Whether to use generative method during validation, default is False.
 - metric_for_best_model: Default is None, which means that when predict_with_generate is set to False, it is set to 'loss'; otherwise, it is set to 'rouge-l' (during PPO training, the default value is not set; in GRPO training, it is set to 'reward').

docs/source_en/Instruction/Supported-models-and-datasets.md

@@ -701,8 +701,8 @@ The table below introduces the models integrated with ms-swift:
 |[AIDC-AI/Ovis2-8B](https://modelscope.cn/models/AIDC-AI/Ovis2-8B)|ovis2|ovis2|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2-8B](https://huggingface.co/AIDC-AI/Ovis2-8B)|
 |[AIDC-AI/Ovis2-16B](https://modelscope.cn/models/AIDC-AI/Ovis2-16B)|ovis2|ovis2|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2-16B](https://huggingface.co/AIDC-AI/Ovis2-16B)|
 |[AIDC-AI/Ovis2-34B](https://modelscope.cn/models/AIDC-AI/Ovis2-34B)|ovis2|ovis2|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2-34B](https://huggingface.co/AIDC-AI/Ovis2-34B)|
-|[AIDC-AI/Ovis2.5-2B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-2B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2.5-2B](https://huggingface.co/AIDC-AI/Ovis2.5-2B)|
-|[AIDC-AI/Ovis2.5-9B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-9B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2718;|vision|[AIDC-AI/Ovis2.5-9B](https://huggingface.co/AIDC-AI/Ovis2.5-9B)|
+|[AIDC-AI/Ovis2.5-2B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-2B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2714;|vision|[AIDC-AI/Ovis2.5-2B](https://huggingface.co/AIDC-AI/Ovis2.5-2B)|
+|[AIDC-AI/Ovis2.5-9B](https://modelscope.cn/models/AIDC-AI/Ovis2.5-9B)|ovis2_5|ovis2_5|transformers>=4.46.2, moviepy<2|&#x2714;|vision|[AIDC-AI/Ovis2.5-9B](https://huggingface.co/AIDC-AI/Ovis2.5-9B)|
 |[XiaomiMiMo/MiMo-VL-7B-SFT](https://modelscope.cn/models/XiaomiMiMo/MiMo-VL-7B-SFT)|mimo_vl|mimo_vl|transformers>=4.49, qwen_vl_utils>=0.0.6, decord|&#x2718;|vision, video|[XiaomiMiMo/MiMo-VL-7B-SFT](https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-SFT)|
 |[XiaomiMiMo/MiMo-VL-7B-RL](https://modelscope.cn/models/XiaomiMiMo/MiMo-VL-7B-RL)|mimo_vl|mimo_vl|transformers>=4.49, qwen_vl_utils>=0.0.6, decord|&#x2718;|vision, video|[XiaomiMiMo/MiMo-VL-7B-RL](https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-RL)|
 |[mispeech/midashenglm-7b](https://modelscope.cn/models/mispeech/midashenglm-7b)|midashenglm|midashenglm|transformers>=4.52, soundfile|&#x2718;|audio|[mispeech/midashenglm-7b](https://huggingface.co/mispeech/midashenglm-7b)|

swift/llm/argument/train_args.py

@@ -133,7 +133,7 @@ def _check_padding_free(self):
         if self.padding_free or self.packing:
             if self.packing:
                 feature = 'packing'
-                self.padding_free = False
+                self.padding_free = True
             else:
                 feature = 'padding_free'
             if self.attn_impl not in {'flash_attn', 'flash_attention_2', 'flash_attention_3'}:

swift/llm/dataset/utils.py

@@ -141,7 +141,7 @@ def __init__(
         **kwargs,
     ):
         template.packing = True
-        template.padding_free = True
+        template.padding_free = True  # TODO: remove
         self.template = template
         self.dataset = dataset
         self.num_proc = num_proc
@@ -200,7 +200,7 @@ def __init__(
         **kwargs,
     ):
         template.packing = True
-        template.padding_free = True
+        template.padding_free = True  # TODO: remove
         self.template = template
         self.dataset = dataset
         self.num_proc = num_proc

swift/llm/model/register.py

@@ -226,7 +226,7 @@ def get_model_tokenizer_from_local(model_dir: str,
         model_config.keys_to_ignore_at_inference.append('past_key_values')
 
     torch_dtype = model_info.torch_dtype
-    model_config.torch_dtype = torch_dtype
+    HfConfigFactory.set_config_attr(model_config, 'torch_dtype', torch_dtype, include_vit=True)
     HfConfigFactory.compat_zero3(model_config)
     rope_scaling = kwargs.get('rope_scaling')
     max_model_len = kwargs.get('max_model_len')

swift/llm/model/utils.py

@@ -44,9 +44,9 @@ def update_attn_impl(config: PretrainedConfig,
             attn_impl_keys = [attn_impl_keys]
         attn_impl_keys = attn_impl_keys or AttnImpl.attn_impl_keys
         for key in attn_impl_keys:
-            HfConfigFactory.set_config_attr(config, key, attn_impl, ensure_set=False)
+            HfConfigFactory.set_config_attr(config, key, attn_impl, include_vit=True, ensure_set=False)
         for key in AttnImpl.use_flash_attn_keys:
-            HfConfigFactory.set_config_attr(config, key, use_flash_attn, ensure_set=False)
+            HfConfigFactory.set_config_attr(config, key, use_flash_attn, include_vit=True, ensure_set=False)
 
 
 @dataclass
@@ -88,6 +88,7 @@ def get_torch_dtype(config: Union[PretrainedConfig, Dict[str, Any]],
     @staticmethod
     def _get_config_attrs(config: Union[PretrainedConfig, Dict[str, Any]],
                           attr_name: str,
+                          include_vit: bool = False,
                           parent_key: Optional[str] = None) -> List[Tuple[PretrainedConfig, Any]]:
         res = []
         if isinstance(config, dict):
@@ -96,8 +97,10 @@ def _get_config_attrs(config: Union[PretrainedConfig, Dict[str, Any]],
             keys = dir(config)
         else:
             return []
-
-        if attr_name in keys and parent_key in [None, 'language_config', 'llm_config', 'text_config']:
+        config_keys = [None, 'language_config', 'llm_config', 'text_config']
+        if include_vit:
+            config_keys += ['vit_config', 'vision_config', 'audio_config']
+        if attr_name in keys and parent_key in config_keys:
             res.append((config, deep_getattr(config, attr_name)))
 
         for k in keys:
@@ -106,7 +109,7 @@ def _get_config_attrs(config: Union[PretrainedConfig, Dict[str, Any]],
                     v = config[k]
                 else:
                     v = getattr(config, k)
-                res += HfConfigFactory._get_config_attrs(v, attr_name, k)
+                res += HfConfigFactory._get_config_attrs(v, attr_name, include_vit, k)
         return res
 
     @staticmethod
@@ -119,9 +122,11 @@ def is_moe_model(config) -> bool:
         return False
 
     @staticmethod
-    def get_config_attr(config: Union[PretrainedConfig, Dict[str, Any]], attr_name: str) -> Optional[Any]:
+    def get_config_attr(config: Union[PretrainedConfig, Dict[str, Any]],
+                        attr_name: str,
+                        include_vit: bool = False) -> Optional[Any]:
         """Get the value of the attribute named attr_name."""
-        attrs = HfConfigFactory._get_config_attrs(config, attr_name)
+        attrs = HfConfigFactory._get_config_attrs(config, attr_name, include_vit)
         if len(attrs) == 0:
             return None
         else:
@@ -131,9 +136,10 @@ def get_config_attr(config: Union[PretrainedConfig, Dict[str, Any]], attr_name:
     def set_config_attr(config: Union[PretrainedConfig, Dict[str, Any]],
                         attr_name: str,
                         value: Any,
+                        include_vit: bool = False,
                         ensure_set: bool = True) -> int:
         """Set all the attr_name attributes to value."""
-        attrs = HfConfigFactory._get_config_attrs(config, attr_name)
+        attrs = HfConfigFactory._get_config_attrs(config, attr_name, include_vit)
         if ensure_set and len(attrs) == 0:
             attrs.append((config, None))
         for config, _ in attrs:

swift/llm/template/template/qwen.py

@@ -8,10 +8,11 @@
 import torch.nn.functional as F
 import transformers
 from packaging import version
+from PIL import Image
 from torch import nn
 from transformers.integrations import is_deepspeed_zero3_enabled
 
-from swift.llm import get_packed_seq_params, to_float_dtype
+from swift.llm import get_packed_seq_params, to_device, to_float_dtype
 from swift.utils import get_env_args, is_deepspeed_enabled
 from ..base import Template
 from ..constant import LLMTemplateType, MLLMTemplateType
@@ -717,11 +718,17 @@ def replace_tag(self, media_type: Literal['image', 'video', 'audio'], index: int
 
 
 class Ovis2_5Template(ThinkingTemplate):
-    num_frames = 8
     use_model = True
     skip_prompt = False
     support_padding_free = True
 
+    def init_processor(self, processor) -> None:
+        super().init_processor(processor)
+        self.min_pixels = get_env_args('min_pixels', int, 448 * 448)
+        self.max_pixels = get_env_args('max_pixels', int, 1344 * 1792)
+        self.video_max_pixels = get_env_args('video_max_pixels', int, 896 * 896)
+        self.num_frames = get_env_args('num_frames', int, 8)
+
     def replace_tag(self, media_type: Literal['image', 'video', 'audio'], index: int,
                     inputs: StdTemplateInputs) -> List[Context]:
         if media_type == 'image':
@@ -733,14 +740,10 @@ def replace_tag(self, media_type: Literal['image', 'video', 'audio'], index: int
             if self.mode == 'vllm':
                 return ['<video>']
             else:
-                num_frames = get_env_args('num_frames', int, self.num_frames)
-                inputs.images = load_video_ovis2_5(inputs.videos[index], num_frames)
+                inputs.images = load_video_ovis2_5(inputs.videos[index], self.num_frames)
                 return [[-200], '\n']
 
     def _encode(self, inputs: StdTemplateInputs) -> Dict[str, Any]:
-        min_pixels = get_env_args('min_pixels', int, 448 * 448)
-        max_pixels = get_env_args('max_pixels', int, 1344 * 1792)
-        video_max_pixels = get_env_args('video_max_pixels', int, 896 * 896)
         encoded = super()._encode(inputs)
         images = inputs.images
         input_ids = encoded['input_ids']
@@ -749,7 +752,7 @@ def _encode(self, inputs: StdTemplateInputs) -> Dict[str, Any]:
         if inputs.videos:
             assert len(inputs.videos) == 1, 'only support single video'
             encoded['pixel_values'], encoded['grid_thws'] = visual_tokenizer.preprocess(
-                video=inputs.images, min_pixels=min_pixels, max_pixels=video_max_pixels)
+                video=inputs.images, min_pixels=self.min_pixels, max_pixels=self.video_max_pixels)
             num_video_tokens = encoded['grid_thws'].prod(dim=-1)
             num_video_tokens //= visual_tokenizer.vit.config.hidden_stride**2
             num_video_tokens //= visual_tokenizer.vit.config.temporal_patch_size
@@ -762,7 +765,7 @@ def _get_new_tokens(i):
                 input_ids, encoded['labels'], encoded['loss_scale'], idx_list, _get_new_tokens)
         elif images:
             pixel_values, grid_thws = zip(
-                *(visual_tokenizer.preprocess(image=image, min_pixels=min_pixels, max_pixels=max_pixels)
+                *(visual_tokenizer.preprocess(image=image, min_pixels=self.min_pixels, max_pixels=self.max_pixels)
                   for image in images))
             encoded['pixel_values'] = torch.cat(pixel_values, dim=0)
             encoded['grid_thws'] = torch.cat(grid_thws, dim=0)
@@ -782,10 +785,32 @@ def _get_new_tokens(i):
         return encoded
 
     def _post_encode(self, model: nn.Module, inputs: Dict[str, Any]) -> Dict[str, Any]:
-        inputs_embeds = model.merge_multimodal(
-            input_ids=inputs['input_ids'],
-            pixel_values=inputs.pop('pixel_values', None),
-            grid_thws=inputs.pop('grid_thws', None))
+        input_ids = inputs['input_ids']
+        pixel_values = inputs.get('pixel_values', None)
+        grid_thws = inputs.get('grid_thws')
+        INDICATOR_IDS = [-301, -302, -303, -304]
+        VISUAL_ATOM_ID = -300
+        placeholder_token_mask = torch.lt(input_ids, 0)
+        inputs_embeds = model.get_wte()(torch.masked_fill(input_ids, placeholder_token_mask, 0))
+
+        if pixel_values is not None or is_deepspeed_enabled():
+            visual_indicator_embeds = model.vte(model.indicator_token_indices).to(
+                dtype=inputs_embeds.dtype, device=inputs_embeds.device)
+            for i, indicator_id in enumerate(INDICATOR_IDS):
+                inputs_embeds[input_ids == indicator_id] = visual_indicator_embeds[i]
+        if pixel_values is not None:
+            visual_tokens = model.visual_tokenizer(pixel_values, grid_thws)
+            visual_embeds = model.vte(visual_tokens).to(dtype=inputs_embeds.dtype, device=inputs_embeds.device)
+            inputs_embeds[input_ids == VISUAL_ATOM_ID] = visual_embeds
+        elif is_deepspeed_enabled():
+            media_inputs = model.visual_tokenizer.preprocess(
+                Image.new('RGB', (32, 32), (0, 0, 0)), min_pixels=self.min_pixels, max_pixels=self.max_pixels)
+            media_inputs = to_device(media_inputs, input_ids.device)
+            pixel_values = media_inputs['pixel_values'].type(inputs_embeds.dtype)
+            visual_tokens = model.visual_tokenizer(pixel_values, media_inputs['grid_thws'])
+            visual_embeds = model.vte(visual_tokens).to(dtype=inputs_embeds.dtype, device=inputs_embeds.device)
+            inputs_embeds = inputs_embeds + visual_embeds.mean() * 0.
+
         return {'inputs_embeds': inputs_embeds}
 
     def _data_collator_mm_data(self, batch: List[Dict[str, Any]]) -> Dict[str, Any]:

swift/megatron/model/constant.py

@@ -5,6 +5,7 @@ class MegatronModelType:
     qwen2_vl = 'qwen2_vl'
     qwen2_5_vl = 'qwen2_5_vl'
     qwen2_5_omni = 'qwen2_5_omni'
+    ovis2_5 = 'ovis2_5'
 
     internvl3 = 'internvl3'
     glm4_5v = 'glm4_5v'