v2.3.1

English Version

New Features:

1. ms-swift paper published: https://arxiv.org/abs/2408.05517
2. Web-UI supports audio and video.
3. Support for deploying audio and video models using the OpenAI API.
4. Utilizes a new multimodal training framework.
5. supports inference acceleration for video models (lmdeploy & internvl2 series).

New Models:

1. idefics3-8b-llama3
2. llava-hf 72b, 110b, llama3-llava
3. deepseek-coder-v2, deepseek-coder-lite-v2, deepseek-v2

中文版

新功能：

1. 发布了 ms-swift 论文：https://arxiv.org/abs/2408.05517
2. Web-UI 支持音频和视频。
3. 支持使用 OpenAI API 部署音频和视频模型。
4. 采用新的多模态训练框架。
5. 支持视频模型的推理加速（lmdeploy 和 internvl2 系列）。

新模型：

1. idefics3-8b-llama3
2. llava-hf 72b、110b、llama3-llava
3. deepseek-coder-v2、deepseek-coder-lite-v2、deepseek-v2

What's Changed

• support video deploy by @Jintao-Huang in #1632
• fix infer deploy by @Jintao-Huang in #1665
• Fix bugs by @tastelikefeet in #1664
• Add generation config support for swift eval by @Yunnglin in #1659
• Add a bunch of small features by @tastelikefeet in #1669
• Support lmdeploy&vllm support video by @Jintao-Huang in #1668
• [TorchAcc] fix: add mark_step when GA is greater than 1 by @baoleai in #1677
• Support audio and video in web-ui by @tastelikefeet in #1681
• Fix lmdeploy deploy seed by @Jintao-Huang in #1678
• add paper link by @tastelikefeet in #1684
• set default writer_batch_size argument in mllm rlhf by @hjh0119 in #1689
• New multimodal framework by @Jintao-Huang in #1594
• fix video component by @tastelikefeet in #1692
• refactor internvl2 by @Jintao-Huang in #1625
• fix model dtype in rlhf by @hjh0119 in #1696
• Fix minicpmv 2.6 & video/plain-text bug by @Jintao-Huang in #1698
• Fix internvl2 device map by @Jintao-Huang in #1701
• Support deepseek-v2 base by @Jintao-Huang in #1708
• [TorchAcc] update data bucketing strategy by @baoleai in #1706
• [TorchAcc] consolidate FSDP ckpt after saving instead of during loading by @baoleai in #1709
• fix deploy stream by @Jintao-Huang in #1713
• update docs by @Jintao-Huang in #1714
• Fix mllm rlhf by @hjh0119 in #1707
• support idefics3 by @Jintao-Huang in #1720
• Fix minicpm-v-v2.6 device_map='auto' & agent deploy by @Jintao-Huang in #1733
• Fix kto by @hjh0119 in #1737
• support llava-hf 72b, 110b, llama3-llava-hf by @Jintao-Huang in #1741
• Fix glm4v rlhf by @hjh0119 in #1745

New Contributors

• @Yunnglin made their first contribution in #1659

Full Changelog: v2.3.0...v2.3.1