File tree Expand file tree Collapse file tree 2 files changed +2
-6
lines changed Expand file tree Collapse file tree 2 files changed +2
-6
lines changed Original file line number Diff line number Diff line change @@ -18,7 +18,6 @@ ms-swift引入了Megatron的并行技术来加速大模型的训练,包括数
1818使用Megatron-SWIFT,除了安装swift依赖外,还需要安装以下内容:
1919
2020``` shell
21- # 推荐torch版本:2.5 / 2.6
2221pip install pybind11
2322
2423# transformer_engine
@@ -30,8 +29,6 @@ pip install --no-build-isolation transformer_engine[pytorch]
3029# apex
3130git clone https://github.com/NVIDIA/apex
3231cd apex
33- # https://github.com/modelscope/ms-swift/issues/4176
34- git checkout e13873debc4699d39c6861074b9a3b2a02327f92
3532pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings " --build-option=--cpp_ext" --config-settings " --build-option=--cuda_ext" ./
3633
3734# megatron-core
@@ -50,6 +47,7 @@ export MEGATRON_LM_PATH='/xxx/Megatron-LM'
5047# flash_attn
5148# 选择合适的版本进行安装:https://github.com/Dao-AILab/flash-attention/releases/tag/v2.8.1
5249# 注意:请勿安装高于transformer_engine限制的最高版本:https://github.com/NVIDIA/TransformerEngine/blob/release_v2.6/transformer_engine/pytorch/attention/dot_product_attention/utils.py#L109
50+ MAX_JOBS=8 pip install " flash-attn<2.8.2" --no-build-isolation
5351```
5452
5553或者你也可以使用镜像:(历史镜像查看[ 这里] ( ../GetStarted/SWIFT安装.md#镜像 ) )
Original file line number Diff line number Diff line change @@ -17,7 +17,6 @@ ms-swift incorporates Megatron's parallelization techniques to accelerate the tr
1717To use Megatron-SWIFT, in addition to installing the ` swift ` dependencies, you also need to install the following:
1818
1919``` shell
20- # Recommended PyTorch version: 2.5 / 2.6
2120pip install pybind11
2221
2322# transformer_engine
@@ -29,8 +28,6 @@ pip install --no-build-isolation transformer_engine[pytorch]
2928# apex
3029git clone https://github.com/NVIDIA/apex
3130cd apex
32- # https://github.com/modelscope/ms-swift/issues/4176
33- git checkout e13873debc4699d39c6861074b9a3b2a02327f92
3431pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings " --build-option=--cpp_ext" --config-settings " --build-option=--cuda_ext" ./
3532
3633# megatron-core
@@ -49,6 +46,7 @@ export MEGATRON_LM_PATH='/xxx/Megatron-LM'
4946# flash_attn
5047# Choose an appropriate version to install: https://github.com/Dao-AILab/flash-attention/releases/tag/v2.8.1
5148# Note: Do not install a version higher than the maximum supported by transformer_engine: https://github.com/NVIDIA/TransformerEngine/blob/release_v2.6/transformer_engine/pytorch/attention/dot_product_attention/utils.py#L109
49+ MAX_JOBS=8 pip install " flash-attn<2.8.2" --no-build-isolation
5250```
5351
5452Alternatively, you can also use the image: (See historical images [ here] ( ../GetStarted/SWIFT-installation.md#mirror ) )
You can’t perform that action at this time.
0 commit comments