Enabled high-performance Automatic Tensor Parallelism (auto TP) for the MoE models on multiple GPUs/HPUs#6964

Open

gyou2021 wants to merge 64 commits intodeepspeedai:masterfrom

gyou2021:autoTP_Qwen2Moe_DeepSeekv2

Commits on Feb 28, 2025

Reduced the experts allreduce number per layer to ONCE for the Qwen2-MoE and DeepSeek-V2 models.
gyou2021
committed
Fixed format
gyou2021
committed
Removed print
gyou2021
committed
Fix a bug about set.
gyou2021
committed
Add the missing view operations from sequence parallel(async). (#6750)

authored and
gyou2021
committed
Update `torch.norm` to `torch.linalg.norm` and `torch.linalg.vector_norm` (#6931)

loadams
authored and
gyou2021
committed
Using explicit GPU upcast for ZeRO-Offload (#6962)

xylian86
authored and
gyou2021
committed
Update version.txt after 0.16.3 release (#6965)

loadams
authored and
gyou2021
committed
Precisely track nvme optimizer offload (#6963)

tjruwase
authored and
gyou2021
committed
Update build_win.bat script to exclue GDS op as it lacks Windows support. (#6971)

loadams
authored and
gyou2021
committed
Add CUDA 12.8 support and comment on CUDA 12.7 (#6975)

loadams
authored and
gyou2021
committed
Update torch versions to support 2.6 (#6977)

loadams
authored and
gyou2021
committed
generalize deepspeed linear and implement it for non cuda systems (#6932)

authored and
gyou2021
committed
Update recommended Windows whl building versions (#6983)

loadams
authored and
gyou2021
committed
Title: Fix setup_env_ranks to Properly Set Environment Variables Instead of Raising Error (#6979)

authored and
gyou2021
committed
Specify torchvision in nv-ds-chat workflow (prevents errors with torch 2.6) (#6982)

loadams
authored and
gyou2021
committed
Remove assumption that padding only occurs on last rank (#6974)

authored and
gyou2021
committed
Use ds-specific module id to avoid conflicts (#6847)

authored and
gyou2021
committed
Update A6000 workflows to use newer docker container - 24.09 vs 24.03 (#6967)

loadams
authored and
gyou2021
committed
Allow NVIDIA Blackwell (#6991)

fabiendupont
authored and
gyou2021
committed
Update GH org references (#6998)

authored and
gyou2021
committed
Update CNAME

loadams
authored and
gyou2021
committed
Update CNAME

loadams
authored and
gyou2021
committed
[XPU] max1100 workflow update for docker and softwares (#7003)

Liangliang-Ma
authored and
gyou2021
committed
autotp training(fix dco) (#7004)

inkcherry
authored and
gyou2021
committed
import triton files when triton is supported and installed (#6989)

oelayan7
authored and
gyou2021
committed
Update A6000 tests transformers version (#7016)

authored and
gyou2021
committed
Fix ds-chat CI regression (#7015)

tjruwase
authored and
gyou2021
committed
[Ulysses tutorial] typos (#7024)

stas00
authored and
gyou2021
committed
fix hostname -I for macOS #6497 (#6990)

authored and
gyou2021
committed
Update workflows to cuda 12.4 (#7000)

authored and
gyou2021
committed
[ROCm] Enable fp_quantizer on ROCm (#7027)

authored and
gyou2021
committed
add gds chinese blog (#7034)

authored and
gyou2021
committed
Add chinese blog for deepspeed windows, and fix format (#7035)

authored and
gyou2021
committed
AIO on ROCM (#7023)

authored and
gyou2021
committed
Control trace cache warnings (#7039)

tjruwase
authored and
gyou2021
committed
Update CUDA compute capability to support Blackwell (#7047)

hwchen2017
authored and
gyou2021
committed
Update setup.py handling of ROCm cupy (#7051)

loadams
authored and
gyou2021
committed
nv-ds-chat breaks with latest transformers (#7052)

loadams
authored and
gyou2021
committed
Rename aio_thread_count to intra_op_parallelism (#7056)

tjruwase
authored and
gyou2021
committed
add autoTP training zero2 tests (#7049)

authored and
gyou2021
committed
Fix, bf16 optimizer remove dup loop (#7054)

authored and
gyou2021
committed
Update version.txt after 0.16.4 release (#7063)

loadams
authored and
gyou2021
committed
fix an outdated doc wrt CUDA_VISIBLE_DEVICES (#7058)

stas00
authored and
gyou2021
committed
Tecorigin sdaa accelerator (#6903)

authored and
gyou2021
committed
Handle special case of libuv for Windows (#7064)

loadams
authored and
gyou2021
committed
Update README with info on newest accelerator (#7065)

loadams
authored and
gyou2021
committed
Bug Fix for offload_states API (#7050)

authored and
gyou2021
committed
Fix TOCTOU issues, switch to fstat (#7067)

loadams
authored and
gyou2021
committed
config torch to avoid graph breaks caused by logger (#6999)

authored and
gyou2021
committed
Fix meta load tensor imcompatible issue (#7073)

authored and
gyou2021
committed
Replace calls to `python setup.py sdist` with `python -m build --sdist` (#7069)

loadams
authored and
gyou2021
committed
Revert "Handle special case of libuv for Windows (#7064)" (#7076)

loadams
authored and
gyou2021
committed
Add DeepseekV3 AutoTP. (#7045)

Yejing-Lai
authored and
gyou2021
committed
Improve inference tutorial docs (#7083)

loadams
authored and
gyou2021
committed
Added support for the environment variable DS_MOE_EXPERTS_REDUCE_ONCE, which indicates whether moe expert all-reduce once optimization was used.
gyou2021
committed
Changed env variable name to 'DS_MOE_TP_SINGLE_ALLREDUCE'
gyou2021
committed
Pin transformers version on tests that use latest. (#7085)

loadams
authored and
gyou2021
committed
Update README.md with ICS '23 MoE paper link (#7087)

authored and
gyou2021
committed
Update parallelism for nv-torch-latest/nightly tests due to more GPUs/runner (#7086)

authored and
gyou2021
committed
Remove workflows for very old torch versions (#7090)

loadams
authored and
gyou2021
committed
Fixed conflicts
gyou2021
committed

Commits on Mar 5, 2025