how to train wan using 8 GPUs

[tanshuai0219]

I notice that there is only 4 GPUs scripts, even though I modify the script for 8 GPU training, it gets some errors.

[a-r-r-o-w]

Could you elaborate on what the error is and what you're trying? It'll help me check if there's any modifications required to make it run

[tanshuai0219]

Could you elaborate on what the error is and what you're trying? It'll help me check if there's any modifications required to make it run

I change the train.sh as :
`NUM_GPUS=8
CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" #

Check the JSON files for the expected JSON format

TRAINING_DATASET_CONFIG="lora_motions/wave_delete/training.json"
VALIDATION_DATASET_FILE="lora_motions/wave_delete/validation.json"

Depending on how many GPUs you have available, choose your degree of parallelism and technique!

DDP_1="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 1 --dp_shards 1 --cp_degree 1 --tp_degree 1"
DDP_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 2 --dp_shards 1 --cp_degree 1 --tp_degree 1"
DDP_4="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 4 --dp_shards 1 --cp_degree 1 --tp_degree 1"
DDP_8="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 8 --dp_shards 1 --cp_degree 1 --tp_degree 1"
FSDP_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 1 --dp_shards 2 --cp_degree 1 --tp_degree 1"
FSDP_4="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 1 --dp_shards 4 --cp_degree 1 --tp_degree 1"
HSDP_2_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 2 --dp_shards 2 --cp_degree 1 --tp_degree 1"
HSDP_4_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 4 --dp_shards 2 --cp_degree 1 --tp_degree 1"

Parallel arguments

parallel_cmd=(
$DDP_8
)`

only
DDP_8="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 8 --dp_shards 1 --cp_degree 1 --tp_degree 1"
and
HSDP_4_2="--parallel_backend $BACKEND --pp_degree 1 --dp_degree 4 --dp_shards 2 --cp_degree 1 --tp_degree 1"
only these lines are modified to support 8 gpu training.
but both report error as:

`Processing data on rank 4: 100%|██████████| 25/25 [00:20<00:00, 1.24it/s]
Processing data on rank 2: 100%|██████████| 25/25 [00:20<00:00, 1.25it/s]
Processing data on rank 1: 100%|██████████| 25/25 [00:19<00:00, 1.26it/s]
[rank5]:[E326 11:12:42.837110054 ProcessGroupNCCL.cpp:616] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600082 milliseconds before timing out.
[rank5]:[E326 11:12:42.837274367 ProcessGroupNCCL.cpp:1785] [PG ID 0 PG GUID 0(default_pg) Rank 5] Exception (either an error or timeout) detected by watchdog at work: 4, last enqueued NCCL work: 4, last completed NCCL work: 3.
[rank0]:[E326 11:12:42.311841732 ProcessGroupNCCL.cpp:616] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600005 milliseconds before timing out.
[rank0]:[E326 11:12:42.311978156 ProcessGroupNCCL.cpp:1785] [PG ID 0 PG GUID 0(default_pg) Rank 0] Exception (either an error or timeout) detected by watchdog at work: 4, last enqueued NCCL work: 4, last completed NCCL work: 3.
[rank3]:[E326 11:12:43.623350961 ProcessGroupNCCL.cpp:616] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600028 milliseconds before timing out.
[rank3]:[E326 11:12:43.623470545 ProcessGroupNCCL.cpp:1785] [PG ID 0 PG GUID 0(default_pg) Rank 3] Exception (either an error or timeout) detected by watchdog at work: 4, last enqueued NCCL work: 4, last completed NCCL work: 3.
[rank5]:[E326 11:12:43.702233611 ProcessGroupNCCL.cpp:1834] [PG ID 0 PG GUID 0(default_pg) Rank 5] Timeout at NCCL work: 4, last enqueued NCCL work: 4, last completed NCCL work: 3.
[rank5]:[E326 11:12:43.702262793 ProcessGroupNCCL.cpp:630] [Rank 5] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[E326 11:12:43.702268998 ProcessGroupNCCL.cpp:636] [Rank 5] To avoid data inconsistency, we are taking the entire process down.
[E326 11:12:43.703531515 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600082 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcbf8d6c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fcbaebcc772 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fcbaebd3bb3 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fcbaebd561d in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fcbf91745c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x89867 (0x7fcc009b3867 in /usr/lib64/libc.so.6)
frame #6: + 0x10948c (0x7fcc00a3348c in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 5] Process group watchdog thread terminated with exception: [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600082 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcbf8d6c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fcbaebcc772 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fcbaebd3bb3 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fcbaebd561d in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fcbf91745c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x89867 (0x7fcc009b3867 in /usr/lib64/libc.so.6)
frame #6: + 0x10948c (0x7fcc00a3348c in /usr/lib64/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcbf8d6c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fcbae84271b in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fcbf91745c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x89867 (0x7fcc009b3867 in /usr/lib64/libc.so.6)
frame #4: + 0x10948c (0x7fcc00a3348c in /usr/lib64/libc.so.6)

2025-03-26 11:12:43,529 - finetrainers - ERROR - Error during training: value cannot be converted to type int without overflow
[rank0]:[E326 11:12:43.217052463 ProcessGroupNCCL.cpp:1834] [PG ID 0 PG GUID 0(default_pg) Rank 0] Timeout at NCCL work: 4, last enqueued NCCL work: 4, last completed NCCL work: 3.
[rank0]:[E326 11:12:43.217082605 ProcessGroupNCCL.cpp:630] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank0]:[E326 11:12:43.217088133 ProcessGroupNCCL.cpp:636] [Rank 0] To avoid data inconsistency, we are taking the entire process down.
2025-03-26 11:12:43,612 - finetrainers - ERROR - An error occurred during training: value cannot be converted to type int without overflow
[E326 11:12:43.218348068 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600005 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f768676c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f763c5cc772 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f763c5d3bb3 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f763c5d561d in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7f7686b1a5c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x89867 (0x7f768e28d867 in /usr/lib64/libc.so.6)
frame #6: + 0x10948c (0x7f768e30d48c in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600005 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f768676c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f763c5cc772 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f763c5d3bb3 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f763c5d561d in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7f7686b1a5c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x89867 (0x7f768e28d867 in /usr/lib64/libc.so.6)
frame #6: + 0x10948c (0x7f768e30d48c in /usr/lib64/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f768676c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7f763c24271b in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7f7686b1a5c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x89867 (0x7f768e28d867 in /usr/lib64/libc.so.6)
frame #4: + 0x10948c (0x7f768e30d48c in /usr/lib64/libc.so.6)

[rank3]:[E326 11:12:44.783260852 ProcessGroupNCCL.cpp:1834] [PG ID 0 PG GUID 0(default_pg) Rank 3] Timeout at NCCL work: 4, last enqueued NCCL work: 4, last completed NCCL work: 3.
[rank3]:[E326 11:12:44.783300440 ProcessGroupNCCL.cpp:630] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank3]:[E326 11:12:44.783306348 ProcessGroupNCCL.cpp:636] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
[E326 11:12:44.784554433 ProcessGroupNCCL.cpp:1595] [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600028 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fac0d76c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fabc4bcc772 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fabc4bd3bb3 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fabc4bd561d in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fac0f0965c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x89867 (0x7fac1688d867 in /usr/lib64/libc.so.6)
frame #6: + 0x10948c (0x7fac1690d48c in /usr/lib64/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=4, OpType=ALLGATHER, NumelIn=1, NumelOut=8, Timeout(ms)=600000) ran for 600028 milliseconds before timing out.
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fac0d76c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7fabc4bcc772 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7fabc4bd3bb3 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fabc4bd561d in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fac0f0965c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #5: + 0x89867 (0x7fac1688d867 in /usr/lib64/libc.so.6)
frame #6: + 0x10948c (0x7fac1690d48c in /usr/lib64/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fac0d76c446 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: + 0xe4271b (0x7fabc484271b in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fac0f0965c0 in /root/miniconda3/lib/python3.10/site-packages/torch/lib/libtorch.so)
frame #3: + 0x89867 (0x7fac1688d867 in /usr/lib64/libc.so.6)
frame #4: + 0x10948c (0x7fac1690d48c in /usr/lib64/libc.so.6)

W0326 11:12:50.809000 403183 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 403251 closing signal SIGTERM
W0326 11:12:50.810000 403183 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 403254 closing signal SIGTERM
W0326 11:12:50.811000 403183 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 403257 closing signal SIGTERM
W0326 11:12:50.811000 403183 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 403258 closing signal SIGTERM
E0326 11:12:52.092000 403183 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 5 (pid: 403256) of binary: /root/miniconda3/bin/python
Traceback (most recent call last):
File "/root/miniconda3/bin/torchrun", line 8, in
sys.exit(main())
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 355, in wrapper
return f(*args, **kwargs)
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
run(args)
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/root/miniconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(`

If I use the default config (i.e., DDP_4), it runs correctly without errors.

[CoMo6]

I find an interesting problem when finetuning wan1.3b, when encode video using vae, the debugging (ipdb) would increase the GPU memory greatly. wen not enable_precomputing, input video shape:[1,3,49,480,832] only need 20G, but when debug, only the encoding process will cost 46G GPU,
video = video.unsqueeze(0).to(device=device, dtype=dtype)
video = video.permute(0, 2, 1, 3, 4) #[B, C, F, H ,W]#torch.Size([1, 3, 49, 480, 720])
video = torch.cat([video,video,video,video,video,video], dim=0)
import ipdb; ipdb.set_trace() #### remove this line can save much memory
latents = vae.encode(video).latent_dist.sample(generator=None)

[a-r-r-o-w]

@tanshuai0219 Sorry for the late reply. I tried to reproduce the error today with 8 GPUs today using both DDP_8 and HSDP_4_2 but couldn't reproduce it. The training starts shortly after the precomputation completes for me, and I don't see an allgather timeout.

If you're still interested in trying, could you try launching with NCCL_IB_DISABLE=1 and NCCL_P2P_DISABLE=1? I find it very confusing how this may work with DDP_4 but not with 8 if it's the same machine 🤔

[Radioheading]

I encoutered the same issue earlier this day when using DDP_8 and HSDP_4_2. Even with NCCL_IB_DISABLE=1 and `NCCL_P2P_DISABLE=1', it still gets timeout

[a-r-r-o-w]

@Radioheading Could you create a dummy dataset of exactly 8 videos and try to launch training with DDP_8? I think I have an idea what might be causing the problem

[Radioheading]

Hi @a-r-r-o-w, thanks for the advice. I've tried an 8-video dataset with DDP_8 but it seems that it stills gets timeout error.
By the way, if I want to train longer videos (e.g. 100+ frames with Wan2.1 T2V 14B), what's the desirable parallel configuration for saving memory in your current framework. Thanks a lot!

[a-r-r-o-w]

Oh that is very weird and I can't figure out how to replicate the problem yet 🥲 I will spend some time to try and figure out the problem you're facing. As mentioned by the author, DDP_4 seems to work but DDP_8 doesn't - do you experience the same?

Additionally, for training with a lot of frames, I think it will be necessary to use context parallel because otherwise it will be too slow to train and not utilize GPU capacity very well. Support for the same is near completion: #381. My tests are mostly on 2 node setting at the moment and this seems to be working best: (dp_shards: 4, dp_degree: 2, cp_degree: 2) or (dp_shards: 2, dp_degree: 2, cp_degree: 4)

[a-r-r-o-w]

Oh that is very weird and I can't figure out how to replicate the problem yet 🥲 I will spend some time to try and figure out the problem you're facing. As mentioned by the author, DDP_4 seems to work but DDP_8 doesn't - do you experience the same?

Additionally, for training with a lot of frames, I think it will be necessary to use context parallel because otherwise it will be too slow to train and not utilize GPU capacity very well. Support for the same is near completion: #381. My tests are mostly on 2 node setting at the moment and this seems to be working best: (dp_shards: 4, dp_degree: 2, cp_degree: 2) or (dp_shards: 2, dp_degree: 2, cp_shards: 4)

[a-r-r-o-w]

Maybe it's something related to specific pytorch or NCCL versions. I'll try to check on a few of them to see if I can replicate

[Radioheading]

Exactly, all the configurations about 4 GPUs work fine but those with 8 GPUs can't work properly. FYI, my pytorch version is 2.7.0 and my nccl version is "nvidia-nccl-cu12 2.26.2". Could you kindly share your configurations? Thanks a lot!

[Radioheading]

I found that FSDP_8 works with my aforementioned environment, and for 720p setting, it currently supports up to 97 frames' LoRA fine-tuning without OOM.