parameters which did not receive grad for rank 1

[OnlyIanA]

thanks to ur work, however i met a problem i cannot fixed. how can i solve this problem.

[zqcrafts]

It seems like an issue with PyTorch DDP. You might want to check if the number of GPUs set matches the number of local GPUs available. If that's not the problem, you can use the following code for debugging.

import torch
import torch.distributed as dist

rank = dist.get_rank()
for name, param in model.named_parameters():
    if param.grad is None:
        print(f"WARNING: Rank {rank} - Param '{name}' has NO grad!")
    else:
        print(f"Rank {rank} - Param '{name}' grad norm: {param.grad.norm().item()}")

[OnlyIanA]

Thanks to ur reply, and i find that if i only use one GPU, there is no issue. The problem only occurs when i use multiple GPUs and there is no GPU matching problem. I will further explore where the problem is. Thanks to ur reply and code again.