How to run code on multi-machine multi-gpu?

[ljz756245026]

I am going to train a model on 2 machines (each machine with 2 GPUs).

I can run DDP on single machine(2 GPUs) through python singlenode.py. It runs successfully.
However, I cannot run DDP on 2 machines.

I write the core code like:

    trainer = Trainer(max_epochs=10, gpus=2, accelerator="ddp",num_nodes=2)
    trainer.fit(model, train_dataloaders=train_loader, val_dataloaders=val_loader)

When I run python multinodes.py on two machines. Both of the two machine output things as follows.Program will pause here.

(light1) xxx@mobitec-Precision-Tower-7910:~/pytorch_lightning_code$ python multinode.py
Using Cached file: data/cora/ch7_cached.pkl
Global seed set to 42
Using Cached file: data/cora/ch7_cached.pkl
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Using Cached file: data/cora/ch7_cached.pkl
Global seed set to 42
Global seed set to 42
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
Using Cached file: data/cora/ch7_cached.pkl
Global seed set to 42
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All DDP processes registered. Starting ddp with 4 processes
----------------------------------------------------------------------------------------------------

NCCL version 2.7.8+cuda10.2

Both of these 2 machine will output the following GLOBAL_RANK.

initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/4

Then ,I tried use command like

WORLD_SIZE=2 NODE_RANK=0 LOCAL_RANK=0 python multinode.py
WORLD_SIZE=2 NODE_RANK=0 LOCAL_RANK=1 python multinode.py

The output is as follows:
on node 0:

(light1) xxx@AMD-Workstation:~/pytorch_lightning_code$ WORLD_SIZE=2 NODE_RANK=0 LOCAL_RANK=0 python multinode.py
Using Cached file: data/cora/ch7_cached.pkl
Global seed set to 42
Using Cached file: data/cora/ch7_cached.pkl
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
Global seed set to 42
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/4

on node 1:

(light1) xxx@mobitec-Precision-Tower-7910:~/pytorch_lightning_code$ WORLD_SIZE=2 NODE_RANK=1 LOCAL_RANK=0 python multinode.py
Using Cached file: data/cora/ch7_cached.pkl
Global seed set to 42
Using Cached file: data/cora/ch7_cached.pkl
Global seed set to 42
initializing ddp: GLOBAL_RANK: 2, MEMBER: 3/4

I found that these 2 node won't initializing ddp successfully. They will stop on initializing the 1st process.

What should I do to train on multi-machine multi-GPU ??