How to train on a cluster with many nodes?

[rohskopf]

In README.md, we have the example:

For example if you wish to use 3 cores of 2 CPUs on one node, you may set the environmental variables and run DeePMD-kit as follows:

export OMP_NUM_THREADS=6
export TF_INTRA_OP_PARALLELISM_THREADS=3
export TF_INTER_OP_PARALLELISM_THREADS=2
dp train input.json

But how do we incorporate many nodes?

I have a cluster with many nodes, 2 CPUs per node, and 20 cores per node. So if I was running on a single node, I would do:

export OMP_NUM_THREADS=40
export TF_INTRA_OP_PARALLELISM_THREADS=20
export TF_INTER_OP_PARALLELISM_THREADS=2
dp train input.json

If I want to extend to 3 nodes, for example, should I just triple OMP_NUM_THREADS? So for 3 nodes, 2 CPUs per node, and 20 cores per CPU, I would do:

export OMP_NUM_THREADS=120
export TF_INTRA_OP_PARALLELISM_THREADS=20
export TF_INTER_OP_PARALLELISM_THREADS=2
dp train input.json

Is this correct?

[njzjz]

OpenMP does not support multi-nodes, and currently we don't support multi-nodes training as well.

[jiangzihanict]

Hi, does DeepMD now support distributed training?

[njzjz]

does DeepMD now support distributed training?

No.