Parallel training with dp train

[maverick422]

First of all thank you very much，so when i using a Parallel training with the code
CUDA_VISIBLE_DEVICES=0,1 horovodrun -np 2 \ dp train --mpi-log=workers input.json

in the beginning it shows that

[1] DEEPMD rank:1 INFO _____ _____ __ __ _____ _ _ _ [1] DEEPMD rank:1 INFO | __ \ | __ \ | \/ || __ \ | | (_)| | [1] DEEPMD rank:1 INFO | | | | ___ ___ | |__) || \ / || | | | ______ | | __ _ | |_ [1] DEEPMD rank:1 INFO | | | | / _ \ / _ \| ___/ | |\/| || | | ||______|| |/ /| || __| [1] DEEPMD rank:1 INFO | |__| || __/| __/| | | | | || |__| | | < | || |_ [1] DEEPMD rank:1 INFO |_____/ \___| \___||_| |_| |_||_____/ |_|\_\|_| \__| [1] DEEPMD rank:1 INFO Please read and cite: [1] DEEPMD rank:1 INFO Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018) [1] DEEPMD rank:1 INFO installed to: /tmp/pip-req-build-mfodjfod/_skbuild/linux-x86_64-3.9/cmake-install [1] DEEPMD rank:1 INFO source : v2.0.3 [1] DEEPMD rank:1 INFO source brach: HEAD [1] DEEPMD rank:1 INFO source commit: 159e45d2 [1] DEEPMD rank:1 INFO source commit at: 2021-10-15 10:31:09 +0800 [1] DEEPMD rank:1 INFO build float prec: double [1] DEEPMD rank:1 INFO build with tf inc: /public/home/huangan21/deepmd-kit/envs/deepmd/lib/python3.9/site-packages/tensorflow/include [1] DEEPMD rank:1 INFO build with tf lib: [1] DEEPMD rank:1 INFO ---Summary of the training--------------------------------------- [1] DEEPMD rank:1 INFO distributed [1] DEEPMD rank:1 INFO world size: 2 [1] DEEPMD rank:1 INFO my rank: 1 [1] DEEPMD rank:1 INFO node list: ['node008'] [1] DEEPMD rank:1 INFO running on: node008 [1] DEEPMD rank:1 INFO computing device: gpu:1 [1] DEEPMD rank:1 INFO CUDA_VISIBLE_DEVICES: 0,1 [1] DEEPMD rank:1 INFO Count of visible GPU: 2 [1] DEEPMD rank:1 INFO num_intra_threads: 0 [1] DEEPMD rank:1 INFO num_inter_threads: 0 [1] DEEPMD rank:1 INFO -----------------------------------------------------------------

with two NVIDIA GeForce RTX 3070 on the cluster i uesd , it goes

[0] 2021-12-20 23:06:54.668172: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: [0] pciBusID: 0000:3b:00.0 name: NVIDIA GeForce RTX 3070 computeCapability: 8.6 [0] coreClock: 1.785GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s

but sooner it went wrong like

`[0] cuda assert: no kernel image is available for execution on the device /home/conda/feedstock_root/build_artifacts/libdeepmd_1634271603078/work/source/lib/src/cuda/coord.cu 366
[1] cuda assert: no kernel image is available for execution on the device /home/conda/feedstock_root/build_artifacts/libdeepmd_1634271603078/work/source/lib/src/cuda/coord.cu 366

===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 43976 RUNNING AT localhost
= EXIT CODE: 11
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES

Intel(R) MPI Library troubleshooting guide:
https://software.intel.com/node/561764
===================================================================================`

what's happening and what shuold i do? thanks again!

[njzjz]

See #986.

[maverick422]

Thanks a lot