Parallel training using horovodrun not working

[AnuragKr]

Command --
CUDA_VISIBLE_DEVICES=0,1 horovodrun -np 2 \ dp train --mpi-log=workers input.json

Output --
[1,0]:WARNING:tensorflow:From /home/anurag/.local/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[1,0]:Instructions for updating:
[1,0]:non-resource variables are not supported in the long term
[1,1]:WARNING:tensorflow:From /home/anurag/.local/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
[1,1]:Instructions for updating:
[1,1]:non-resource variables are not supported in the long term
[1,0]:Switch to serial execution due to lack of horovod module.
[1,1]:Switch to serial execution due to lack of horovod module.
[1,0]:DEEPMD INFO Calculate neighbor statistics... (add --skip-neighbor-stat to skip this step)
[1,1]:DEEPMD INFO Calculate neighbor statistics... (add --skip-neighbor-stat to skip this step)
[1,1]:DEEPMD INFO training data with min nbor dist: 0.8854385688525511
[1,1]:DEEPMD INFO training data with max nbor size: [38, 72]
[1,1]:DEEPMD INFO _____ _____ __ __ _____ _ _ _
[1,1]:DEEPMD INFO | __ \ | __ \ | / || __ \ | | ()| |
[1,1]:DEEPMD INFO | | | | ___ ___ | |__) || \ / || | | | ______ | | __ _ | |
[1,1]:DEEPMD INFO | | | | / _ \ / _ | / | |/| || | | |||| |/ /| || |
[1,1]:DEEPMD INFO | || || /| /| | | | | || || | | < | || |
[1,1]:DEEPMD INFO |/ _| _||| || |_||____/ ||_|| __|
[1,1]:DEEPMD INFO Please read and cite:
[1,1]:DEEPMD INFO Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018)
[1,1]:DEEPMD INFO installed to: /tmp/pip-req-build-pjks4pue/_skbuild/linux-x86_64-3.8/cmake-install
[1,1]:DEEPMD INFO source : v2.1.1
[1,1]:DEEPMD INFO source brach: master
[1,1]:DEEPMD INFO source commit: c4f0cec
[1,1]:DEEPMD INFO source commit at: 2022-04-16 11:11:16 +0800
[1,1]:DEEPMD INFO build float prec: double
[1,1]:DEEPMD INFO build with tf inc: /tmp/pip-build-env-dfkmanfm/normal/lib/python3.8/site-packages/tensorflow/include
[1,1]:DEEPMD INFO build with tf lib:
[1,1]:DEEPMD INFO ---Summary of the training---------------------------------------
[1,1]:DEEPMD INFO running on: hp-HP-Z8-G4-Workstation
[1,1]:DEEPMD INFO computing device: gpu:0
[1,1]:DEEPMD INFO CUDA_VISIBLE_DEVICES: 0,1
[1,1]:DEEPMD INFO Count of visible GPU: 2
[1,1]:DEEPMD INFO num_intra_threads: 6
[1,1]:DEEPMD INFO num_inter_threads: 5
[1,1]:DEEPMD INFO -----------------------------------------------------------------

Kindly help me out unable to find root cause.I have installed OpenMPI for MPI.

[AnuragKr]

This discussion replication of this particular bug.#1774
Discussion Closed.