Error when attempting to train deep mimic using mpi_run.py

[rohit-kumar-j]

This is in a conda env configured to python 3.6 and TensorFlow 1.15

Error when running python3 mpi_run.py --arg_file train_humanoid3d_walk_args.txt --num_workers 16 on Google cloud platform (Ubuntu 20.04)
However, python3 testrl.py works flawlessly

Error Output:

$ python3 mpi_run.py --arg_file train_humanoid3d_walk_args.txt --num_workers 1
parentdir= /home/user/Work/bullet3/pybullet/examples/pybullet/gym
Running with 1 workers
cmd: mpiexec -n 1 python3 DeepMimic_Optimizer.py --arg_file train_humanoid3d_walk_args.txt --num_workers 1

$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              2
Core(s) per socket:              1
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           63
Model name:                      Intel(R) Xeon(R) CPU @ 2.30GHz
Stepping:                        0
CPU MHz:                         2299.998
BogoMIPS:                        4599.99
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       32 KiB
L1i cache:                       32 KiB
L2 cache:                        256 KiB
L3 cache:                        45 MiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Mitigation; PTE Inversion
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT Host state u
                                 nknown
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled v
                                 ia prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user
                                  pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB condit
                                 ional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtr
                                 r pge mca cmov pat pse36 clflush mmx fxsr sse s
                                 se2 ss ht syscall nx pdpe1gb rdtscp lm constant
                                 _tsc rep_good nopl xtopology nonstop_tsc cpuid 
                                 tsc_known_freq pni pclmulqdq ssse3 fma cx16 pci
                                 d sse4_1 sse4_2 x2apic movbe popcnt aes xsave a
                                 vx f16c rdrand hypervisor lahf_lm abm invpcid_s
                                 ingle pti ssbd ibrs ibpb stibp fsgsbase tsc_adj
                                 ust bmi1 avx2 smep bmi2 erms invpcid xsaveopt a

[rohit-kumar-j]

Never got down to the bottom of it.. 😢