Problem on train on GPUs on single node

[TinacciL]

I Installed the last DeepMd-kit GPU version (ghcr.io/deepmodeling/deepmd-kit:2.0.3_cuda11.3_gpu) via docker in a cluster.
And the program it seems to be installed right.
I'm tring to run DeepMD on a single node with the followed specific :

• CPU - 2x Intel® Xeon® Processor E5-2680 v3, 12 core 2.1Ghz
• GPU - 2 x NVIDIA K40 su PCI-E Gen3 x16
  But when I run it (with one of deepmd-kit water example) the followed error is rise:
  RuntimeError: cudaGetDevice() failed. Status: initialization error

here below the full output:

2021-11-11 11:18:35.856752: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From /opt/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
/opt/deepmd-kit/lib/python3.9/importlib/__init__.py:169: UserWarning: The NumPy module was reloaded (imported a second time). This can in some cases result in small but subtle issues and is discouraged.
  _bootstrap._exec(spec, module)
Traceback (most recent call last):
  File "/opt/deepmd-kit/bin/dp", line 10, in <module>
    sys.exit(main())
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
    train_dp(**dict_args)
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 71, in train
    run_opt = RunOptions(
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/train/run_options.py", line 96, in __init__
    self._try_init_distrib()
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/train/run_options.py", line 191, in _try_init_distrib
    self._init_serial()
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/train/run_options.py", line 219, in _init_serial
    nodename, _, gpus = get_resource()
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/cluster/__init__.py", line 22, in get_resource
    return get_local_res()
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/cluster/local.py", line 48, in get_resource
    gpus = get_gpus()
  File "/opt/deepmd-kit/lib/python3.9/site-packages/deepmd/cluster/local.py", line 32, in get_gpus
    raise RuntimeError('Failed to detect availbe GPUs due to:\n%s' % decoded)
RuntimeError: Failed to detect availbe GPUs due to:
2021-11-11 11:18:43.528695: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-11-11 11:18:49.011946: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-11 11:18:49.017184: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-11-11 11:18:49.169964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: Tesla K40m computeCapability: 3.5
coreClock: 0.745GHz coreCount: 15 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 268.58GiB/s
2021-11-11 11:18:49.170860: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 1 with properties: 
pciBusID: 0000:82:00.0 name: Tesla K40m computeCapability: 3.5
coreClock: 0.745GHz coreCount: 15 deviceMemorySize: 11.17GiB deviceMemoryBandwidth: 268.58GiB/s
2021-11-11 11:18:49.170913: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-11-11 11:18:49.179178: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-11-11 11:18:49.179237: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-11-11 11:18:49.180961: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-11-11 11:18:49.181312: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-11-11 11:18:49.182172: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-11-11 11:18:49.183560: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-11-11 11:18:49.183780: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-11-11 11:18:49.187189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0, 1
2021-11-11 11:18:49.187233: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/deepmd-kit/lib/python3.9/site-packages/tensorflow/python/client/device_lib.py", line 43, in list_local_devices
    _convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)

Do you have any suggestion?

[TinacciL]

With the cuda version of 10.1 it seems to work.
But on the used example water/se_e2_a it is seems to run only on one GPU card and not two.
The program see both in the initialization.

[TinacciL]

Using the setup provided in the doc for run in an HPC now DeepMD is running on both the GPU.
Still do not understand why it not run on both the GPUs without mpi since is the same node.