Error of new compiled DeePMD-kit 2.2.5 on the DCU machine

[KawaSakiHakuButsu]

Hello everyone,

I have compiled DeePMD-kit 2.2.5 on the sugon platform with following environments

• CMake 3.21.0
• ROCm 4.0.1
• TensorFlow 2.5.0
• Developer Toolset 7.3.1
• HPC-X 2.7.4

Then I use the DP inference C++ interface based on the shared library libdeepmd_cc.so. Before performing the inference, I also load the required GNU 8.2.0 module. The inference process ran successfully on CPU, but met following errors on DCU (similar to the errors in #2655 and #1927)

DeePMD-kit WARNING: Environmental variable TF_INTRA_OP_PARALLELISM_THREADS is not set. Tune TF_INTRA_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit WARNING: Environmental variable TF_INTER_OP_PARALLELISM_THREADS is not set. Tune TF_INTER_OP_PARALLELISM_THREADS for the best performance. See https://deepmd.rtfd.io/parallelism/ for more information.
2023-11-02 14:27:42.125986: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  SSE3 SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-11-02 14:27:42.222218: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libamdhip64.so
2023-11-02 14:27:42.224345: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1747] Found device 0 with properties: 
pciBusID: 0000:26:00.0 name: Device 66a1     ROCm AMDGPU Arch: gfx906
coreClock: 1.7GHz coreCount: 64 deviceMemorySize: 15.98GiB deviceMemoryBandwidth: 953.67GiB/s
2023-11-02 14:27:42.353544: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library librocblas.so
2023-11-02 14:27:42.454869: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libMIOpen.so
2023-11-02 14:27:42.830965: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library librocfft.so
2023-11-02 14:27:43.233225: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library librocrand.so
2023-11-02 14:27:43.236464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2023-11-02 14:27:43.236841: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-11-02 14:27:43.236887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264]      0 
2023-11-02 14:27:43.236917: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0:   N 
2023-11-02 14:27:43.243746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14731 MB memory) -> physical GPU (device: 0, name: Device 66a1, pci bus id: 0000:26:00.0)
2023-11-02 14:27:43.913152: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2000074999 Hz
hip assert: hipErrorInvalidValue /public/home/hutaiping/JinB/software/deepmd-kit-2.2.5/source/op/prod_env_mat_multi_device.cc 1473
2023-11-02 14:27:44.996711: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at custom_op.cc:18 : Internal: Operation received an exception: DeePMD-kit Error: HIP Assert, in file /public/home/hutaiping/JinB/software/deepmd-kit-2.2.5/source/op/custom_op.cc:18
terminate called after throwing an instance of 'deepmd::tf_exception'
  what():  DeePMD-kit Error: TensorFlow Error: Internal: 2 root error(s) found.
  (0) Internal: Operation received an exception: DeePMD-kit Error: HIP Assert, in file /public/home/hutaiping/JinB/software/deepmd-kit-2.2.5/source/op/custom_op.cc:18
         [[{{node ProdEnvMatA}}]]
         [[o_energy/_29]]
  (1) Internal: Operation received an exception: DeePMD-kit Error: HIP Assert, in file /public/home/hutaiping/JinB/software/deepmd-kit-2.2.5/source/op/custom_op.cc:18
         [[{{node ProdEnvMatA}}]]
0 successful operations.
0 derived errors ignored.

I have verified hipMemcpyDeviceToDevice of ROCm 4.0.1 on DCU due to the above error messages

hip assert: hipErrorInvalidValue /public/home/hutaiping/JinB/software/deepmd-kit-2.2.5/source/op/prod_env_mat_multi_device.cc 1473

using a simple test. (Corresponding code attached in the bottom) However, there's nothing wrong about this process itself.

How can I solve the above errors? Thanks for every ideas and suggestions.

test.cpp

#include <iostream>
#include <hip/hip_runtime.h>

__global__ void test( float * a, float * b )
{
		const int idx = blockIdx.x * blockDim.x + threadIdx.x;
		a[ idx ] = b[ idx ] * 2;
		return;
}

int main(  )
{
	hipSetDevice( 0 );
	float * a( nullptr );
	float * b( nullptr );
	const int n_bl( 32 );
	const int n_th( 32 );
	float * a_h = new float[ n_bl * n_th ];
	float * b_h = new float[ n_bl * n_th ];
	hipMalloc( & a, n_bl * n_th * sizeof( float ) );
	hipMalloc( & b, n_bl * n_th * sizeof( float ) );
	for( int i = 0; i < n_bl * n_th; ++ i )
			b_h[ i ] = i;
	hipMemcpy( b, b_h, n_bl * n_th * sizeof( float ), 
			   hipMemcpyDefault ); 

	// test<<< n_bl, n_th >>>( a, b );
	hipMemcpy( a, b, n_bl * n_th * sizeof( float ), 
			   hipMemcpyDeviceToDevice ); 

	hipMemcpy( a_h, a, n_bl * n_th * sizeof( float ), 
			   hipMemcpyDefault ); 
	for( int i = 0; i < n_bl * n_th; ++ i )
		std::cout << a_h[ i ] << ' ';
        std::cout << std::endl;
	delete [  ] a_h;	
	delete [  ] b_h;
	hipFree( a );
	hipFree( b );
	return 0;
}

[njzjz]

It's like the same issue in #1801, and the bug may come from the old ROCm library. By the way, the default float type is double.

[xdzhu]

Sorry, I failed to solve the problem by myself. I asked help from the engineers, and they compiled the DeePMD-kit ROCm successfully. However, I didn't inquire the details about the compile strategy and process.

Thanks for your reply. I will also ask them for help. Could you please provide any details? Like:
GNU version
ROCm/DTK version
DeepMD-kit version
Python version
Tensorflow version

I'll appreciate it very much.

[KawaSakiHakuButsu]

You can tell them which version of DeePMD-kit you required, and the versions of other packages may relate to the DeePMD-kit version. (They compiled DeePMD-kit 2.2.0 for me. GNU 9.3.0, DTK 22.10.1, Python 3.7.13, TensorFlow 2.7.0+git67f0ade9.dtk2210)

[xdzhu]

Thank you very much. I will try this chain of version 2.2.0. I also noticed the higher version of DeepMD-kit requires higher version of Python, which may raise other problems in the Tensorflow with DTK.

[xdzhu]

Thanks again. Sorry to bother you beacuse there are still some extra questions.

1. In this morning I try with the DeePMD-kit 2.1.5 (higher verison is not compitable with python3.6, I use DeePMD-kit 2.1.5 with some modified cmake files from 2.2.0, see Fix compatibility with ROCm>=3.5.1; fix typo in hip neighbor_list).
   GNU 7.3.1, DTK 21.10, Python 3.6.8 and TensorFlow 2.7.0. And it seems can run now but the build variant in the job.err shows it's a cpu version. Is your build variant also cpu? (It is worthy mentioning that previously I was always trying with DP_VARIANT=rocm . It shows to be installed successfully but it cannot run.)
2. Although I installed horovod successfully, I still do not know how to horovodrun the dp train correctly. Now I can only run the serial execution. Could you share the SLURM job script which you run deepmd-kit on the sugon platform? I want to test the parallel training on DCU. Or have you already make the test? It's well appreciated if you could share some experiences. Thanks a lot.

[KawaSakiHakuButsu]

1. The DeePMD-kit 2.2.0 installed by the engineers is exactly ROCm version, which successfully leverages the DCU performing inference.
2. Sorry I haven't used Horovod so far, and can share few experiences for you. In fact, I don't train DP potential on sugon platform but perform inference through the C++ interface using DP potential trained on other platform. As for the inference, since 2.2.0 version, the function dp.compute (dp is a deepmd::DeepPot object) supports to infer the energies and forces of more than one structure, which is almost sufficient for me to improve the computational efficiency.

I'm grateful to discuss with you, but may not share you too much beyond my knowledge.