Low CPU usage while training

[youngjae142]

When I was using DP 1.3.3 version, "dp" process was using all CPUs for training. (Maybe OpenMP?)
Training starts as soon as running "dp train".

However, dp v2.0.2 uses only 1 CPU resource and it takes half an hour to start training. (Also when restarts from a ckpt.)
After starts the training, it still uses only 1 CPU.

I set environment variables like below.
export OMP_NUM_THREADS=64
export TF_INTRA_OP_PARALLELISM_THREADS=64
export TF_INTER_OP_PARALLELISM_THREADS=1

3 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train _____ _____ __ __ _____ _ _ _
4 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train | __ \ | __ \ | / || __ \ | | ()| |
5 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train | | | | ___ ___ | |__) || \ / || | | | ______ | | __ _ | |
6 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train | | | | / _ \ / _ | / | |/| || | | |||| |/ /| || |
7 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train | || || /| /| | | | | || || | | < | || |
8 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train |/ _| _||| || |_||____/ ||_|| __|
9 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train Please read and cite:
10 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018)
11 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train installed to: /home/grad/ssrokyz/src/deepmd-kit/_skbuild/linux-x86_64-3.8/cmake-in stall
12 [2021-10-12 22:37:49,429] DEEPMD INFO deepmd.entrypoints.train source : v2.0.2
13 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.entrypoints.train source brach: HEAD
14 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.entrypoints.train source commit: 5a84796
15 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.entrypoints.train source commit at: 2021-09-29 16:48:37 +0800
16 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.entrypoints.train build float prec: double
17 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.entrypoints.train build with tf inc: /home/grad/ssrokyz/.conda/envs/tf24/lib/python3.8/site-packages/tens orflow/include
18 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.entrypoints.train build with tf lib:
19 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options ---Summary of the training---------------------------------------
20 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options running on: pcs_gpu6
21 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options computing device: gpu:0 [2 ]
22 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options CUDA_VISIBLE_DEVICES: 0
23 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options Count of visible GPU: 1
24 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options num_intra_threads: 32
25 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options num_inter_threads: 1
26 [2021-10-12 22:37:49,430] DEEPMD INFO deepmd.train.run_options -----------------------------------------------------------------

[njzjz]

The log shows you are using a GPU instead

[youngjae142]

Do you mean that inactivate OpenMP and half an hour pending before training is perfectly normal? But it isn't in v1.3.3 (I'm still using this version of DP too). Below is the log of v1.3.3, and it uses all CPUs (6400%) and training starts without 1 second of pending. All training parameters are set identical. 1 # DEEPMD: _____ _____ __ __ _____ _ _ _ 2 # DEEPMD: | __ \ | __ \ | \/ || __ \ | | (_)| | 3 # DEEPMD: | | | | ___ ___ | |__) || \ / || | | | ______ | | __ _ | |_ 4 # DEEPMD: | | | | / _ \ / _ \| ___/ | |\/| || | | ||______|| |/ /| || __| 5 # DEEPMD: | |__| || __/| __/| | | | | || |__| | | < | || |_ 6 # DEEPMD: |_____/ \___| \___||_| |_| |_||_____/ |_|\_\|_| \__| 7 # DEEPMD: 8 # DEEPMD: Please read and cite: 9 # DEEPMD: Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018) 10 # DEEPMD: 11 # DEEPMD: ---Summary of the training--------------------------------------- 12 # DEEPMD: installed to: /home/grad/ssrokyz/src/deepmd-kit/_skbuild/linux-x86_64-3.8/cmake-install 13 # DEEPMD: source : v1.3.3 14 # DEEPMD: source brach: HEAD 15 # DEEPMD: source commit: 3a59596 16 # DEEPMD: source commit at: 2021-03-20 00:53:44 +0800 17 # DEEPMD: build float prec: double 18 # DEEPMD: build with tf inc: /home/grad/ssrokyz/.conda/envs/dp2/lib/python3.8/site-packages/tensorflow/include;/home/grad/ssrokyz/.conda/envs/dp2/lib/python3.8/site-package s/tensorflow/include 19 # DEEPMD: build with tf lib: 20 # DEEPMD: running on: pcs_gpu9 21 # DEEPMD: CUDA_VISIBLE_DEVICES: [0] 22 # DEEPMD: num_intra_threads: 64 23 # DEEPMD: num_inter_threads: 1 24 # DEEPMD: -----------------------------------------------------------------

[njzjz]

The operators on CPUs are all moved into GPUs now if you have a GPU.
The new version also adds an extra step to check if sel is enough, and this may pend for some time if you have a lot of data... We may consider adding an option to skip this step in the following version.

[youngjae142]

Is it possible to remove the time-consuming part which you mentioned manually from the source code?
It will be very helpful if you specify the file and lines to remove.
Just pointing it at a glance will be enough. I will check carefully.

[njzjz]

You can comment

deepmd-kit/deepmd/utils/neighbor_stat.py

Lines 73 to 104 in 5a84796

	for ii in range(len(data.system_dirs)):
	for jj in data.data_systems[ii].dirs:
	data_set = data.data_systems[ii]._load_set(jj)
	for kk in range(np.array(data_set['type']).shape[0]):
	mn, dt \
	= run_sess(self.sub_sess, [self._max_nbor_size, self._min_nbor_dist],
	feed_dict = {
	self.place_holders['coord']: np.array(data_set['coord'])[kk].reshape([-1, data.natoms[ii] * 3]),
	self.place_holders['type']: np.array(data_set['type'])[kk].reshape([-1, data.natoms[ii]]),
	self.place_holders['natoms_vec']: np.array(data.natoms_vec[ii]),
	self.place_holders['box']: np.array(data_set['box'])[kk].reshape([-1, 9]),
	self.place_holders['default_mesh']: np.array(data.default_mesh[ii]),
	})
	if dt.size != 0:
	dt = np.min(dt)
	else:
	dt = self.rcut
	log.warning("Atoms with no neighbors found in %s. Please make sure it's what you expected."%jj)
	if dt < self.min_nbor_dist:
	if math.isclose(dt, 0., rel_tol=1e-6):
	# it's unexpected that the distance between two atoms is zero
	# zero distance will cause nan (#874)
	raise RuntimeError(
	"Some atoms in %s are overlapping. Please check your"
	" training data to remove duplicated atoms." % jj
	)
	self.min_nbor_dist = dt
	for ww in range(self.ntypes):
	var = np.max(mn[:, ww])
	if var > self.max_nbor_size[ww]:
	self.max_nbor_size[ww] = var

[youngjae142]

It works like a charm! You literally saved my life. Thx!

[youngjae142]

That totally makes sense. Sincerely thank you for your answer!