Shouldn't transfare learning of descriptors lead to much faster learning time?

[omaraek]

Hi all,
I was expecting that by freezing the descriptors I would reduce significantly the time required for the training but what I get is about 10% speed up only.

For this test, I am using the water/se_atten example, in which I changed "axis_neuron" to 40.

If I set "descriptors/trainable" to false I get the following result:

batch    5000 training time 323.71 s, testing time 0.11 s
batch   10000 training time 321.40 s, testing time 0.12 s
batch   15000 training time 322.44 s, testing time 0.11 s

while with "descriptors/trainable" set to true:

batch    5000 training time 365.16 s, testing time 0.09 s
batch   10000 training time 362.45 s, testing time 0.11 s
batch   15000 training time 362.41 s, testing time 0.11 s

By fixing a large portion of the trainable parameters shouldn't I get more speed up?

I am using 4 gpus (Tesla V100), cuda 10.1, TF 2.3.0 and deepmd v2.2.1 (installed with pip)
I also have observed similar behavior with the example se_e2_a/.

Thanks

Omar

[njzjz]

In the example, the number of nodes in the embedding network of descriptors is only 1/4 of that in the fitting network ((25+50+100)/(240+240+240)). So I think it's expected.

[omaraek]

Thanks for your comment. To verify that I set fitting network to [40, 40], so I would expect a speed up of about 80%. Yet I get the same numbers as before.

[njzjz]

80% is not expected. The OPs during training are divided into three parts:

• Original OPs to compute energy;
• The gradient OPs to train energy and compute forces;
• The second gradient OPs to train forces.

Here when you disable the training, the original OPs and the gradient OPs are still computed, so there should be, at most, a speed-up of about 1/3.

I suggest profiling the timings like #942 to get the details. The profiling tool can be enabled by setting enable_profiler to true.

[omaraek]

80% is not expected. The OPs during training are divided into three parts:

* Original OPs to compute energy;

* The gradient OPs to **train energy** and compute forces;

* **The second gradient OPs to train forces.**

Here when you disable the training, the original OPs and the gradient OPs are still computed, so there should be, at most, a speed-up of about 1/3.

It is true, but if a part of the network is fixed, in principle you need to compute the gradient of the fixed part only once at the beginning of the training process. Am I wrong?

I suggest profiling the timings like #942 to get the details. The profiling tool can be enabled by setting enable_profiler to true.

Okay, I will do it and let you know. Thanks for your helpful response

[njzjz]

It is true, but if a part of the network is fixed, in principle you need to compute the gradient of the fixed part only once at the beginning of the training process. Am I wrong?

It's a function of input coordinates and not a constant.