Does mixed precision only benefit double-precision models?

[cihe13375]

Hi! I'm trying to run two water examples se_e2_a and se_e2_a_mixed_prec, and test the effect of mixed precision settings on MD performance. I used a script like this to train models:

eval "$(/path/to/deep-md/bin/conda shell.bash hook)"
export OMP_NUM_THREADS=4
export TF_INTRA_OP_PARALLELISM_THREADS=4
export TF_INTER_OP_PARALLELISM_THREADS=1
export DP_INTERFACE_PREC=low     # comment this for double precision
export DP_ENABLE_MIXED_PREC=fp16 # only set for se_e2_a_mixed_prec

module load cuda/10.2 # load cuda on a cluster
dp train input.json
dp freeze -o graph.pb
dp compress -t input.json -i graph.pb -o graph_compressed.pb

Then I run the example input in the lmp folder, and replaced the original input file with a larger supercell (24000 atoms, so that the hours/ns scales linearly with system size); I run 1000 steps but the ns/day seems to be same as running only 1 step. For single prec models I exported LAMMPS_PLUGIN_PATH="/path/to/deep-md/lib/deepmd_lmp_low", and for double prec its deepmd_lmp. I'm getting following results on a single RTX 2080:

precision	mixed_prec	ns/day
double	no	~0.1
double	yes	~0.6
single	no	~0.8
single	yes	~0.8

So it seems that the mixed precision only benefit (originally) double prec models, and it has no effect on performance if DP_INTERFACE_PREC=low is set. Also it seems that single precision outperforms mixed precision in this case --- so does it mean we should not use mixed precision, at least on consumer-level GPUs (since their fp64 performance is weak and fp32 with no mixed_prec is better)?

[wanghan-iapcm]

@denghuilu could you please explain?

[ipe-zhangyz]

Maybe the DP_ENABLE_MIXED_PREC=fp16 only have effect on the neural network part. However, when you set DP_INTERFACE_PREC=low, the custom op will use fp32, and when DP_INTERFACE_PREC is not set, the default precsion used in custom op is fp64, which may be much slower than fp32 and have greater influence on the performance than the mixed precsion in neural network, thus the single precision without mixed precision outperforms double precision with mixed precision. Actually I'm very confused why there are so many ways to set the precisions of different parts of the DP computation. As I asked in #1860, if the custom op must use fp64, why do they have fp32 implementations? If it is not the truth, why the SC20 paper test the accuracy of mixed precision only in cases of fp64 custom ops?