The performance gap between Install from source code and conda installation is very large (training speed performance)

[wangych6]

The performance gap between Install from source code and conda installation is very large (training speed performance)

I tried to improve the performance of deepmd.

After testing the performance of deepmd on different GPU and different tf versions, I found that the training speed directly installed from conda can be nearly 100% faster than the source code (pip install .). My installation method strictly follows the steps at https://github.com/deepmodeling/deepmd-kit/blob/master/doc/install/install-from-source.md#install-the-python-interface.

I would like to ask why the conda method is faster, is the code different from the github version? Or added a special compilation method, can you tell me?

[njzjz]

The performance difference comes from NumPy. Anaconda NumPy uses MKL and pip NumPy uses OpenBLAS. But I don't know the reason why OpenBLAS has performance degradation.

[wangych6]

@njzjz Thank you for your reply. I made a profile and found that the biggest performance difference is in the kernel launch, but I don't understand why there is a big difference in the kernel launch time?

[njzjz]

You can try OPENBLAS_NUM_THREADS to limit NumPy threads.