MatMul Bottleneck

[npiroozan]

Hello,

I've noticed a significant bottleneck regarding running DeePMD with multiple workers on a single node. This specifically concerns the MatMul operator. Using an optimized release of Tensorflow (oneDNN in place of eigen), I had expected to see MKL being called with the execution of Tensorflow operators such as MatMul. However, this is not the case.

I understand that MatMul is used in place of GEMM due to your use of model compression to allow for large speedups in the inference phase.

In your experience, do you know of ways to speed-up MatMul's compute time?

Thank you very much for your time.

[njzjz]

oneDNN doesn't support FP64 (the default setting of the deepmd-kit). Have you set the precision to FP32?

[npiroozan]

Thank you for the reply. I see, no I have not set the precision to FP32. Please forgive me, but how would the precision be changed?

Thank you very much.

[njzjz]

See https://docs.deepmodeling.com/projects/deepmd/en/master/train/train-input.html. Search "precision".

[npiroozan]

Understood, thank you. One observation, if I may:

"Precision" is not accepted as an input under "Model", but it is accepted under "Descriptor" and "Fitting Net".

When placed as an argument under "Model", the error "Not permitted in strict mode" applies".