How much of loss function and RMS error are accurate?

[Zhang-pchao]

I found that after the order of magnitude reaches 10^-2, the errors are difficult to decrease.

For the general model (such as a water molecule model in CPC 228 (2018) 178–184, the RMS energy and force errors over the whole testing set are 2.8 × 10^−2 eV and 2.4 × 10^−2 eV/Å, respectively), how much of loss function and RMS error ("lcurve.out" file) are accurate enough?

[amcadmus]

One may check the standard deviation (std) in the labels. In general, if the RMSE is smaller than the std in labels, we can say that the DP model has accuracy.

[AnuragKr]

@amcadmus Following your suggestion I tried to create script for calculating standard deviation --

Script for Force

 import numpy as np

force_data = np.load("../../data/water/ice_triple_II/test/data_10/set.000/force.npy")

no_of_frames = force_data.shape[0]

#Initialization
var = np.zeros(3)
mean = np.zeros(3)
weightage = 1/no_of_frames

#Variance Calculation
for idx in range(0,no_of_frames):
    force_mean_per_frame = np.mean(force_data[idx],axis=0)
    force_var_per_frame = np.var(force_data[idx],axis=0)
    var += weightage * (force_var_per_frame + np.square(force_mean_per_frame))
    mean += weightage * force_mean_per_frame


#Standard Deviation Calculation
std_dev = np.sqrt(var - np.square(mean))
print(std_dev)

Script For Energy --

import numpy as np

ener_data = np.load("../../data/water/ice_triple_II/test/data_10/set.000/energy.npy")

no_of_frames = ener_data.shape[0]

#Initialization
var = np.zeros(1)
mean = np.zeros(1)
weightage = 1/no_of_frames

#Variance Calculation
for idx in range(0,no_of_frames):
    ener_mean_per_frame = np.mean(ener_data[idx])
    ener_var_per_frame = np.var(ener_data[idx])
    var += weightage * (ener_var_per_frame + np.square(ener_mean_per_frame))
    mean += weightage * ener_mean_per_frame


#Standard Deviation Calculation
net_var = var - np.square(mean)
std_dev = 0
if net_var > 0:
    std_dev =  np.sqrt(net_var)

print(std_dev)

Output for my system test data --

Std Dev --
Force -- [0.6721139 0.6721139 0.6721139]
Energy -- 0

RMSE --
Energy RMSE: 2.986803e-02 eV
Energy RMSE/Natoms: 1.037084e-04 eV
Force RMSE: 2.086066e-02 (0.02086066) eV/A

First thing I want to ask -- whether my script correct for calculating the standard deviation? In the case of force having three coordinates can we take the min of the three-axis value into consideration?
If there will be multiple systems I will take the weightage of different std dev values.

I followed this link for calculating std dev.

Second -- My RMSE of test data for force is lesser than std dev, so I can say my DP model is accurate.