Dimenet trainable parameters

[sudaksh14]

Hi PyG community,
I was trying to implement the dimenet model from PyG, but while the training my gradients are all nan.
I am not sure as to what parameters should be trained exactly, for now I just used model.parameters().

[rusty1s]

Thanks for reporting. Is it possible to debug this via with torch.autograd.detect_anomaly()?

[sudaksh14]

I am using the minmax scaler from sklearn and apply that directly to the pd df of my input data and ground truths separately.(2 different scalers)

[rusty1s]

Does the minmax scaler produces NaNs if one of the features is all the same? Can you check whether the forward pass already produces NaN values?

[EdisonLeeeee]

I don't think it is a necessary way to normalize the data in such a regression task. I've run the code with the minmax normalization, and the output of the model is close to zero while the ground truth y is much larger than the output (e.g., 76). Hence the gradient would become large and the loss comes to NaNs after a few iterations of gradient descent.

[sudaksh14]

In my data all the y coordinate is 0 due to molecules in same plane, hence it could be the issue I think.
I checked with np.argwhere(np.isnan(nodes)) that minmax scaler doesn't produce any nan values. Still I tried to just scale the x&z coordinates only(leaving y-coordinate) and the target values, then tried to train the network. It now gives me very high loss value (like in e+15).
So do you suggest dropping the scaling altogether as its leading to such high losses? Or is there any other way around

Thanks a lot! Really Appreciate your time and effort.

[EdisonLeeeee]

As the problem is related to the very high loss, or gradients in other words. Could you have a try on gradient clipping? For example:

loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), grad_norm) # grad_norm could be 1.0 or 2.0, it depends.