NaN Loss

[ico1036]

Dear,

I tried the TopLandscape sample following the raedme.md with little modification due to my limitied RAM size.
However, I have encountered some issues during the training process, including NaN loss, low accuracy for each batch (4-50%), and NaN values in the prediction score output. It seems that the training process may not be working as expected, and you suspect that the YAML file might be the cause.

Could you give me some comments about my situation?

Here is my executer code (I just clone your git) https://github.com/JW-corp/J.W_AI/blob/main/particle_transformer/train_TopLandscape.sh

Best,
Jiwoong.

[ico1036]

I found that "--fetch-step 0.01" makes all input variables as NaN and NaN loss