v2.0.0-beta.0: Nan in summary histogram

[AlexanderOnly]

Hello,
I try to run v2.0.0-beta.0 with:
"loss": {
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0.02,
"limit_pref_v": 1".
And an error appears:
" tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: filter_type_all/bias_2_0_0_1/histogram "

But when I set {"start_pref_v": 0, "limit_pref_v":0}, it works. I was wondering why I can not train with virial, could you please help me what I should do?
Thanks,

( Also, it works with v1.3.3 )

[AlexanderOnly]

Some input as follows:

"model": {
"type_map": [
"Fe",
"O"
],
"descriptor": {
"type": "se_e3",
"sel": [
60,
80
],
"rcut_smth": 0.5,
"rcut": 4.0,
"neuron": [
25,50,100
],"activation_function":"tanh",
"resnet_dt": false
},
"fitting_net": {
"neuron": [
240,240,240
],"activation_function":"tanh",
"resnet_dt": true
}
},
"loss": {
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0.02,
"limit_pref_v": 1
},

[amcadmus]

It should be a bug in the tensorboard support.

Could you please try to switch off the tensorboard by tensorboard=false?

[AlexanderOnly]

It should be a bug in the tensorboard support.

Could you please try to switch off the tensorboard by tensorboard=false?

Thanks!
But when I set tensorboard=false, another problem occurs：

step rmse_val rmse_trn rmse_e_val rmse_e_trn rmse_f_val rmse_f_trn rmse_v_val rmse_v_trn lr
0 3.77e+01 4.44e+01 8.00e+00 9.15e+00 1.16e+00 1.38e+00 1.24e+01 1.36e+01 4.0e-03
100 nan nan nan nan nan nan nan nan 4.0e-03
200 nan nan nan nan nan nan nan nan 4.0e-03

[amcadmus]

Thanks a lot for reporting the bug! It is fixed by #685 and will soon be released in beta1.

[marian-code]

@AlexanderOnly Do you by any change have the whole error log/output? I would like to see the exact line that throws this error. This should be a consequence of numerical instability that can be encountered in the first iterations.

See: SO-1 or SO-2

[amcadmus]

Could you please try release v2.0.0-beta.1 that fixes a bug in training with virials, and let us know if you still encounters the issue either with or without Tensorboard? Thank you!