Continuing model fit with new data

[jacksund]

We've trained our model using a small set of data, and now we want to continue the training after introducing some new data. However, the dp train --restart and dp train --init-model commands are working just fine when using the original data, but fails when we introduce the new data.

Our traceback error makes it look like the network is trying to change when we introduce the new data:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [1,720] rhs shape= [1,704]

One key difference between datasets is the temperature that simulations were ran at -- but would this lead to new network architectures? If so, is there a way to fix our architecture to prevent this error?

[wanghan-iapcm]

Could you paste the complete error log? Thanks.

Could you also check the input file is consistent with the one used before.

[jacksund]

Actually, I was just able to fix it!

We were using the exact same input.json for both attempts, but the issue was with the default value for the sel parameter (of se_e2_a). Because the default value was auto, we were getting different values upon restarting (bc of the new dataset). By setting sel in the input.json, we can now continue jobs without any issues.

[linkx-stu]

I'm having this problem, how did you solve it?
I also set sel to auto. if I set sel manually, this error will not happen.