Why rmse_val and rmse_trn differ when --restart the training?

[Lewis-YL]

Hello everyone,

Why rmse_val and rmse_trn are different when using --restart or --init-model?

In my training output file "lcurve.out"
At step 1000000:
rmse_val: 4.55e-01
rmse_trn: 2.36e-01

If using the command:
dp train input.json --restart model.ckpt-1000000
At step 100000:
rmse_val: 2.92e+00
rmse_trn: 2.41e+00

If using the command:
dp train input.json --init-model model.ckpt-1000000
At step 0:
rmse_val: 9.19e+00
rmse_trn: 8.06e+00

[njzjz]

I cannot reproduce your results. Did you use the latest version?

[Lewis-YL]

I edited my local files following
#1692

Here I attached an example input files
example.zip

If you train some steps (e.g. 2000 steps) and stop, you get a rmse_val and rmse_trn in the lcurve.out.
If using commands
dp train input.json --restart model.ckpt-2000,
or
dp train input.json --init-model model.ckpt-2000,
The rmse_val and rmse_trn obtained at that checkpoint (2000 steps) are different from when stopped at that checkpoint.

My primary question is
Shouldn't they be the same?

This difference is small when there is only 1 training folder.
This difference can be an order of magnitude when I have 10 training folders.

[njzjz]

I think the problem is that the random state is not recovered. As a result, they visit the different data to test.

[njzjz]

I will track this issue in #1698.

[Lewis-YL]

Sounds good!