What are all of the differences between .validate() and .test()?

[jbuckman]

After .fit() concludes, I run first .validate(ckpt_path=None) and then .test(ckpt_path=None). I use the exact same code for both of them: validation_step is identical to test_step, validation_epoch_end matches test_epoch_end, and val_dataloader is identical to test_dataloader. My trainer also has limit_val_batches=1.0, running the whole dataset. Yet, the two values I get are dramatically different: test error is far lower, typically up to an order of magnitude. What is going on in the test phase that could cause this?

[awaelchli]

Maybe because there is some state that persists and influences one method after the other. Have you seen the same results if you switch the order? Maybe a bug on our side. Is it reproducible in our simple bug report model?

[jbuckman]

In the end it appears to have been some sort of seed issue -- I still don't understand quite what was happening, but seed_everything(1) fixed it.

[awaelchli]

a possible reason could be that your dataloader does not produce the same batches if you iterate over it twice