How to evaluate every X steps?

[zorikg]

What is the best practice for performing evaluation on the dev and/or test set/s every X steps and not every epoch end?

In the documentation I saw the options of overriding validation_epoch_end and test_epoch_end, but could not find how to evaluate "more frequently".

Thanks!

[akihironitta]

Hi @zorikg! You can configure it through the trainer flag val_check_interval. e.g.

# check validation set 4 times during a training epoch
trainer = Trainer(val_check_interval=0.25)

https://pytorch-lightning.readthedocs.io/en/latest/common/trainer.html#val-check-interval

[zorikg]

Thank you @akihironitta! Exactly what I was looking for!

[zorikg]

@akihironitta another quick question - can I change this value dynamical from the lightning module? I want to start with val_check_intervalset to 1 and after a few epochs set it to 0.25.

[akihironitta]

@zorikg It may be a bit hacky and I believe its behaviour isn't guaranteed/recommended, but you could try overwriting the trainer's flag trainer.val_check_interval and calling trainer.reset_train_dataloader() to make sure that the related properties are updated in:

https://github.com/PyTorchLightning/pytorch-lightning/blob/8b4abe4edb6912abeb3906c48bb822a3681b08c4/pytorch_lightning/trainer/trainer.py#L1918-L1938

[zorikg]

Thank you @akihironitta for the detailed answer, I will try this!