Customizing check_val_n_epoch in Trainer using callbacks

[scho97]

Hello,

I am trying to customize the epochs on which the validation is performed. For instance, if I am training 20 epochs, I want to validate on [5, 10, 15, 16, 17, 18, 19, 20] epochs, where check_val_n_epoch=5 but, for the last 5 epochs, each epoch would be evaluated.

I tried my own callback function as below:

class CustomValEpochs(pl.Callback):
        def __init__(self, max_epochs, check_val_every_n_epoch, reserved_epochs):
            self.max_epochs = max_epochs
            self.check_val_every_n_epoch = check_val_every_n_epoch
            self.reserved_epochs = reserved_epochs

        def on_epoch_end(self, trainer, pl_module):
            if trainer.current_epoch <= self.max_epochs - self.reserved_epochs:
                if trainer.current_epoch % self.check_val_every_n_epoch == 0 and trainer.current_epoch != 0:
                    trainer._run_evaluate()
            else:
                trainer._run_evaluate()

validation_callback = CustomValEpochs(max_epochs=opt.epochs, check_val_every_n_epoch=opt.check_val_every_n_epoch, reserved_epochs=2)

but this yielded an AssertionError in trainer._run_evaluate() function.

Is there a way to customize the epochs at which we evaluate? Or can we change check_val_n_epoch in the middle of training (for instance, from 5 to 1)?

Version

python==3.7.11
torch==1.10.0
pytorch_lightning==1.4.9

[rohitgr7]

hey @scho97 !!

currently, there is no way to configure specific epochs for validation.
Either you can try this: #12097
or open an issue with the proposal.

[scho97]

@rohitgr7 Thank you for the reply!

I edited my code based on your suggestion. Is this what you are suggesting?

class CustomValEpochs(pl.Callback):
        def __init__(self, max_epochs, check_val_every_n_epoch, reserved_epochs):
            self.max_epochs = max_epochs
            self.check_val_every_n_epoch = check_val_every_n_epoch
            self.reserved_epochs = reserved_epochs

        def on_epoch_end(self, trainer, pl_module):
            if trainer.current_epoch <= self.max_epochs - self.reserved_epochs:
                if trainer.current_epoch % self.check_val_every_n_epoch == 0 and trainer.current_epoch != 0:
                    trainer._run_evaluate()
            else:
                trainer.validating = True
                trainer.reset_val_dataloader()
                trainer.val_loop.run()
                trainer.training = True

Do you have to reset the validation dataloader every epoch? Alos, is trainer.val_loop.run() essentially same as trainer._run_evaluate()?

[rohitgr7]

close!

class CustomValEpochs(pl.Callback):
        def __init__(self, check_val_epochs):
            self.check_val_epochs = check_val_epochs  # [5, 10, 15, 16, 17, 18, 19, 20]

        def on_train_epoch_end(self, trainer, pl_module):
            if trainer.current_epoch in self.check_val_epochs:
                trainer.validating = True
                trainer.reset_val_dataloader()
                trainer.val_loop.run()
                trainer.training = True

trainer = Trainer(check_val_every_n_epochs=0)  # to disable lightning running validation since it's configured in the callback

I haven't actually tested it, so give it a try.

but I'd rather suggest opening a feature request in the issues for it since the API might change in future and this callback will eventually break.

[scho97]

Thank you! I'll give it a try, and let you know.
Yes, opening a feature request seems to be a better idea in the long term. I'll open it in the issues.