Trainer Issue: Validation does not occur after initial validation

[PeterM18]

I have run my model with Trainer having no validation check epoch / interval parameters, with check_val_every_n_epoch=1, and with both check_val_every_n_epoch=1 and val_check_interval=1.0. The result is always the same: When my model runs, it starts with a validation step and then it does all the training steps up to max number of epochs, with no validation step in between.

I'm guessing the explanation is that Lightning is not set up to handle the custom dataset class I am using. My dataset generates data on the fly based on the current model at the start of each batch. I cannot say in advance how many observations will be available for an epoch. I've set dataset to return an arbitrary, huge number for the len function so that the model keeps running until my code in dataset sees the end of the data and triggers the end of the epoch.

If check_val_every_n_epoch and val_check_interval depend on 'len' from dataset, I'm not surprised validation is never triggered. Maybe it would be better if one of them, say check_val_every_n_epoch, triggers when lightningmodule receives an end of epoch signal rather than being dependent on the assumption that dataset lengths are fixed?

Anyway, I'll be trying to manually run validation in lightningmodule.

[PeterM18]

Manual validation works.