How to hold 'validation_step' until training one epoch finished

[JongbinWoo]

Hi.
It seems like validation_step runs simultaneously at the end of training.
As soon as validation_step start, the percentage of gpu memory allocated is shooting up and RuntimeError: CUDA out of memory occur.
How can i fix it?
Thanks in advance!

[carmocca]

validation_step runs simultaneously at the end of training.

Validation is considered part of the fitting procedure but it never runs concurrent to training.

As soon as validation_step start, the percentage of gpu memory allocated is shooting up and RuntimeError: CUDA out of memory occur. How can i fix it?

It's definitely caused by a bug either on your end or ours. Can you try and reproduce it?

You can adapt https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pl_examples/bug_report_model.py to do it

[JongbinWoo]

Sorry. That was my mistake. I thought it runs concurrently because progress bar goes up simultaneously.
In validation step, I was doing Seq2Seq generation with num_beams=4. I think this is a cause of the error.
Thanks for answering!!