Does PL train and validate at the same time?

[arunpatro-meesho]

I wasn't able to find this on github -> https://forums.pytorchlightning.ai/t/dose-pl-validate-and-train-at-the-same-time/1362/2

I am logging step wise in training_step and validation_step

but i see this while training ->

Epoch 0:  82%|████████▏ | 49/60 [00:59<00:13,  1.22s/it, loss=0.23, v_num=19, train_loss=0.0837] 
Validating: 0it [00:00, ?it/s]
Validating:   0%|          | 0/11 [00:00<?, ?it/s]
Epoch 0:  85%|████████▌ | 51/60 [01:05<00:11,  1.28s/it, loss=0.23, v_num=19, train_loss=0.0837]
Validating:  18%|█▊        | 2/11 [00:05<00:20,  2.25s/it]
Epoch 0:  88%|████████▊ | 53/60 [01:05<00:08,  1.24s/it, loss=0.23, v_num=19, train_loss=0.0837]
Validating:  36%|███▋      | 4/11 [00:07<00:09,  1.38s/it]
Epoch 0:  92%|█████████▏| 55/60 [01:09<00:06,  1.26s/it, loss=0.23, v_num=19, train_loss=0.0837]
Validating:  55%|█████▍    | 6/11 [00:09<00:06,  1.25s/it]
Epoch 0:  95%|█████████▌| 57/60 [01:09<00:03,  1.23s/it, loss=0.23, v_num=19, train_loss=0.0837]
Validating:  73%|███████▎  | 8/11 [00:10<00:02,  1.11it/s]
Epoch 0:  98%|█████████▊| 59/60 [01:13<00:01,  1.24s/it, loss=0.23, v_num=19, train_loss=0.0837]
Validating:  91%|█████████ | 10/11 [00:13<00:01,  1.01s/it]
Epoch 0: 100%|██████████| 60/60 [01:13<00:00,  1.23s/it, loss=0.23, v_num=19, train_loss=0.0837, val_loss=0.757]

How can the model start validating before training has completed?

[rohitgr7]

by default, it validates after each epoch given that you can configured validation using validation_step and val_dataloader.
do you need to validate the model right after training is done?

[arunpatro-meesho]

Also, I have only done logging for train_epoch_end and no logging in validation_step, yet this still occurs. What is the right way?

[rohitgr7]

Isn't that the right way? To validate the model right after a training epoch?

depends upon the user's use-cases but in general, yes and PL does that.

okay got it now.
the main progress bar represents total train + val steps.. so when the new validation bar comes up, it means that the training is done and both the main progress bar and validation progress bar progress for the validation steps.

[arunpatro-meesho]

Is there any way to seperate it? Lets say I want to see one progress bar for training and one for validation? This can appear quite confusing.

[rohitgr7]

you might need to make few changes to make this compatible

also there is a tracking issue: #12623

[arunpatro-meesho]

thanks. will look out for the issue.