The training process is incomplete. One epoch can only execute part of it and then jump to the next epoch

[niumq]

I have encountered a bug, the training can be carried out normally in the training process, but the epoch can only be executed, so it will jump to the next epoch, and the training will be terminated when the training reaches the 11th epoch. The error is due to the data shape problem, please see the following figure for the specific content, I hope someone can help me, thank you very much!

[akihironitta]

I think there're two issues:

1. your progress bar shows that the loop ends in the middle of an epoch, and
2. your script raises ValueError at the 11th epoch.

For 1., I'm not sure unless you provide your script.
For 2., It's because the batch size is 1 and the last channel of the batch is 1. You need to always have batch_size>=2 when you're using batch normalisation. See this thread for detail.

[niumq]

Thank you for your reply.
For the first question, I found that the jump progress would be different with different Settings. For example, after I reduced the number of GPU, the jump progress is now 68%, which I used someone else's code, so I don't know where the problem might be.
The second problem didn't occur when I set drop_last to true in the Dataloader

[akihironitta]

For the first question, I found that the jump progress would be different with different Settings. For example, after I reduced the number of GPU, the jump progress is now 68%, which I used someone else's code, so I don't know where the problem might be.

Are you able to reproduce the behaviour with a simple script? https://github.com/Lightning-AI/lightning/tree/master/examples/pl_bug_report

The second problem didn't occur when I set drop_last to true in the Dataloader

👍

[niumq]

I'm new to PyTorch-Lightning, it's the first project I use, but I'll try to run it

[akihironitta]

@niumq By the way, could you paste your current script (trainer args, etc.)? I might be able to point out a possible cause.