Multi-gpu training lag time between epochs

[OmarAshkar]

Hi,

I have been using hovernet tutorial here https://github.com/Project-MONAI/tutorials/tree/main/pathology/hovernet

Running on multi-gpu distributes the data and successfully shorten the training time. However, between each epoch there is 20 mins gap. The delay with single gpu is about 2 minutes only.

I do believe this is metrics/evaluation calculations. Is there is anyway to solve or debug that?

CC. @KumoLiu

Thanks in advance!