Proper Workflow for Evaluation in DDP Training Loop

[benjaminrwilson]

Hi maintainers,

What is the correct way to perform task evaluation after an epoch of the training loop in DDP? For example, I would like to do the following for an object detection task:

1. Train for one epoch on the training set
2. Evaluate on validation set
3. Gather loss and predictions over all gpus -- the latter used for computing mAP.
4. Resume training

Thanks!

Ben