How to do contrastive learning with accumulate_grad_batches?

[yipliu]

I have found an excellent solution for contrastive learning with DDP. However, a more difficult scenario is to use accumulate_grad_batches and DDP for contrastive learning.

For a clearer discussion, suppose I am training with accumulate_grad_batches=4 with two GPUs.

Anyone can help me?