SimCLR train NX-Tent loss dips at regular intervals (related to batch size)

[elizastarr]

I'm training an encoder using SimCLR on my own dataset, and my NX-Tent training loss suddenly dips to the (about) same value for just one iteration about every 25 epochs when I use a batch size of 256 (global batch size of 512 and 771 iterations/epoch) and every 10 epochs when I use a batch size of 350 (global 600 and 564 iterations/epoch). Any ideas for why this is happening would be very much appreciated!

Other settings:

• Adam optimizer with learning rate of 1e10
• Also happens with the LARS optimizer
• 10 epochs of linear warmup followed by cosine annealing for 90 epochs without restarts
• Also happens without a learning rate scheduler
• using SSL online evaluator
• Defined my own strong augmentations rather than using SimCLR's
• Shuffling the training data

Thanks!!

[tchaton]

@ananyahjha93 Any idea ?