the loss of pretrain

[blackdyc]

When I used the HyperGlobal-450K dataset to reproduce the pre training process of the code in your paper, during the training of the spatial subnet, the loss rapidly decreased to around 0.005 after the first two epochs. I want to know if this is normal？

[DotWang]

@blackdyc that’s normal. In unsupervised training, the loss value is only for reference.

[blackdyc]

Thank you for your reply. I would like to ask how to determine if the model has converged. I have loaded the weight file of the spa vit base that you published, and the loss in the first epoch is around 0.002. Does this mean that the loss of the model has decreased so slowly in 1600 epochs? (0.005 to 0.002 took 1600 epochs)

[DotWang]

@blackdyc No. Even if you load our released weights, the relatively large learning rate in the first epoch will already change them significantly, so the loss curve is not simply a slow decrease over 1600 epochs. The choice of 1600 epochs is just an empirical setting. (following MAE)