If IMS_PER_BATCH increases, the overall training time will increase. Shouldn’t it be decreased?

[ahong007007]

When I set IMS_PER_BATCH = 32, the training takes 2 days. When I set IMS_PER_BATCH = 128, the estimated training time takes 7 days, which feels very unreasonable, but other conditions have not changed, just change IMS_PER_BATCH。

Please tell me, how does IMS_PER_BATCH affect the total training time? Thank you！

[sukjunhwang]

IMS_PER_BATCH is literally how many images are going to be fed into a batch.
You should change the iterations accordingly, and I add an example below.

Case 1: IMS_PER_BATCH=32, MAX_ITER=90000
total number of input images for training = 32 * 90000 = 2880000

Case 2: IMS_PER_BATCH=128, and you want the same setting as Case 1.
You want 2880000 images to be fed for training, thus MAX_ITER should be 2880000 / 128 = 22500.

To summarize, MAX_ITER should be scaled by 1/K if IMS_PER_BATCH increased xK.
For more information about learning rate and iteration configurations, please refer the paper https://arxiv.org/abs/1706.02677