pretrain fails when image categories are similar

i want to use your work to perform few epochs pretrain on  my dataset,which contains sevceral similar vehicle categories.
So i load the imagenet-pretrained checkpoint and run another pretrain on my dataset,and it fails cause the loss not falling。
What's the reason? 