overfitness to first batch issue

[mfarazi1991]

Hi,

I have a shape correspondence task with a dataset consisting of 100 shapes. Although I have dataloader "shuffle = True", after testing my model, it highly overfits to the first batch (batch size of one) or data sample. In other words, my test accuracy is high for the first sample and much lower for the rest. So here is the test accuracy for my 20 test samples:

accuracy: 0.6181
accuracy: 0.5443
accuracy: 0.5376
accuracy: 0.5251
accuracy: 0.5210
accuracy: 0.5014
accuracy: 0.4616
accuracy: 0.4720
accuracy: 0.4741
accuracy: 0.4778
accuracy: 0.4814
accuracy: 0.4782
accuracy: 0.4676
accuracy: 0.4705
accuracy: 0.4721
accuracy: 0.4744
accuracy: 0.4756
accuracy: 0.4768
accuracy: 0.4784

Any thoughts would be appreciated.

[rusty1s]

Is that also the case when setting shuffle=True in the test dataloader?

[mfarazi1991]

Hi Matthias,

Sorry to bring up this topic again. I wonder if you know how can I learn my model stage-wise. Meaning, first learn the U-net (first layers) by making the second part of my model fixed. Then after converging, I add a second part of the network for whole network training and fine-tuning. I do not want to learn the first part save it and then use it as a pre-trained model (i want it to be end-to-end).

[rusty1s]

You can set requires_grad_(False) for any parameters of model components you do not want to train:

for param in self.conv1.parameters():
    param.requires_grad_(False)

Do not forget to enable it again in the second part :)

[mfarazi1991]

Thanks for getting back to me. I tried that and unfortunately, I can freeze my CNN layer but not the activation function and normalization layers I have. So, relu cannot be disabled after those layers. I have no idea how we can disable these activation functions.

[rusty1s]

ReLU has no parameters, so there is no need to disable anything there.

[mfarazi1991]

Thank you, my problem was my own normalization blcok. Solved!