Strange running_var Values in RadImageNet Pre-trained Weights PyTorch #20
Description
Hello,
I am using the pre-trained weights from RadImageNet (DenseNet121) for my model training in PyTorch, but I have encountered an issue with the running_var values in certain layers. Specifically, the running_var values for layers such as denseblock4.denselayer13.norm1.running_var are significantly high: Min: 14.656, Max: 56296, Mean: 3797.477.
These extreme values in the RadImageNet weights seem unusual and are affecting the stability of the model during validation (i.e., producing extreme logits for certain batches).
I would like to understand the following:
- Is this behavior expected, or could it be related to a specific issue in the training or weight initialization process?
- How were the images in RadImageNet normalized prior to training? There seems to be inconsistencies for both PyTorch (image = (image-127.5)*2 / 255) and Tensorflow (rescale=1./255) implementations and I would like to know exactly what needs to be employed.
Has anyone experienced this with DenseNet121 in PyTorch?
Any insights or guidance on this would be greatly appreciated!
Thank you!