Batch norm parameters are not updating

[sfarkya]

Hello,

I tried to replicate your results for Resnet50+LSTM on UCF101 data. The reported performance in the default setting is 80.20 and I got 80.30. However, I think there's a bug in training. The batch norm parameters (mean and variance) are not updating during the training, the model is using moving mean and variance from the loaded model (imagenet weights for resnet). Since, the params are not learnable params you need to put explicitly collect those update params and put control dependency on that. I think this will improve performance though I haven't run this change.

If this is true then this might improve performance for all the models and combinations.