batch normalization in Unet

[anais2390]

Hi everyone !

I would like to add batch normalization in monai UNet but I can't figure out how. I suppose I should use the argument norm but I don't understand how ? I tried norm=Norm.BATCH but this didn't work.

Can anyone give me an exemple ?
Thanks you for your help !

[wyli]

could you give more details, I had a quick test it seems the norm works fine with the latest codebase:

>>> from monai.networks.layers.factories import Act, Norm
>>> from monai.networks.nets import UNet
>>> net = UNet(2, 1, 1, (4,8,16), (2, 2), norm=Norm.BATCH)
>>> net(torch.zeros(1, 1, 64, 64))
tensor([[[[-0.3094, -0.3094, -0.3094,  ..., -0.3094, -0.3094, -0.3094],
          [-0.3094, -0.3094, -0.3094,  ..., -0.3094, -0.3094, -0.3094],
          [-0.3094, -0.3094, -0.3094,  ..., -0.3094, -0.3094, -0.3094],
          ...,
          [-0.3094, -0.3094, -0.3094,  ..., -0.3094, -0.3094, -0.3094],
          [-0.3094, -0.3094, -0.3094,  ..., -0.3094, -0.3094, -0.3094],
          [-0.3094, -0.3094, -0.3094,  ..., -0.3094, -0.3094, -0.3094]]]],
       grad_fn=<SlowConvTranspose2DBackward0>)
>>> 

[anais2390]

Hi @wyli, thanks a lot for your answer.

Actually I don't have any error when running UNet training with norm=Norm.BATCH but my training curves don't show any convergence, whereas using the norm default value, I have a nice convergence of the loss and MeanDice (I am running a segmentation training).

I run the test several times with and without norm=Norm.BATCH and I had every times, the same results.

It seemed very unlikely that the batch normalization would degrade the results that much, so I thought I had set it up wrong.

What do you think ?

[wyli]

I think maybe the batchnorm doesn't work, you can try other normalization with additional parameters such as

net = UNet(2, 1, 1, (4, 8, 16), (2, 2), norm=(Norm.GROUP, {'num_groups': 2, 'affine': False}))

[anais2390]

I'll try that, thanks @wyli .

This is confusing me, I though batch normalization was a sure option that was worth using for any problematic.

Could you explain how the Norm.GROUP is working ?

[wyli]

no problem, internally it's a thin wrapper of torch.nn.GroupNorm via:

MONAI/monai/networks/layers/utils.py

Lines 28 to 31 in 4b66746

	from monai.networks.layers import get_norm_layer
	g_layer = get_norm_layer(name=("group", {"num_groups": 1}))
	n_layer = get_norm_layer(name="instance", spatial_dims=2)

and

MONAI/monai/networks/layers/factories.py

Line 227 in 4b66746

return nn.GroupNorm