Design of U-Net

[dongyang0122]

Hi, I am wondering how the ADN (act--dropout-norm) is designed in the current U-Net implementation as follows.
https://docs.monai.io/en/stable/_modules/monai/networks/nets/unet.html

I would like to add dropout layers in another network. What is the best practice to place dropout layers between other layers? Thanks!

@Nic-Ma @ericspod

[Nic-Ma]

Hi @ericspod ,

Could you please help share some guidance about this question?

Thanks in advance.

[wyli]

It might be useful to include the adn_ordering option from convolution

MONAI/monai/networks/blocks/convolutions.py

Line 110 in 8f5c992

adn_ordering: str = "NDA",

in the unet, such as:

MONAI/monai/networks/nets/unet.py

Lines 240 to 243 in 8f5c992

	act=self.act,
	norm=self.norm,
	dropout=self.dropout,
	bias=self.bias,

this will bring a bit more flexibilities for activation/dropout/normalisation... cc @ericspod

[ericspod]

We can enhance the class to add this argument, that would solve changing the order. @dongyang0122 what do you mean by "add dropout layers in another network"?

[dongyang0122]

@ericspod I would like to add dropout layers into another network architecture. not sure where to add those layers (before or after activation layers). I am wondering if you have any guideline or reference about the design order of layers in the U-Net implementation. Thanks!

[ebrahimebrahim]

@dongyang0122
throwing out some links that might be helpful:

• NDA is the default ordering used in convolution blocks, which are the ones used in the UNet
• NDA is also the default ordering chosen by the ADN block
• the documentation for the ADN block may be useful

[ericspod]

UNet was also recently updated by @wyli to implement his proposed addition.

I would also recommend the default NDA ordering.

[dongyang0122]

@ebrahimebrahim thank you for sharing the links! It would be better if there are any existing experiments or numbers supporting the design.