SwinUNETR's Parameter Selection

[amir-sanaat]

Hello

For an image-to-image translation task how we can set SwinUNETR's parameter for better training?
My mean by parameters are:

"norm_name='instance', drop_rate=0.5, attn_drop_rate=0.5, dropout_path_rate=0.0, normalize=True, downsample='merging' "

For example how "drop_rate" and "attn_drop_rate" can impact results? What values could be selected as a safe options?

Thanks

[tangy5]

Hi @amir-sanaat , thanks for the interest of Swin UNETR, regarding your question.
Drop_rate or dropout rate is to prevent network from overfitting, if your dataset is large, you probably want lower drop_rate, or set to 0. If your dataset is small you could set it higher, but not too high, such as 0.2. attn_drop_rate is to highlight the informative region for improving the recognition power of the model. 0 for the most discriminative region 1 on the contrary. Drop path also named Stochastic Depth which is a technique to “deactivate” some layers during training. If you are using a "small" model, drop path set to 0 or 0.1 are preferred, larger model can have larger drop path rate, such as 0.2.

Thanks

[amir-sanaat]

Thanks @tangy5
It was very helpful.