Vit hidden size

[victolee0]

The default hidden size value of monai.networks.nets.ViT is 768 now.

In my opinion, the reason why ViT developers decided the hidden layer value to 768 is because the patch size they used is 16 and the channel of patch is 3. (3x16x16)

so I think, ViT hidden size should be the product of patch size and channel

For example)
if the patch size is (16, 16, 16) and the image is grayscale in 3D Vision Transformer Model,
hidden size should be 16*16*16=4096.

Is this right?

[wyli]

I think it's a free system parameter, and 4096 for single channel 3d is a logical choice...(cc @ahatamiz @finalelement )

[ahatamiz]

Hi @victolee0

Thanks for the comment. Yes, theoretically a hidden size of 4096 is desirable, but I kept it to 768 to manage the computational requirements. In addition, my experiments with larger hidden sizes (e.g. 1536) did not show significant improvement, hence 768 seemed an optimal choice.

Best

[victolee0]

Thank you for your help.
I could understand it through your efforts.

Have a nice day!