The significance of position_embeddings

[Lnan1615]

I cann't understand what the point of adding position_embeddings to the sequence is. For image blocks, when 2d or 3D images are transformed into 1d sequence or 1d sequence is transformed into 2D or 3D images, the operation is carried out in strict order, and the sequence sequence is not disrupted later, so it seems that position embedding of the sequence is no longer needed. In addition, I didn't understand the calculation method of position_embeddings. Could you give me a general explanation?

[wyli]

I assume you are talking about the vision transformers. The position embeddings are trainable position representations instead of hard-coded position indices.

MONAI/monai/networks/blocks/patchembedding.py

Line 99 in c9302e4

self.position_embeddings = nn.Parameter(torch.zeros(1, self.n_patches, hidden_size))

cc @ahatamiz

[ahatamiz]

Hi @wyli and @Lnan1615

In addition to the above, the position embedding is particularly used to retrain the spatial information with respect to each token in an input sequence. In a conventional CNN, the inductive image bias mitigates the need for such a mechanism, but in a ViT model ( or generally transformers), the locations of token need to be specified. This can be done via hard-coded (fixed) or trainable position embeddings. The ViT and UNETR both use the latter.

Thanks