How can we effectively handle embeddings from a long sequence?

[shiyegao]

I noticed that the paper averages the token embeddings, transforming (seq_len, embed_dim) into (embed_dim).

Would this operation discard positional information from the tokens?

What would be a better approach to fully utilize the (seq_len, embed_dim) representation?

[garykbrixi]

Yes using the full (seq_len, embed_dim) embeddng should provide more information