Vectors corresponding to other language

[sonynavdeep81]

Hi. I have a very basic question. I am creating a blank spacy model with 'en' language. As we know, that it is a blank model, it does not contain token vectors. Now I add the tok2vec component using add_pipe from the en_core_web_sm pipeline and thus now it contains token vectors for english text. The real surprise for me is that all these vectors corresponds to english language. Now if rather than providing the english text, I pass punjabi text (another language for which the support has not yet been provided by spacy) then also it returns the vectors as can be seen in the below code and corresponding output;

Although we are using english language then how does it contain vectors corresponding to punjabi language? Please explain.

[adrianeboyd]

This is a confusing legacy backoff behavior from doc.tensor to doc.vector.

What you're seeing here aren't static word vectors like from word2vec or glove, but the context-sensitive tensors from the tok2vec component. The tok2vec component is able to generate a vector for any token, but they're not really useful for anything other than the following pipeline components (tagger, parser). They're not particularly good for word similarity. See the first yellow warning box here: https://spacy.io/usage/linguistic-features#vectors-similarity

If you download en_core_web_md or en_core_web_lg, you'll see static word vectors under token.vector with OOV (all zero) vectors for unknown words like Punjabi tokens.

[sonynavdeep81]

Thanks for the nice explanation. Can you please explain

1. What are context-sensitive tensors and how are they generated automatically?
2. What purpose do the context-sensitive tensors serve (What is their need)?
3. How are they useful for (tagger, parser)?
4. If we load static vectors then will tok2vec still generate vectors? If yes, what for?

Actually I am a beginner, so I am sorry if I am asking really basic questions. Thanks.

[polm]

Please read the docs on pipelines and embeddings, which should answer your questions.