What does it mean when the vector for a token is all 0's?

[moonman239]

Sometimes, when I call "vector" on a token I get an array of all 0's.
For example,

>>> import pandas as pd
>>> import spacy
>>> import pandas as pd
>>> import numpy as np
>>> nlp = spacy.load("en_core_web_lg")
>>> doc1 = nlp("pics from our waltdisneyworld2022 trip")
>>> doc2[4].vector

will be all zeros.
Am I to understand that this is because the token is not in spaCy's corpus?

[koaning]

It means that there is no entry with a vector in spaCy's vocabulary. You might enjoy this Twitter thread for more details.

[moonman239]

I figured as much (and have a tentative plan to compensate)

[koaning]

What do you mean by "compensate"? When you train a spaCy pipeline it will still be able to learn from tokens likewaltdisneyworld2022. So if you're training your own system you may not need to "compensate".