lookups in Language.vocab do not reproduce

[mpjanus]

How to reproduce the behaviour:

With this function:

import spacy
import numpy as np

nlp = spacy.load("en_core_web_md")

def get_similar_words(aword, top_k=4):
    word = nlp.vocab[str(aword)]
    others = (w for w in nlp.vocab if np.count_nonzero(w.vector))
    similarities = ( (w.text, w.similarity(word)) for w in others )
    return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_k]

First calls to the function reproduce:

# starting 'blank', results reproduce
print(get_similar_words('cat'))
print(get_similar_words('cat'))  # calling twice to show reproducability

Results in:

[('cat', 1.0), ("'Cause", 0.2827487885951996), ('Ol', 0.2824869751930237), ('you', 0.27984926104545593)]
[('cat', 1.0), ("'Cause", 0.2827487885951996), ('Ol', 0.2824869751930237), ('you', 0.27984926104545593)]

So far, so good. But after performing some lookup in the vocab, the results become different:

# Outcome changes if before the call, a (related) word is looked up in the dictionary
_ = nlp.vocab['dog']   # some lookup

print(get_similar_words('cat'))   # same call as before

Results in:

[('cat', 1.0), ('dog', 0.8220816850662231), ("'Cause", 0.2827487885951996), ('Ol', 0.2824869751930237)]

which is different from before ('dog' wasn't there). If you do another lookup on vocab (e.g. 'kitten') and repeat the call to the function,
the result also contains the last looked up word:

_ = nlp.vocab['kitten']
print(get_similar_words('cat'))

Result:

[('cat', 1.0), ('kitten', 0.9999999403953552), ('dog', 0.8220816850662231), ("'Cause", 0.2827487885951996)]

Very strange. Actually I would have expected the words that I looked up my self to be part of the initial results, but it seems those words are somehow not yet present or fully initialized with a vector until they are explicitly looked up.

Your Environment

• Operating System: Ubuntu 22.04.2 LTS
• Python Version Used: 3.10.12
• spaCy Version Used: 3.5.3
• Environment Information: in Jupyter notebook

[svlandeg]

Hi,

I definitely can see how this is confusing. Basically there's a few different objects at play. The vocab, the stringstore, and the vectors.

Somewhat unintuitively, the Vocab in spaCy is really mostly just a cache. And as you've hypothesized and demonstrated with your code snippet - "dog" simply isn't in the cache in the beginning, and thus not iterated over in your custom loop for w in nlp.vocab. You can see this clearly by printing "dog" in nlp.vocab which is False before your first lookup, and True after. But as you can also see, the vectors for "dog" and "kitten" are in fact available and make sense.

So what you could do, is to loop over the strings in the StringStore, which is accessible through nlp.vocab.strings, and which will have "dog" and "kitten" etc. But beware that this loop will be take a very long time. Instead, you might have a look at building on top of vectors.most_similar. We have some more detailed docs available here as well: https://spacy.io/usage/linguistic-features#vectors-similarity

I hope that helps!

[mpjanus]

Thank you for the clear answer. Can you elaborate a bit more on the cache. Will it only cache the last n entries or eventually all objects?
On my system, if I first forcefully iterate through the the strings and perform look ups like this
_ = [nlp.vocab[w] for w in nlp.vocab.strings] , I can than afterwards iterate as I did before in my get_similar_words function. But that will be a risky approach if the cache is not caching all look-ups.

I do recommend to clarify the caching behavior in the spacy docs at https://spacy.io/api/vocab . The example that I shared is based on solutions posted at various blogs (e.g. https://stackoverflow.com/questions/57697374/list-most-similar-words-in-spacy-in-pretrained-model), where people just complain about the results not being very good.

Anyway, thanks for the help and I will also try the vectors.most_similar and see what that brings.

Regards,
Michael

[mpjanus]

Hmm, I used the 'most_similar' as described here #10465 , but I am not convinced the results is correct:

words=['cat']

keyss, _, scoress = nlp.vocab.vectors.most_similar(np.asarray([nlp(word).vector for word in words]), n=10)
for word, keys, scores in zip(words, keyss, scoress):
    strings = [nlp.vocab.strings[key] for key in keys]
    print(f'word={word!r}')
    for string, score in zip(strings, scores):
        print(f'\tstring={string!r}, score={score!r}')

gives:

word='cat'
	string='Baeg', score=1.0
	string='dogsbody', score=0.8221
	string='Kebira', score=0.7866
	string='moggie', score=0.7724
	string='pet(s', score=0.733
	string='duppy', score=0.7244
	string='wolfdogs', score=0.6751
	string='Sniglet', score=0.6638
	string='bristlebird', score=0.6482
	string='Coypu', score=0.6317

The results from my prior implementation based on vector.similar (when prefilling the cache) makes more sense:

_ = [nlp.vocab[w] for w in nlp.vocab.strings]   # force caching all words (vocab is a cache)
def get_similar_words(aword, top_k=10):
    word = nlp.vocab[str(aword)]
    others = (w for w in nlp.vocab if np.count_nonzero(w.vector))
    similarities = ( (w.text, w.similarity(word)) for w in others )
    return sorted(similarities, key=lambda x: x[1], reverse=True)[:top_k]

sims = get_similar_words('cat')
for string, score in sims:
    print(f'\tstring={string!r}, score={score!r}')

Results in:

	string='cat', score=1.0
	string='meows', score=0.9999999403953552
	string='ragdoll', score=0.9999999403953552
	string='peeves', score=0.9999999403953552
	string='paw', score=0.9999999403953552
	string='sphynx', score=0.9999999403953552
	string='felid', score=0.9999999403953552
	string='cat-', score=0.9999999403953552
	string='kitten', score=0.9999999403953552
	string='miaow', score=0.9999999403953552

As far as I understand from the docs, both use cosine similarity. How can the result be so different?

[svlandeg]

Could you retry that with en_core_web_lg instead? Because the en_core_web_md model's vector tables are typically pruned to save memory (hence its "medium" (md) size instead of "large" (lg)).

When I run this:

    import numpy as np
    nlp = spacy.load("en_core_web_lg")
    words = ['cat']

    keyss, _, scoress = nlp.vocab.vectors.most_similar(np.asarray([nlp(word).vector for word in words]), n=10)
    for word, keys, scores in zip(words, keyss, scoress):
        strings = [nlp.vocab.strings[key] for key in keys]
        print(f'word={word!r}')
        for string, score in zip(strings, scores):
            print(f'\tstring={string!r}, score={score!r}')

I get very reasonable results:

word='cat'
string='cat', score=1.0
string='dog', score=0.8221
string='DEFINE', score=0.8214
string='cats', score=0.7866
string='kitty', score=0.7724
string='kitten', score=0.7655
string='kitt', score=0.7384
string='pet', score=0.733
string='catsup', score=0.7245
string='puppy', score=0.7244

[svlandeg]

Another recent discussion post that may interest you: #12953