"number" still not in spacy's vocab

[minimario]

There's a relevant issue here, but "number" doesn't seem to be in spacy's vocab? #2561

How to reproduce the behaviour

def is_word_in_nlp_vocab(word: str, nlp) -> bool:
    return word in list(nlp.vocab.strings)
nlp2 = spacy.load('en_core_web_lg')
is_word_in_nlp_vocab("number", nlp2)
False

Your Environment

• spaCy version: 3.2.1
• Platform: Linux-5.11.0-46-generic-x86_64-with-glibc2.31
• Python version: 3.9.7
• Pipelines: en_core_web_lg (3.2.0), en_core_web_sm (3.2.0), en_core_web_md (3.2.0)

[polm]

The particular way you're checking this returning false is a quirk of the implementation of the string store. "number" is one of the strings stored as a symbol, which means it is stored in a special way and won't come up if you iterate over the keys in the string store. On the other hand this is True:

import spacy

nlp = spacy.load('en_core_web_lg')

print("number" in nlp.vocab.strings) # True

I didn't address this last time because the issue just seemed resolved either way, but can you clarify why this is a problem for you? The string store is an implementation detail and the presence or absence of strings in it is basically arbitrary - it shouldn't be used to check if something is OOV or anything.

[minimario]

Thanks, that makes sense :)

So my use case is that I have a set of words, either fairly common words or garbles phrases that aren't words and want to separate the two... I thought this was the right way to do that but supposedly not. Should I be using is_oov instead?

[adrianeboyd]

No, there aren't any word lists in the provided trained pipelines that are suitable for spell checking or similar tasks. is_oov is used to check whether there's a word vector (for v2.3+; note that earlier versions had a slightly different use), but the list of tokens with vectors is too large and noisy to be useful for spell checking. I mean, I suppose it wouldn't hurt for you to try and see if it did perform well enough, but it's definitely not intended for this kind of task.