token.is_stop not the same as token.lemma_.lower() in nlp.Defaults.stop_words

[ariansajina]

How to reproduce the behaviour

import spacy

nlp = spacy.load("en_core_web_sm")

example = 'Both go and goes should be removed as stopwords.'

result_1 = [token for token in d if not token.is_stop]

result_2 = [token.lemma_ for token in d if not token.lemma_.lower() in nlp.Defaults.stop_words]

Your Environment

• spaCy version: 3.0.6
• Platform: macOS-10.16-x86_64-i386-64bit
• Python version: 3.8.5
• Pipelines: en_core_web_sm (3.0.0)

[ariansajina]

To give a concrete example, given that "go" is a stopword, I assumed token.is_stop would be True for token.text equals "goes" and so on.

[adrianeboyd]

Hi, the is_stop attribute is designed so that it works without any additional pipeline components like a tagger or a lemmatizer, so by default it currently only checks whether the lowercase form of the token is in the stop word list. You can extend the stop word list by customizing the language defaults before loading the model or by creating a custom language. See: https://spacy.io/usage/linguistic-features#language-subclass

In general, for modern NLP techniques, it's not helpful to remove stop words, so you may not need this step at all. Some relevant discussions: #7228 (comment), #7637 (comment)