LOWER not working for span matcher

[iamyihwa]

Applying 'LOWER' to span matcher does not seem to work well.

brands = ['nike corp', 'adidas corp']
(1) patterns = [{"label": "BRAND", "pattern": brand} for brand in brands]

(2) patterns = [{"label": "BRAND", "pattern": [{"LOWER": brand }]} for brand in brands]

In scenarios (1) and (2) does not detect reliably the brand names.

How to reproduce the behaviour

brands = ['nike corp', 'adidas corp']
nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("span_ruler")
#patterns = [{"label": "BRAND", "pattern": brand} for brand in brands]
patterns = [{"label": "BRAND", "pattern": [{"LOWER": brand }]}   for brand in brands]
with nlp.select_pipes(enable="tagger"):
    ruler.add_patterns(patterns)
print(patterns)

text = "nike corp is a brand."
doc1 = nlp(text)
print([(span.text, span.label_, span.start) for span in doc1.spans["ruler"]])

text = "NIKE CORP is a brand."
doc1 = nlp(text)
print([(span.text, span.label_, span.start) for span in doc1.spans["ruler"]])

Your Environment

• Operating System: Windows
• Python Version Used: 3.7.2
• spaCy Version Used: 3.4.1
• Environment Information:

[rmitsch]

Hi @iamyihwa, the components of the patterns passed on span_ruler have to correspond to spaCy's tokenization. I.e. each part of your pattern has to align with a token as recognized by spaCy.

If you want to want to match for spans consisting of multiple tokens (such as "nike corp" or "adidas corp"), the pattern has to reflect this (see here for another example with a pattern for "san francisco"). So instead of

patterns = [{"label": "BRAND", "pattern": [{"lower": "nike corp"}]}]

you'll want to do:

patterns = [{"label": "BRAND", "pattern": [{"lower": "nike", "lower": "corp"}]}]

In your example you can replace

patterns = [{"label": "BRAND", "pattern": [{"LOWER": brand }]}   for brand in brands]

with

patterns = [{"label": "BRAND", "pattern": [{"lower": token} for token in brand.split()]} for brand in brands]

to obtain correct results.