EntityRuler label is always "ENTITY" -- how to set a custom one?

[DSLituiev]

I would like to add a rule-based named entity recognizer to my pipeline.

I see there is Matcher and EntityRuler options. I couldn't get quite figure out what the difference is, but for now trying the latter one to obtain an ORGAN named entity and running into an issue that the "label" attribute is getting ingored, and all matches are classified as "ENTITY":

import re
import spacy
from spacy.pipeline import EntityRuler

nlp = spacy.load("en_core_sci_sm")
ruler = nlp.add_pipe("entity_ruler", "organs")
organs = organs = ['bladder',  'kidney',  'ureter',  
                   'brain', 'stomach', 'pancreas', 'heart',
                     # ... 
                          ]
patterns = [{"label":"ORGAN", "pattern":[{"LEMMA":organ}]} for organ in organs]
ruler.add_patterns(patterns)

text = "This affects the heart, stomach, and pancreas only."
doc = nlp(text)
displacy.render(doc, style="ent", jupyter=True)

This affects the heart ENTITY , stomach ENTITY , and pancreas ENTITY only.

is there a way to change "ENTITY" to a custom NER label?

[DSLituiev]

Looks like I was able to do what I want after switching from en_core_sci_sm to en_core_web_md.

How does that happen? Is this feature (or idiosyncrasy) documented somewhere?

[polm]

I'm not sure why this is happening but it seems to be something to do with the SciSpaCy pipelines. Since we don't maintain those we also can't change their behavior or update their docs.

[DSLituiev]

Another context issue I've encountered with this example:

This example catches "brain" as an entity:

text = "Brain, left frontal lobe, resection"
print(nlp(text).ents)
[tok.text for tok in nlp(text)]

# (Brain,)
# ['Brain', ',', 'left', 'frontal', 'lobe', ',', 'resection']

This example does not catch "brain" as an entity:

text = "\nBrain, left frontal lobe, resection"
# text =  ": Brain, left frontal lobe, resection" # similarly
print(nlp(text).ents)
[tok.text for tok in nlp(text)]

# ()
# ['\n', 'Brain', ',', 'left', 'frontal', 'lobe', ',', 'resection']    

The two differ by one token, which is correctly parsed. What have gone wrong and how to debug this in general?

[DSLituiev]

This got solved when I replaced "LEMMA" with "LOWER" in

patterns = [{"label":"ORGAN", "pattern":[{"LOWER":organ}]} for organ in organs]
ruler.add_patterns(patterns)

I see that lemma is supposed to be case-sensitive, but hear that is not the case. That is, it does find "Brain" with the given lemma "brain", but trips over the context of preceding whitespaces.

[polm]

The issue with your example is that "Brain" is being detected as a proper noun and lemmas depend on part of speech. "Brain" is an English surname, so it's more susceptible to that issue than other words to start with, and your sentence looks like an address ("John, please do..."). So it looks like you just have an unlucky sentence there.