How to use EntityRuler and label my entity correctly

[dasseya1]

How it should work

• "my doaq is ABCDAE" - because ABCDAE is six characters and we have doaq in the sentence, it should label ABCDAE a CODE

• "my account code which is A7C6A9" - because A7C6A9 is six characters and we have account code in the sentence, it should label A7C6A9 a CODE

• "I like my dog with 7861A3" - because 7861A3 is a six characters and we don't have "account code" and "doaq" in the sentence, it shouldn't label 7861A3 as a CODE

"""

My code

import spacy
from spacy.pipeline import EntityRuler

from spacy.lang.en import English

nlp = English()

# Initialize EntityRuler
ruler = nlp.add_pipe("entity_ruler")

# Define patterns for the EntityRuler
patterns = [
    {"label": "CODE", "pattern": [{"LOWER": "doaq"}, {"SHAPE": "xxxxxx"}]},
    {"label": "CODE", "pattern": [{"LOWER": "account"}, {"LOWER": "code"}, {"SHAPE": "xxxxxx"}]},
]

# Add patterns to the EntityRuler
ruler.add_patterns(patterns)


# Example text
texts = [
    "my doaq is ABCDAE",
    "my account code which is A7C6A9",
    "I like my dog with 7861A3"
]

# Process each text
for text in texts:
    doc = nlp(text)

    # Extract entities
    entities = [(ent.text, ent.label_) for ent in doc.ents]
    print(f"Text: {text}, Extracted Entities: {entities}")

Current result

Text: my doaq is ABCDAE, Extracted Entities: []
Text: my account code which is A7C6A9, Extracted Entities: []
Text: I like my dog with 7861A3, Extracted Entities: []

Expected Result

Text: my doaq is ABCDAE, Extracted Entities: [('ABCDAE', 'CODE')]
Text: my account code which is A7C6A9, Extracted Entities: [('A7C6A9', 'CODE')]
Text: I like my dog with 7861A3, Extracted Entities: []

How can I fix my code?

[EDMPhoenix]

[dasseya1]

@EDMPhoenix Thanks for answering. But your provided code didn't run for me.

I got an error for matcher.add("CODE", None, pattern1, pattern2). Even tough I fixed it, the script doesn't give me the expected output.

This a google colab I am using. - https://colab.research.google.com/drive/1C6aTP25rSPCkjkCDDR8q57jcYNbQTDi0#scrollTo=K9cZdDKrkQTB

[EDMPhoenix]

Sorry for my misunderstanding. Then please try this once.

import spacy
from spacy.pipeline import EntityRuler

nlp = spacy.load("en_core_web_sm")

nlp.remove_pipe('ner')

ruler = nlp.add_pipe("entity_ruler")

patterns = [
    {"label": "CODE", 
     "pattern": [{"LOWER": "doaq"}, {"LOWER": {"REGEX": "is"}}, {"TEXT": {"REGEX": "^[A-Za-z0-9]{6}$"}}]},
    {"label": "CODE",
     "pattern": [{"LOWER": "account"}, {"LOWER": "code"}, {"LOWER": {"REGEX": "which"}}, {"LOWER": {"REGEX": "is"}}, {"TEXT": {"REGEX": "^[A-Za-z0-9]{6}$"}}]},
]

ruler.add_patterns(patterns)

texts = [
    "my doaq is ABCDAE",
    "my account code which is A7C6A9",
    "I like my dog with 7861A3",
]

for text in texts:
    doc = nlp(text)
    entities = []
    for ent in doc.ents:
        # Extract the code from the entity
        code = ent.text.split()[-1]
        entities.append((code, ent.label_))
    print(f"Text: {text}, Extracted Entities: {entities}")

This script will output

Text: my doaq is ABCDAE, Extracted Entities: [('ABCDAE', 'CODE')]
Text: my account code which is A7C6A9, Extracted Entities: [('A7C6A9', 'CODE')]
Text: I like my dog with 7861A3, Extracted Entities: []

Thank you for understanding.

[adrianeboyd]

Writing single matcher patterns to label one token based on the presence of another token AND within sentence boundaries is going to be tricky with the matcher or entity ruler by itself.

I'd recommend trying a two-step approach instead:

• use an entity ruler with separate patterns to label the FLAG ("account code", etc.) and CODE (the six-char regex) spans
• write a custom component that iterates over doc.sents and modifies doc.ents so that you only save the CODE entity if there's also a FLAG entity in that sentence