How to mix Regex Expressions and Regular Strings to SpanRuler?

[grahamanderson]

For SpanRuler, How do I add terms AND regular expressions to my terms list?

For the below code, everything works but the regex term:
{'ent': 'ACRO_UNKNOWN', "patterns": [{"TEXT": {"REGEX":"[A-Z][A-Z-0-9]+"}}]} #T1, THING2

when I add the regex pattern to the terms list, I get:

ValueError: [E097] Invalid pattern: expected token pattern (list of dicts) or phrase pattern (string) but got:
{'TEXT': {'REGEX': '[A-Z][A-Z-0-9]+'}}

Myself, I'm mixing extracted terms (from the corpus) and (many) patterns I'm also looking for....
Is SpanRuler the right choice to mix both, or do I need a different Spacy Object (like EntityRuler) to handle regex?

Many thanks in advance

import spacy
from spacy.pipeline import SpanRuler
import pandas as pd

def extract_named_entities(text: str, terms: list, nlp=None, merge=True) -> pd.DataFrame:
    """
    Extract custom entities from a given text and return them in a pandas DataFrame.

    Args:
        text (str): The input text to process.
        terms (list of dict): A list of dictionaries where each dictionary specifies an entity type and its patterns. 
                              For example: {'ent': 'TYPE', 'patterns': ['pattern1', 'pattern2']}.
        nlp (Optional[spacy.lang]): A preloaded spaCy language model. If not provided, the function will load "en_core_web_sm".
        merge (bool, default=True): Determines whether to merge contiguous entities into single tokens.

    Returns:
        pd.DataFrame: A dataframe containing columns 'text', 'tag_', 'pos_', 'dep_', and 'ent_type_' 
                      for each token in the input text.

    Dependencies:
        - spacy
        - pandas

    Examples:
        >>> terms = [{'ent': 'ACRO', 'patterns': ['AA']}]
        >>> extract_named_entities("This is AA.", terms)
           text    tag_    pos_   dep_  ent_type_
        0  This     DET   NOUN  nsubj
        1    is    VERB   VERB   ROOT
        2    AA  PROPN  PROPN   attr      ACRO
        3     .  PUNCT  PUNCT  punct

    Notes:
        Ensure the provided spaCy model (if any) contains the necessary components (e.g., tokenizer, tagger) for processing.
    """


    if nlp is None:
        nlp = spacy.load("en_core_web_sm")

    # Ensure only one span ruler exists
    if 'span_ruler' not in nlp.pipe_names:
        ruler = nlp.add_pipe('span_ruler', before="ner")
    else:
        ruler = nlp.get_pipe('span_ruler')
        ruler.clear()  # Clear existing patterns

    # Add patterns to the SpanRuler
    for d in terms:
        patterns = [{"label": d['ent'], "pattern": pattern} for pattern in d['patterns']]
        ruler.add_patterns(patterns)

    doc = nlp(text)

    # Modify token's entity types based on SpanRuler results
    for span in doc.spans.get("ruler", []):
        for token in span:
            token.ent_type_ = span.label_

            
    # Merging contiguous entities (SpanRuler Doesn't seem to merge tokens without it)
    if merge:
        with doc.retokenize() as retokenizer:
            for ent in doc.ents:
                # new rules for not merging
                retokenizer.merge(ent)
                
            # Find the noun phrases by chunking
        with doc.retokenize() as retokenizer:
            for np in doc.noun_chunks:
                # Only merge the noun phrases that are more than one word long
                if len(np) > 1:
                    # new rules for not merging
                    retokenizer.merge(np)
    
    tokens = [{
        'text': token.text, 
        'tag_':token.tag_, 
        'pos_':token.pos_, 
        'dep_': token.dep_, 
        'ent_type_': token.ent_type_
    } for token in doc]
    return pd.DataFrame(tokens).T

# Example usage:
terms = [
    {'ent': 'ACRO', 'patterns': ['AA']},
    {'ent': 'DEF', 'patterns': ['Airplane Agoraphobia']},
    {'ent': 'ACRO_UNKNOWN', "patterns": [{"TEXT": {"REGEX":"[A-Z][A-Z-0-9]+"}}]}  #T1, THING2
    #[{"label": "FRT", "pattern": [{'TEXT' : {'REGEX': "[Aa]ppl[e|es]"}}]},
]

text = "When T1 is False, the acronym AA shall change T1 definition to Airplane Agoraphobia"
extract_named_entities(text, terms, merge=True)

[rmitsch]

Hi @grahamanderson,

enclose the pattern for your regex in [], as a list of patterns is expected (see the docs):

terms = [
    {'ent': 'ACRO', 'patterns': ['AA']},
    {'ent': 'DEF', 'patterns': ['Airplane Agoraphobia']},
    {'ent': 'ACRO_UNKNOWN', "patterns": [[{"TEXT": {"REGEX":"[A-Z][A-Z-0-9]+"}}]]}  #T1, THING2
    #[{"label": "FRT", "pattern": [{'TEXT' : {'REGEX': "[Aa]ppl[e|es]"}}]},
]

[grahamanderson]

Thank you :)