How to Dynamically Assign Entities with no model training

[grahamanderson]

Hello.
Background:
I have some acronyms and definitions (and other things) that are dynamically extracted from a document BEFORE Spacy starts encoding sentences
the dynamically generated list of dictionaries look like (extracted from a section within the word document)
[{acronym: AA, definition: Aircraft Abomination}...]

Question: What's the best way to get Spacy to tag occurences of the acronym and/or definitions with the 'ACRO' or 'DEF' entity tag (token.ent_type_)?

Simplest Best Practice Way:
Is what I'm trying below the best practice way, or is there a simpler way to accomplish this with Spacy. In my case, I do not know the entities in advance, they are document specific--extracted from the document.

Reality Check:
It's possible that Spacy is not the best tool for my use case. If so, I'm open to writing something more customized? Seems that there is much in Spacy I can take advantage of :)

Much appreciation in advance

import spacy
from spacy.matcher import PhraseMatcher
from spacy.tokens import Span

nlp = spacy.load("en_core_web_sm")
ner = nlp.get_pipe("ner")

# Create a doc object by processing the text but without NER
doc = nlp.make_doc("This is a sentence with a the acronym AA in it, its definition is Airplane Agoraphobia")

matcher = PhraseMatcher(nlp.vocab)

# List of custom phrases to match
terms = [
    {'ent': 'ACRO', 'patterns': ['AA']},
    {'ent': 'DEF', 'patterns': ['Airplane Agoraphobia']}
]

# Add the patterns to the matcher
for d in terms:
    patterns = [nlp.make_doc(pattern) for pattern in d['patterns']]
    matcher.add(d['ent'], patterns)

# Find matches in the doc
matches = matcher(doc)

# Add the matched spans to a new list
custom_ents = [Span(doc, start, end, label=label) for label, start, end in matches]

# Run spaCy's NER
ner(doc)

# Combine the entities, prioritizing the custom entities
final_ents = []
seen_tokens = set()

# Add custom entities first
for ent in custom_ents:
    if any(token.i in seen_tokens for token in ent):
        continue
    final_ents.append(ent)
    seen_tokens.update(token.i for token in ent)

# Add spaCy's entities, but don't overwrite custom ones
for ent in doc.ents:
    if any(token.i in seen_tokens for token in ent):
        continue
    final_ents.append(ent)
    seen_tokens.update(token.i for token in ent)

doc.ents = final_ents

# Merging contiguous entities
with doc.retokenize() as retokenizer:
    for ent in doc.ents:
        retokenizer.merge(ent)

# Print tokens and their entity labels
for token in doc:
    print(token.text, token.ent_type_)

[rmitsch]

Hi @grahamanderson! Do I understand correctly that ultimately you want all the entities recognized from spaCy's NER + your rule-based ones? The PhraseMatcher is a suitable tool for this, but your workflow can probably be simplified.

[grahamanderson]

So something like the code below?

My use case (and your expert thoughts are appreciated):
Because of flexibility, I have been using somewhat complex regular expressions to replace matches with a inline JSON string and then explode into attributes within Spacy...the downside is that its cumbersome getting it ito Spacy and I can't seem to change the 'text' attribute

• extract all document acronymns, definitions into a tuple list
• extracted string: This is an AA moment.
• Regex subst to: This is an {acronym:AA, defn: Ambiguous Apples, final_text_token: AA (Ambiguous Apples), pos_: Noun, ent_type_: JSON, type: Component X, subtype: External lookup SubComponent thing} moment.
• Explode that json string into Spacy (with spaces replaced so it's treated one token)

Trying to get Spacy to create complex regex matches was difficult and inflexible--maybe I was doing it wrong?
I'm doing similar substitutions with many other patterns...dealing with components and code.
I guess I could store the generated dictionary elsewhere and retrieve by it's token index

The next step is is divide up the sentence into subjects, predicate, and conditional clauses....which Spacy does not seem to do by itself. Unless, you know of a good method to do this...Currently, I'm creating logic by iterating over token attributes...seems like this should already exist.

The next step is take each important extracted component (name, unit, value, clause type) and export it :)

Final step is using each extracted component (in a datalake of documents) as context for an LLM query / function call.

My use case us highly technical and specific...hardly looks like English....any good patterns/ideas of getting this into Spacy are welcome :) I wrote this from scratch, last time around. I'm trying to determine if Spacy would be more efficient --less code.

import spacy
from spacy.pipeline import SpanRuler

def custom_ner_pipeline(text, terms):
    # Load spaCy's small English model
    nlp = spacy.load("en_core_web_sm")

    # Create a SpanRuler component
    ruler = SpanRuler(nlp)

    # Add patterns to the SpanRuler
    for d in terms:
        patterns = [{"label": d['ent'], "pattern": pattern} for pattern in d['patterns']]
        ruler.add_patterns(patterns)
    
    # Add the SpanRuler to the pipeline, before the NER
    nlp.add_pipe('span_ruler', before="ner")

    # Process the text with spaCy
    doc = nlp(text)
    
    # Merging contiguous entities
    with doc.retokenize() as retokenizer:
        for ent in doc.ents:
            retokenizer.merge(ent)
    
    # Print tokens and their entity labels
    for token in doc:
        print(token.text, token.ent_type_)

# Example usage:
terms = [
    {'ent': 'ACRO', 'patterns': ['AA']},
    {'ent': 'DEF', 'patterns': ['Airplane Agoraphobia']}
]
text = "This is a sentence with a the acronym AA in it, its definition is Airplane Agoraphobia"
custom_ner_pipeline(text, terms)

[rmitsch]

This definitely seems cleaner to me. In terms of best practices I recommend looking into spaCy's config system - this will help reduce the (boilerplate) code you have to write when setting up your workflow with spaCy. E. g. your NER component and the SpanRuler can be configured in a config file; you can load your pipeline from that config file.

    # Merging contiguous entities
    with doc.retokenize() as retokenizer:
        for ent in doc.ents:
            retokenizer.merge(ent)

This shouldn't be necessary. Any particular reason why you're doing this?

[grahamanderson]

well I thought the above code would work. PhraseMatcher works but SpanRuler does not, yet.
It's not flagging any of the ent types :\ And, many iterations/derivations have done no better. Anyone know what needs to be changed? Much appreciation
This
is
a
sentence
with
a
the
acronym
AA
in
it
,
its
definition
is
Airplane Agoraphobia GPE
/path/span_ruler.py:535: UserWarning: [W036] The component 'span_ruler' does not have any patterns defined.
warnings.warn(Warnings.W036.format(name=self.name))

[rmitsch]

1. ruler is not the same SpanRuler instance that you have in your pipeline.
2. SpanRuler doesn't store its results in Token.ent_type_.

A fixed version:

import spacy
from spacy.pipeline import SpanRuler


def custom_ner_pipeline(text, terms):
    # Load spaCy's small English model
    nlp = spacy.load("en_core_web_sm")

    ### CHANGE: get ruler instance from .add_pipe(). ###
    # Add the SpanRuler to the pipeline, before the NER
    ruler = nlp.add_pipe('span_ruler', before="ner")

    # Add patterns to the SpanRuler
    for d in terms:
        patterns = [{"label": d['ent'], "pattern": pattern} for pattern in d['patterns']]
        ruler.add_patterns(patterns)

    # Process the text with spaCy
    doc = nlp(text)

    ### CHANGE: retokenization not necessary ###

    # Print tokens and their entity labels
    for token in doc:
        print(token.text, token.ent_type_)
    print("---")
    ### CHANGE: read SpanRuler results from correct attribute ###
    for span in doc.spans["ruler"]:
        print(span.text, span.label_)

# Example usage:
terms = [
    {'ent': 'ACRO', 'patterns': ['AA']},
    {'ent': 'DEF', 'patterns': ['Airplane Agoraphobia']}
]
text = "This is a sentence with a the acronym AA in it, its definition is Airplane Agoraphobia"
custom_ner_pipeline(text, terms)

Output:

This 
is 
a 
sentence 
with 
a 
the 
acronym 
AA 
in 
it 
, 
its 
definition 
is 
Airplane GPE
Agoraphobia GPE
---
AA ACRO
Airplane Agoraphobia DEF

[grahamanderson]

Thank you! That was very helpful, and certainly aids in my understanding of Spacy! Working code below.

import spacy
from spacy.pipeline import SpanRuler
import pandas as pd

def extract_named_entities(text: str, terms: list, nlp=None, merge=True) -> pd.DataFrame:
    """
    Extract custom entities from a given text and return them in a pandas DataFrame.

    Args:
        text (str): The input text to process.
        terms (list of dict): A list of dictionaries where each dictionary specifies an entity type and its patterns. 
                              For example: {'ent': 'TYPE', 'patterns': ['pattern1', 'pattern2']}.
        nlp (Optional[spacy.lang]): A preloaded spaCy language model. If not provided, the function will load "en_core_web_sm".
        merge (bool, default=True): Determines whether to merge contiguous entities into single tokens.

    Returns:
        pd.DataFrame: A dataframe containing columns 'text', 'tag_', 'pos_', 'dep_', and 'ent_type_' 
                      for each token in the input text.

    Dependencies:
        - spacy
        - pandas

    Examples:
        >>> terms = [{'ent': 'ACRO', 'patterns': ['AA']}]
        >>> extract_named_entities("This is AA.", terms)
           text    tag_    pos_   dep_  ent_type_
        0  This     DET   NOUN  nsubj
        1    is    VERB   VERB   ROOT
        2    AA  PROPN  PROPN   attr      ACRO
        3     .  PUNCT  PUNCT  punct

    Notes:
        Ensure the provided spaCy model (if any) contains the necessary components (e.g., tokenizer, tagger) for processing.
    """

    if nlp is None:
        nlp = spacy.load("en_core_web_sm")

    # Ensure only one span ruler exists
    if 'span_ruler' not in nlp.pipe_names:
        ruler = nlp.add_pipe('span_ruler', before="ner")
    else:
        ruler = nlp.get_pipe('span_ruler')
        ruler.clear()  # Clear existing patterns

    # Add patterns to the SpanRuler
    for d in terms:
        patterns = [{"label": d['ent'], "pattern": pattern} for pattern in d['patterns']]
        ruler.add_patterns(patterns)

    doc = nlp(text)

    # Modify token's entity types based on SpanRuler results
    for span in doc.spans.get("ruler", []):
        for token in span:
            token.ent_type_ = span.label_
            
    # Merging contiguous entities (SpanRuler Doesn't seem to merge tokens without it)
    if merge:
        with doc.retokenize() as retokenizer:
            for ent in doc.ents:
                retokenizer.merge(ent)
    
    tokens = [{
        'text': token.text, 
        'tag_':token.tag_, 
        'pos_':token.pos_, 
        'dep_': token.dep_, 
        'ent_type_': token.ent_type_
    } for token in doc]
    return pd.DataFrame(tokens).T

# Example usage:
terms = [
    {'ent': 'ACRO', 'patterns': ['AA']},
    {'ent': 'DEF', 'patterns': ['Airplane Agoraphobia']}
]
text = "When Y is False, This is a sentence with a the acronym AA in it, its definition is Airplane Agoraphobia"
extract_named_entities(text, terms, merge=True)