on_match remove entities doesn't work properly

[bwang482]

I have an on_match function below that should in theory remove all the entities if they are in sentences that contain one of["ABCD", "walking out of work", "coming out of work"]. But in practice, there is always one missing, I am puzzled, why?

import spacy
from spacy.matcher import PhraseMatcher

def ignore_ents(matcher, doc, i, matches):
    (_, start, end) = matches[i]
    span = doc[start:end]
    
    sent = span.sent
    for _phrase in ["ABCD", "walking out of work", "coming out of work"]:
        if _phrase in sent.text:
            matches.pop(i)
            return

nlp = spacy.blank("en")
nlp.add_pipe('sentencizer')

matcher = PhraseMatcher(nlp.vocab)
terms = ["out of work"]
patterns = [nlp.make_doc(text) for text in terms]
matcher.add("TerminologyList", patterns, on_match=ignore_ents)
doc = nlp("He is out of work! unable to work ABCD. I am coming out of work. I am walking out of work")

matches = matcher(doc)
for match_id, start, end in matches:
    print(start, end)
    span = doc[start:end]
    print(span.text)

This returns:

2 5
out of work
21 24
out of work

[polm]

It looks like the issue is just that none of your phrases match "He is out of work", so I'm not sure what is unexpected here?

I find your code pretty confusing due to the indirection involved. If you're just filtering a list of matches it might be easier to just filter it in the for loop rather than mutating the list. Something like:

matches = matcher(doc)
for match_id, start, end in matches:
    span = doc[start:end]
    for phrase in _phrases:
        if phrase in span.sent.text:
            continue # skip this one
    ... do stuff ...

[bwang482]

Thanks! Can I then please what would be the most computationally efficient way to prevent matching something e.g. out of work if some terms e.g. one of ["ABCD", "walking out of work", "coming out of work"] appear in the same sentence?

[bwang482]

@polm Any advice pls?

[polm]

Don't bump threads, especially not after just one day.

You can just use the sample code I provided above instead of mutating the list.

[bwang482]

Thanks @polm. Would this postprocessing you suggested be more efficient than using on_match?

[polm]

Mutating the list in on_match doesn't work, so I'm not sure the question makes sense.

Are you actually finding this process a speed bottleneck or are you concerned about hypotheticals? The right thing here depends on how often you have matches and how often they are negated by your ignore patterns. But honestly it seems unlikely that this would be a bottleneck in a normal application.