Apply rule-based matching in isolation, one token at a time

[e-e]

Is it possible to apply the rule-based matching rules in isolation, given a single token and a rule following the same format as when used with a full Matcher class? Specifically, I would like to use this in an on_match function to search the tokens in the doc following a match, leveraging the same rules used to find the initial match. For example:

def on_match(matcher, doc, i, matches):
  match_id, start, end = matches[i]
  entity = Span(doc, start, end, label=match_id)

  rule = { "TEXT": "sentence" }
  for token in doc[end:]:
    # HERE: apply the above rule to each token
    if is_match(token, rule):
      # do something
  

doc = nlp("An example sentence.")
matcher = Matcher(nlp.vocab)
pattern = [
  { "TEXT": "An" },
]
matcher.add("LABEL", [pattern], on_match=on_match)

[bdura]

Hello @e-e, would something like this answer your problem?

import spacy
from spacy.matcher import Matcher

nlp = spacy.load("blank:en")

doc1 = nlp("An example sentence.")
doc2 = nlp("Another example.")

matcher = Matcher(nlp.vocab)
pattern = [
    {"TEXT": "An"},
    {"OP": "*"},
    {"TEXT": "sentence"},
]
matcher.add("LABEL", [pattern])

matcher(doc1)  # Matches
matcher(doc2)  # Does not match

You could use the callback to bootstrap a second Matcher instance and mutate the matches list, but it will likely be harder to maintain.

[e-e]

Thanks for the reply. I would like to avoid setting up a new matcher instance, and instead was looking for a way to apply the matching rules on a token-by-token basis for an already parsed doc. Basically some way of checking "yes, this token matches this rule", or "no, this token does not match this rule"

The example I provided was a much simpler case than what I'm actually looking to do. Regardless of whether I'm able to do what I'm asking about or not, I will need to mutate the match

[bdura]

The callback can mutate the match list, so you can really do anything you want.

def callback(matcher: Matcher, doc: Doc, i: int, matches: list[tuple[int, int, int]]):
    _, _, end = matches[i]
    
    # modify tokens in place
    for token in doc[end:]:
        # check for some condition
        if token.text == "sentence":
            # mutate tokens
            token.norm_ = "st"

            # remove current match
            matches.pop(i)

            # replace with dummy match
            matches.index(i, (0, 0, 0))
            
    
    # match on the rest of the document
    # Beware, this will run the callback on subsequent matches
    # which is most likely not what you want.
    # The last match will have this run N-1 time...
    for match in matcher(doc[end:]):
        print(match)

[e-e]

I appreciate the response, but I think we're slightly missing on the actual ask of the question. Maybe this is not possible, but I am specifically looking for a way to apply a rule, as they are defined for rule-base matching patterns, to one token at a time, instead of a series of if-statements that manually checks for those conditions.

With your example from above:

import spacy
from spacy.matcher import Matcher

nlp = spacy.load("blank:en")

doc1 = nlp("An example sentence.")
doc2 = nlp("Another example.")

matcher = Matcher(nlp.vocab)
pattern = [
    {"TEXT": "An"},
    {"OP": "*"},
    {"TEXT": "sentence"},
]
def callback(matcher: Matcher, doc: Doc, i: int, matches: list[tuple[int, int, int]]):
    _, _, end = matches[i]

    pattern = { "TEXT":   "sentence" }

    for token in doc[end:]:
        # !LOOKING FOR SOMETHING LIKE THIS!
        if matcher.is_match(token, pattern):
          # do something
            
    
    # match on the rest of the document
    # Beware, this will run the callback on subsequent matches
    # which is most likely not what you want.
    # The last match will have this run N-1 time...
    for match in matcher(doc[end:]):
        print(match)
matcher.add("LABEL", [pattern])

matcher(doc1)  # Matches
matcher(doc2)  # Does not match

specifically, something like the line with matcher.is_match(token, pattern)

And of course this is a very over-simplified example

[bdura]

Hi @e-e, you can apply a matcher that works with one-token-patterns, and iterate over the results.

# second_matcher only has one-token-patterns
for _, i, _ in second_matcher(doc[end:]):
    token = doc[i]
    # do something

Be careful though, the callback is applied every time so using the same matcher is probably not what you want.

[bdura]

You cannot apply the matcher to an individual token and check that it matches directly. If you insist on doing something like that, you may apply the matcher to spans of individual tokens, but that's very inefficient.

for token in doc[end:]:
    # check that there is a match
    if list(matcher(doc[token.i:token.i+1])):
        # do something