How to get DependencyMatcher results as Span?

[fatihbozdag]

Greetings all,

I am trying work with multiple patterns on DependencyMatcher applied on a single doc. I would like to know if it is possible to get matcher returns as Span?

For instance;

import spacy
from spacy.matcher import DependencyMatcher
nlp = spacy.load('en_core_web_lg')
matcher = DependencyMatcher(nlp.vocab)
pattern_perfect_no_adv = [
    {
        "RIGHT_ID": "modals",
        "RIGHT_ATTRS": {"DEP": "aux", "TAG":"MD"}
    },    
    {"LEFT_ID": "modals",
     "REL_OP": ".",
     "RIGHT_ID": "Modal_Perfect",
     "RIGHT_ATTRS": {"DEP":"aux", "ORTH": "have"}
     },
      {
        "LEFT_ID": "modals",
        "REL_OP": ".",
        "RIGHT_ID": "modals_adv",
        "RIGHT_ATTRS": {"DEP": "advmod", "OP": "!"}
    },
    {
        "LEFT_ID": "modals",
        "REL_OP": "<",
        "RIGHT_ID": "modals_verb",
        "RIGHT_ATTRS": {"DEP": "ROOT"}
    },
    
    {"LEFT_ID": "modals_verb", 
     "REL_OP": ">", 
     "RIGHT_ID":"modals_subject",
     "RIGHT_ATTRS": {"DEP": "nsubj"}
     }
    ]
pattern_perfect_with_adv = [
    {
        "RIGHT_ID": "modals",
        "RIGHT_ATTRS": {"DEP": "aux", "TAG":"MD"}
    },    
    {"LEFT_ID": "modals",
     "REL_OP": ".",
     "RIGHT_ID": "Modal_Perfect",
     "RIGHT_ATTRS": {"DEP":"aux", "ORTH": "have"}
     },
      {
        "LEFT_ID": "modals",
        "REL_OP": ".",
        "RIGHT_ID": "modals_adv",
        "RIGHT_ATTRS": {"DEP": "advmod"}
    },
    {
        "LEFT_ID": "modals",
        "REL_OP": "<",
        "RIGHT_ID": "modals_verb",
        "RIGHT_ATTRS": {"DEP": "ROOT"}
    },
    
    {"LEFT_ID": "modals_verb", 
     "REL_OP": ">", 
     "RIGHT_ID":"modals_subject",
     "RIGHT_ATTRS": {"DEP": "nsubj"}
     }]
pattern_simple_no_adv = [
    {
        "RIGHT_ID": "modals",
        "RIGHT_ATTRS": {"DEP": "aux", "TAG":"MD"}
    },    
    {"LEFT_ID": "modals",
     "REL_OP": ".",
     "RIGHT_ID": "Modal_Perfect",
     "RIGHT_ATTRS": {"DEP":"aux", "ORTH": "have", "OP": "!"}
     },
      {
        "LEFT_ID": "modals",
        "REL_OP": ".",
        "RIGHT_ID": "modals_adv",
        "RIGHT_ATTRS": {"DEP": "advmod", "OP": "!"}
    },
    {
        "LEFT_ID": "modals",
        "REL_OP": "<",
        "RIGHT_ID": "modals_verb",
        "RIGHT_ATTRS": {"DEP": "ROOT"}
    },
    
    {"LEFT_ID": "modals_verb", 
     "REL_OP": ">", 
     "RIGHT_ID":"modals_subject",
     "RIGHT_ATTRS": {"DEP": "nsubj"}
     }
    ]
pattern_simple_with_adv = [
    {
        "RIGHT_ID": "modals",
        "RIGHT_ATTRS": {"DEP": "aux", "TAG":"MD"}
    },    
    {"LEFT_ID": "modals",
     "REL_OP": ".",
     "RIGHT_ID": "Modal_Perfect",
     "RIGHT_ATTRS": {"DEP":"aux", "ORTH": "have", "OP": "!"}
     },
      {
        "LEFT_ID": "modals",
        "REL_OP": ".",
        "RIGHT_ID": "modals_adv",
        "RIGHT_ATTRS": {"DEP": "advmod"}
    },
    {
        "LEFT_ID": "modals",
        "REL_OP": "<",
        "RIGHT_ID": "modals_verb",
        "RIGHT_ATTRS": {"DEP": "ROOT"}
    },
    
    {"LEFT_ID": "modals_verb", 
     "REL_OP": ">", 
     "RIGHT_ID":"modals_subject",
     "RIGHT_ATTRS": {"DEP": "nsubj"}
     }
    ]

matcher.add("pattern1", [pattern_perfect_no_adv])
matcher.add("pattern2", [pattern_perfect_with_adv])
matcher.add("pattern3", [pattern_simple_no_adv])
matcher.add("pattern4", [pattern_simple_with_adv])
doc = nlp("I should have studied more. I may read some more books. I must really do it. I shouldn't have really failed that bad")`
matches = matcher(doc)
print(matches)
[(13439661873955722336, [1, 2, 2, 3, 0]),
 (17196482192769206822, [7, 8, 8, 8, 6]),
 (17196482192769206822, [20, 21, 21, 24, 19]),
 (685235678504635529, [14, 15, 15, 16, 13])]

So the question is how to get to span from here? More specifically how to get each sentence matched with each pattern?
Like: sent('I should have studied more.') = pattern1 ?

[polm]

The DependencyMatcher doesn't return spans because there's no guarantee that it will match a contiguous string of tokens. For example, your third match there has tokens 19, 20, 21, and 24 - there's a gap.

A simple thing you can do is take the highest and lowest token index in the returned match and use that to create a span.

To get the sentence a match is in, you can use the token.sent attribute

[fatihbozdag]

Can you provide a sample code to use token.sent attribute? With the following, I got duplicated sentences since each token is iterated seperately.

for match_id, token_ids in matches:
    for i in range(len(token_ids)):
        print(doc[token_ids[i]].sent)
I should have studied more.
I should have studied more.
I should have studied more.
I should have studied more.
I should have studied more.
I may read some more books.
I may read some more books.
I may read some more books.
I may read some more books.
I may read some more books.
I shouldn't have really failed that bad
I shouldn't have really failed that bad
I shouldn't have really failed that bad
I shouldn't have really failed that bad
I shouldn't have really failed that bad
I must really do it.
I must really do it.
I must really do it.
I must really do it.
I must really do it.

[polm]

All the tokens in a match will be in the same sentence, so just take it for one token, not all of them.

[fatihbozdag]

But how exactly?

Well, I am applying this to 1.5 million of words corpus. I guess DependencyMatcher will not do it.

[polm]

You can trivially take the first token in each match. Or is that not what you wanted? I'm not sure why this wasn't clear to you.

for match_id, token_ids in matches:
    print(doc[token_ids[0]].sent)

[fatihbozdag]

Thank you, Polm :) Honestly I was having a kind of mental freeze. All is clear now (Y).

[fatihbozdag]

I gave up the idea of using DependencyMatcher to TokenMatcher. TokenMatcher is much easier :)