Debugging False Positives on Matcher (Phrase and Dependency)

[ctufts]

I have a phrase matcher and a dependency matcher set up in my model. When I run my tests I'm seeing a false positive (a label assigned to a span that should not have been recognized). However, I have multiple patterns defined for each match ID.
Something like this (example case, so I replaced some of the attribute section - please disregard if a bracket is misplaced here, but I cannot share the original pattern. I have validated the pattern and each pattern is formatted correctly prior to this.):

rule_a: [
        [
            {
                "RIGHT_ID": "anchor",
                "RIGHT_ATTRS": {
                    "LEMMA": "some"
                    }
                },
            },
            {
                "LEFT_ID": "anchor",
                "REL_OP": "<",
                "RIGHT_ID": "head",
                "RIGHT_ATTRS": {
                    "LEMMA": "word"
                    }
                },
            },
        ],
        [
            {
                "RIGHT_ID": "anchor",
                "RIGHT_ATTRS": {
                    "LEMMA": "other"
                    }
                },
            },
            {
                "LEFT_ID": "anchor",
                "REL_OP": "<",
                "RIGHT_ID": "head",
                "RIGHT_ATTRS": {
                    "LEMMA": "word"
                    }
                },
            },
        ]

    ]

Is there a simple way to debug which pattern of a given match_id is being matched by the dependency matcher or phrase matcher? As of now I can only see the match_id from the match, but cannot see any info about which pattern caused it. Any help is appreciated.

[richardpaulhudson]

• With the dependency matcher, the individual patterns defined under a given match_id are processed using an AND relationship: the presence of a match with a given match_id implies that all of the patterns have been matched.
• With the phrase matcher, the terms defined under a given match_id are processed using an OR relationship, but it should be easy to work out which term matched by examining the results.

[ctufts]

Hi Richard, Thank you for the response.
I misspoke when I said PhraseMatcher - my matchers are token matchers and dependency matchers. So, when I have to debug patterns that are slightly different - searching keywords is a time consuming task for trouble shooting especially since many of our patterns are built based on dictionaries defined in a variable (often in other files and imported).

I may be misunderstanding your first bullet point, but the output of the dependency matcher does not seem to be based on an AND. For instance if I use the following rule with multiple patterns (one which matches my text, one that does not) I will still get a match. So I don't believe this is an AND operation. In which case it seems as though it would be possible to get info on which pattern is matched for that specific match_id.

import spacy
from spacy.matcher import DependencyMatcher

nlp = spacy.load("en_core_web_sm")
text_sample = "Many thanks."
doc = nlp(text_sample)
for tok in doc:
    print(f"text: {tok.text}, POS: {tok.pos_}, DEP: {tok.dep_}, LEMMA: {tok.lemma_}")

text: Many, POS: ADJ, DEP: amod, LEMMA: many
text: thanks, POS: NOUN, DEP: ROOT, LEMMA: thank
text: ., POS: PUNCT, DEP: punct, LEMMA: .

RULES = {
    "setA" : [
        [
            {
                "RIGHT_ID": "anchor",
                "RIGHT_ATTRS": {"LOWER":"thanks"},
            },
            {
                "LEFT_ID": "anchor",
                "REL_OP": ">>",
                "RIGHT_ID": "child",
                "RIGHT_ATTRS": {
                    "LEMMA": "many"
                },
            },
        ],
        [
            {
                "RIGHT_ID": "anchor",
                "RIGHT_ATTRS": {"LOWER":"thanks"},
            },
            {
                "LEFT_ID": "anchor",
                "REL_OP": ">>",
                "RIGHT_ID": "child",
                "RIGHT_ATTRS": {
                    "LEMMA": "no"
                },
            },
        ]
    ]
    }

matcher = DependencyMatcher(nlp.vocab)
for bias_type, rule_list in RULES.items():
    matcher.add(bias_type, rule_list)

matches = matcher(doc)
for match in matches:
    print(match)
    print(matcher.vocab.strings[match[0]])
    print(doc[min(match[1]): max(match[1])+1])

(1021335527674910047, [1, 0])
setA
Many thanks

The above example will also generate the following match for the text no thanks:

(1021335527674910047, [1, 0])
setA
No thanks

[richardpaulhudson]

Is there any reason why you can't define the two sets of patterns under separate match_ids?

RULES = {
    "setA" : [
        [
            {
                "RIGHT_ID": "anchor",
                "RIGHT_ATTRS": {"LOWER":"thanks"},
            },
            {
                "LEFT_ID": "anchor",
                "REL_OP": ">>",
                "RIGHT_ID": "child",
                "RIGHT_ATTRS": {
                    "LEMMA": "many"
                },
            },
        ]
    ],
    "setB" : [
        [
            {
                "RIGHT_ID": "anchor",
                "RIGHT_ATTRS": {"LOWER":"thanks"},
            },
            {
                "LEFT_ID": "anchor",
                "REL_OP": ">>",
                "RIGHT_ID": "child",
                "RIGHT_ATTRS": {
                    "LEMMA": "no"
                },
            },
        ]
    ]
}

Essentially, with both the token matcher and the dependency matcher, you are allowed to define multiple sets of patterns under the same match_id to cover the case where you explictly want them to be reported as a single entity. That doesn't seem to be what you want here, so I would suggest splitting them up.

[ctufts]

I'm sure we could rewrite them as such, but that requires lots of time for a few hundred rules along with code changes to map them back to the larger family of rules. My specific question was more to ensure I'm not missing some component in Spacy that would allow me to parse which specific pattern matched in a match_id with multiple rules. From the sound of it, my assumption was correct that it is not possible to do this at this time.

Thanks for your responses. I appreciate your time.