Spacy Matcher doesn't match with "cannot"

[bwang482]

I am copying and testing the simple example from the documentation, and replacing "hello" with "cannot". Now the matcher returns nothing (it worked with "hello").

import spacy
from spacy.matcher import Matcher

nlp = spacy.blank("en")
matcher = Matcher(nlp.vocab)
# Add match ID "HelloWorld" with no callback and one pattern
pattern = [{"LOWER": "cannot"}, {"LOWER": "world"}]
matcher.add("HelloWorld", [pattern])

doc = nlp("Hello, world! Cannot world!")
matches = matcher(doc)
for match_id, start, end in matches:
    string_id = nlp.vocab.strings[match_id]  # Get string representation
    span = doc[start:end]  # The matched span
    print(match_id, string_id, start, end, span.text)

PhraseMatcher doesn't seem to have a problem with "cannot world" but what is so special about the token "cannot" that cannot be (by Matcher) matched? Or am I missing something here?

[Pandalei97]

Hi,

If you take a look at the tokens in the doc, you will see that "cannot" is separated into 2 tokens: 'can' and 'not'.

Since Matcher patterns are descriptions of tokens to find, your pattern will search for a single token 'cannot', followed by 'world'.

This pattern works in your case : pattern = [{"LOWER": "can"}, {"LOWER": "not"}, {"LOWER": "world"}]

The PhraseMatcher works because il will tokenize the text internally and produce patterns which respect the tokenizer behaviour.

When writing patterns for Matcher, you need to pay attention to the tokenization, especially when it comes to compound words.

[bwang482]

Thanks very much @Pandalei97 ! Great point!