Best method for matching many words and their lemmas

[nlovell1]

Hi-

Suppose I want to detect, for a list of multi word phrases, each time they appear in a body of text, allowing for changes in inflection to be still matched (so, effectively matching by lemma in the correct order).

I have read through the docs and see many options, including but not limited to, using phrase matcher with entity ruler, token matcher with entity ruler, and both of these directly without the entity ruler.

What are the consequences of using each, and, do they produce the same result for my purpose?

[polm]

The PhraseMatcher allows you to match phrases, but only on one attribute.

The Matcher allows for very versatile token-based matching.

The EntityRuler is primarily a wrapper for either/both of the above so you can use it as a pipeline component. It also has the small extra feature that it allows setting entity IDs.

If you just want to match phrases based on lemmas I'd recommend using the PhraseMatcher via the EntityRuler.

[nlovell1]

Suppose I use the phrase matcher on the token attribute and the matcher on the same set of inputs. Will the result on the matcher be different? You said 'token-based' so not sure if there is more heuristics going on.

[polm]

If you are only matching on lemmas it will normally not make a difference. There are no special heuristics.

There can be a difference in some limited situations, like if tokenization is context-dependent. The PhraseMatcher will automatically tokenize the input you give it, and only match things with that tokenization. So this can be an issue if your tokenization is contextual, or if you have case-dependent tokenization. But usually those are not issues.

It may be helpful to think of the PhraseMatcher as being a wrapper that builds a Matcher pattern internally. With the Matcher you can do more complicated things - like match one token on POS and one on surface text - but matching phrases on one attribute is a common enough use case it gets its own class.