Matcher allowing group repetition

[ezorita]

I believe this is not currently possible but would be really helpful and is a common and important feature of other NLP libraries.

The ability to match repeated patterns across multiple tokens. A regex equivalent:

[\d]+(\sand\s[\d]+)*

would match an arbitrarily long sequence of:

4 and 342 and 91 and 13 and 56 ...

Is there a current solution with spacy that would match an arbitrarily long sequence of numbers like the above?

I guess the best would be to allow adding other patterns as attributes:

hello_patterns = [
    [{"LOWER": "hello"}, {"IS_PUNCT": True}, {"LOWER": "world"}],
    [{"LOWER": "hello"}, {"LOWER": "world"}]
]
pattern = [
   [{"PATTERN": hello_patterns, "OP": "*"}, {"LOWER": "friends"}]
]

matches:

hello world hello, world hello world friends

Thanks for your amazing work!

[polm]

Thanks for the suggestion! That sounds like a handy feature, but also one that might be harder than you'd think to add - there's lots of potential for weird interactions with pattern chaining.

One thing you can do that isn't very clean but isn't simple is to duplicate a pattern up to a given number of times. Since it's just a list, you can do something like this:

pattern2 = pattern * 2
pattern3 = pattern * 3

etc, up to some number. While I guess patterns might be unbounded in theory, in practice I would expect you to only need up to a certain number of repetitions.

[ezorita]

Thanks @polm, I understand the difficulty of the implementation.

Thanks for the suggestion but the proposed solution still does not solve the issue.

A possible high-level implementation could be:

1. Traverse the pattern tree from the root
2. Render the innermost PATTERN.
3. Find pattern in text and flag matched tokens with this pattern id, e.g. using token._.pattern_matches.append(pattern_id).
4. Replace the pattern rule in parent by a rule matching pattern_id IN _.pattern_matches
5. Loop