Tokenizer not splitting by infix in some cases

[borjafdezgauna]

How to reproduce the behaviour

I would like the tokenizer to split by nearly any punctuation symbol, and I am having issues in some weird cases.

I initialize the tokenizer this way:

        prefix_re = compile_prefix_regex(nlp.Defaults.prefixes).search
        suffix_re = compile_suffix_regex(nlp.Defaults.suffixes).search
        infix_re = re.compile(r'[-—-−‐/:;,\.\(\)\?ºª“”]').finditer
          
        tokenizer = Tokenizer(nlp.vocab, prefix_search=prefix_re, suffix_search=suffix_re, infix_finditer=infix_re, rules=None)

But, although the dot is set as an infix, I get this:

tokenizer.explain('Alonso A .2014. Membrane')
[('TOKEN', 'Alonso'), ('TOKEN', 'A'), ('TOKEN', '.2014'), ('SUFFIX', '.'), ('TOKEN', 'Membrane')]

I can't understand why '.2014' is output as a token and is not split in '.' and '2014'

Is there something weird going on there? Or am I missing something? Any help is appreciated

Your Environment

• Operating System: Windows 10
• Python Version Used: Python 3.11.4
• spaCy Version Used: Spacy 3.6.1

[adrianeboyd]

The infix matching skips matches that start at index 0 in the token string. Could you match this as a prefix instead (probably still in addition to the infix matching)?

[adrianeboyd]

Let me convert this to a discussion...

[borjafdezgauna]

You are right, after adding the dot as a prefix too, it worked as expected:

> prefix_re = compile_prefix_regex(nlp.Defaults.prefixes + [r'\.']).search
[('TOKEN', 'Alonso'), ('TOKEN', 'A'), ('PREFIX', '.'), ('TOKEN', '2014'), ('SUFFIX', '.'), ('TOKEN', 'Membrane')]

Thank you very much!