Tokenizer exception with regex does not work as expected

[drm-addoptio]

I want spacy to recognize tokens like:

CVE-2021-26855
CVE-2021-26857
CVE-2021-26858
CVE-2021-27065

The following regex is the pattern:
"CVE-\d\d\d\d-\d{2,5}"

I added a token_match to the tokenizer:

matches = re.compile("CVE-\d\d\d\d-\d{2,5}").split
nlp.tokenizer.token_match = matches

This works fine for situations where there are whitespaces between the words.
But in the following situation it doesn't work:
String: "incident:CVE-2021-26855CVE-2021-26857CVE-2021-26858CVE-2021-27065We"

I used the tokenizer.explain function to look what happens:
incident:CVE-2021-26855CVE-2021-26857CVE-2021-26858CVE-2021-27065We TOKEN_MATCH

Why does spacy not recognize the patterns in a string without whitespaces? I tried the regex on itself with the given string and it works. Can you please come up with a working solution to the problem. Thank you in advance!

[polm]

token_match is just designed to prevent things from being split if they would otherwise be split by infixes, it doesn't split out strings from larger tokens. You might be able to add your regex to the infixes list to get the behaviour you want, but looking at strings like the one you have where there are a bunch of concatenated tokens, I think running a regex over your raw text to surround any CVE strings with spaces as a preprocessing step will probably be the best solution.

[drm-addoptio]

@polm The infix solution works great. I will work with it. Thank you very much for this hint. This is how I did it:

infixes = nlp.Defaults.infixes + [r'''CVE-\d\d\d\d-\d{2,7}''',]
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

Leads to the following outcome:

incident         TOKEN
:        INFIX
CVE-2021-26855   INFIX
CVE-2021-26857   INFIX
CVE-2021-26858   INFIX
CVE-2021-27065   INFIX
We       TOKEN