Tokenizing Special cases with whitespace

[Gitclop]

I want to tell the tokenizer to not split Tokens like pi 123 into two seperate tokens, but treat it as one token. I tried a custom infix 'PI \d+', but apperantly infixes are called after whitespace tokenisation. What worked was adding pi 123 into a special_cases file:

special_cases = load_special_cases('special_cases.txt')

        tokenizer_exceptions = nlp.Defaults.tokenizer_exceptions.copy()
        for word in special_cases:
            case = {word: [{"ORTH": word}]}
            tokenizer_exceptions.update(case)

But since there are a lot of combination i need to use regular expressions

[adrianeboyd]

The tokenizer only supports exact string matches as special cases. Instead, you can add a custom component at the beginning of your pipeline that matches based on regexes and uses the retokenizer to adjust the tokenization before the rest of the components are run: https://spacy.io/usage/linguistic-features#retokenization