add_special_case to Tokenizer based on regex

[dennymarcels]

Hi y'all,
Can I do it? I would like that any case starting with some symbol (say ¶) would return the whole case as its ORTH (currently, some of them as split, for example ¶kvhMjNNUwjbuHkVjj91G becomes ¶kvhMjNNUwjbuHkVjj91 + G).

[adrianeboyd]

You can define a token_match pattern that is matched against each full token as described here:

https://spacy.io/usage/linguistic-features#how-tokenizer-works

[dennymarcels]

Sweet!

In case anyone else wonders, this is what I did:

spacy_model.tokenizer = Tokenizer(spacy_model.vocab, token_match=re.compile(r'^¶+').search)

Works beautifully!