How to explicitly split hyphen/dash

[Emporea]

I tokenize a whole pandas DataFrame column with these lines of code:

def clean_pipe(col, nlp, pos_list=["NOUN", "PROPN", "VERB", "ADJ"]):
    result = []
    for doc in nlp.pipe(col):
        single_line_res = []
        for token in doc:
            if token.pos_ in pos_list:
                single_line_res.append(token.text.lower())
        result.append(single_line_res)
    return result

This wont split any words with hypens inside of them.
spacy-tokenizer will be treated as a single token.

I know this might be useful for some usecases but I need them to be seperated.
How can I explicitly tell spacy to split words with a hyphen in-between into two (possibly multiple) tokens?

[ljvmiranda921]

Hi @Emporea , you might want to define a Tokenizer with special cases. For hyphens, you can pass a regex pattern to the Tokenizer's infix_finditer param as demonstrated in this example.