Fastest way to split a sentence to tokens

[antonpibm]

Hello everyone.

I want to split a sentence into tokens using the language model/tokenizer. From my understanding there are 2 approaches I can use (1)iterate over the tokens in nlp(sentence), (2) iterate over the tokens in nlp.tokenizer.explain(sentence)

When I compare the performance of the 2 approaches, I see an interesting behavior, approach (1) is X4.5 times faster!
Now that's counter intuitive for me, as generating the document has a lot more overhead compared to only splitting the sentence into words. From here I have 2 questions:

1. Is there a faster way to split a sentence into tokens?
2. Why and How approach (1) is faster

Adding the snippet of the experiment, note that tqdm outputs both it/s and the number of processed sentences in 10 seconds:

BTW I know about pipe, want to compare the two methods for single text

[polm]

Please don't post screenshots of code or terminal output, post them as text.

nlp.tokenizer.explain is not intended for normal tokenization, it's used to explain or debug the output of the tokenizer. It is not intended to be efficient, and is not the normal way to use the tokenizer.

In spaCy, generally the fastest way to tokenize things is basically to use a blank pipeline (like spacy.blank("en")) which just runs the tokenizer. You can also just call the tokenizer directly (nlp.tokenizer(text)). Note that spaCy tokenizers don't use the language model.