Semicolon doesn't split tokens

[v-JiangNan]

When I tokenize the following sentence:

fast, great screen, beautiful apps for a laptop;priced at 1100 on the apple website;amazon had it for 1098+ tax - plus i had a 10% off coupon from amazon-cost me 998 plus tax- 1070- OTD!

but laptop;price and website;amazon appeared in the result. The expected results should be separated by semicolons.

This is my code:

import spacy
nlp = spacy.load("en_core_web_trf")

text = "fast, great screen, beautiful apps for a laptop;priced at 1100 on the apple website;amazon had it for 1098+ tax -  plus i had a 10% off coupon from amazon-cost me 998 plus tax- 1070- OTD!"
doc = nlp(text)

for token in doc:
    print(token.text)

I have tried token.lemma_, token.orth_ and Tokenizer, but it still doesn't work. What should I do？

Info about spaCy

• spaCy version: 3.1.2
• Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.7.11
• Pipelines: en_core_web_trf (3.1.0)

[polm]

This is happening because the semicolon is not in the list of infixes because normally something like "hot;dog" is a typo, not valid text. You can get what you want by adding the semicolon to the list of infixes.

import spacy
nlp = spacy.blank("en")


# fix it
infixes = nlp.Defaults.infixes + [';']
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer

text = "fast, great screen, beautiful apps for a laptop;priced at 1100 on the apple website;amazon had it for 1098+ tax -  plus i had a 10% off coupon from amazon-cost me 998 plus tax- 1070- OTD!"
doc = nlp(text)

for token in doc:
    print(token.text)

You can see an overview of this in the docs here.