Spacy v2.2.43 TOKENIZE incorrect result

[TimVanDorpe]

Hi,

So, I'm using spacy to tokenize sentences in my application.
Simple things like 'can't' and 'id' are split up in multiple tokens , see attachments.
Is there a reason for this splitting up or is this still an issue/bug from this version (2.2.43) ?

• Operating System: Windows-10-10.0.14393-SP0
• Python Version Used: 3.7.5
• spaCy Version Used: 2.2.43

Tim

[polm]

Splitting "can't" into two tokens is normal and intentional and common in NLP tools in general. It makes processing more consistent since it treats it as "can not". You will see this with other contractions like "don't" or "wouldn't".

"id" is a bit weird. I guess it's by relation to "I'd" but it seems to be treated as two tokens in any instance, including "Freud talked about the id a lot". That looks like a bug to me.

Either way this behavior is the same in the most recent version of spaCy.

[durbin-164]

@TimVanDorpe Hi, How did you overcome these issues?