Incorrect tokenization of dash punctuation in Spanish when not preceded or followed by a space

[jonathanknebel]

This is related to this (now closed) issue: #3277.

How to reproduce the behaviour

Per the fixes related to the above issue (https://github.com/explosion/spaCy/pull/3281/files), the en/em dash now tokenizes into a separate token whenever it is preceded or followed by a space, but whenever this dash is connected to another word or punctuation mark on one side without a space between and also connected to another word or punctuation mark on the other side without a space between, then it seems to be treated as a hyphen and is not tokenized. In fact, it will even cause the dash and it's preceding punctuation mark to be combined into a single token together with whatever word precedes the preceding punctuation mark and whatever word follows the dash, as in the example below.

["—", "Pues", "bien,—dijo", "el", "extranjero,—el", "año", "que", "viene", "debe", "Vd.", "hacer", "el", "tiempo", "para", "sus", "viñas", "."]

There are many instances in which this Spanish dash is connected on both sides in dialogue. Almost every Spanish book at Gutenberg.org that has dialogue has examples of this: https://www.gutenberg.org/ebooks/search/?query=l.spanish

The issue is not unique to 3.5--it's an issue with previous versions of Spacy as well.

Your Environment

• spaCy version: 3.5.0
• Platform: Windows-10-10.0.22623-SP0
• Python version: 3.11.0
• Pipelines: en_core_web_sm (3.5.0), es_core_news_sm (3.5.0), zh_core_web_sm (3.5.0)

[danieldk]

Thank you for reporting this issue! We will evaluate whether it makes sense to change the Spanish default (since it would break the opposite case where text mistakingly uses e/em dashes in place of hyphens).

In the meanwhile, you could add en/em dash to the the tokenizer's infix expression to handle this. For example:

>>> import spacy
>>> nlp = spacy.blank("es")
>>> infixes = spacy.util.compile_infix_regex(nlp.Defaults().infixes + ['—'])
>>> nlp.tokenizer.infix_finditer = infixes.finditer
>>> [token.text for token in nlp("que era hombre docto, graduado en Sigüenza—,")]
['que', 'era', 'hombre', 'docto', ',', 'graduado', 'en', 'Sigüenza', '—', ',']

[jonathanknebel]

Thank you. I ended up implementing basically this very workaround.