Abbreviations Expansion

[SpyriP]

Inside "spacy/lang/es/tokenizer_exceptions.py" for Spanish language, there are the following options for the abbreviation of "Estados Unidos" :

"EE.UU.",
"Ee.Uu.",
"EE. UU.",
"Ee. Uu.",

Can we have composite expression (ie multiple characters with space) in the tokenizer exception? I ran some tests using the above expressions and it doesn't seem it works when I add the following:
"Ee.Uu.",
"EE. UU.",
"Ee. Uu.",

nlp = spacy.load("es_core_news_sm")
doc = nlp("esta bien 10 EE. UU.")
[tok.text for tok in doc]

returns:

['esta', 'bien', 'EE', '.', 'UU', '.']

nlp = spacy.load("es_core_news_sm")
doc = nlp("esta bien 10 Ee. Uu.")
[tok.text for tok in doc]

returns:

['esta', 'bien', 'Ee', '.', 'Uu', '.']

Only the expression "EE.UU." (without space) returns the correct output:

doc = nlp("esta bien 10 EE.UU.")
[tok.text for tok in doc]

['esta', 'bien', '10', 'EE.UU.']

Should we expand them in the 'expand' module instead?

[adrianeboyd]

The exception with the space should work in spaCy v3.x. It isn't supported by the tokenizer in spaCy v2.

It looks like this particular additional exception was added after es_core_news_sm v3.0.0 was trained, but you can see it in action with nlp = spacy.blank("es"). The upcoming v3.1.0 models should include it.

You can potentially modify the tokenizer in es_core_news_sm v3.0.0, but it will probably cause problems for the statistical models, which won't have seen this particular token before, so you'd have to check the output for your cases to know if it's a good idea to add it or not.

[SpyriP]

Thank you, that helps a lot.