The newly split tokens must match the text of the original token. New orths: coComo. Old text: Como.

[sontek]

I'm getting this error when I try to use spacy-affixes on my already existing library. Do you know what would cause this and how to fix it?

    doc = self.nlp(text)
../../../.local/share/virtualenvs/backend-C76VZmPc/lib/python3.7/site-packages/spacy/language.py:435: in __call__
    doc = proc(doc, **component_cfg.get(name, {}))
../../../.local/share/virtualenvs/backend-C76VZmPc/lib/python3.7/site-packages/spacy_affixes/main.py:163: in __call__
    self.apply_rules(retokenizer, token, rule)
../../../.local/share/virtualenvs/backend-C76VZmPc/lib/python3.7/site-packages/spacy_affixes/main.py:140: in apply_rules
    token, [*rule["affix_text"], token_sub], heads

the text is:

Como todos lo han hecho durante tanto tiempo.

[sontek]

Here is the whole command:

>>> import spacy
>>> from spacy_affixes import AffixesMatcher
>>> text = "Como todos lo han hecho durante tanto tiempo."
>>> nlp = spacy.load("es")
>>> affixes_matcher = AffixesMatcher(nlp, split_on=["VERB"])
>>> nlp.add_pipe(affixes_matcher, name="affixes", before="tagger")
>>> nlp(text)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/sontek/.local/share/virtualenvs/backend-C76VZmPc/lib/python3.7/site-packages/spacy/language.py", line 435, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))
  File "/Users/sontek/.local/share/virtualenvs/backend-C76VZmPc/lib/python3.7/site-packages/spacy_affixes/main.py", line 163, in __call__
    self.apply_rules(retokenizer, token, rule)
  File "/Users/sontek/.local/share/virtualenvs/backend-C76VZmPc/lib/python3.7/site-packages/spacy_affixes/main.py", line 140, in apply_rules
    token, [*rule["affix_text"], token_sub], heads
  File "_retokenize.pyx", line 88, in spacy.tokens._retokenize.Retokenizer.split
ValueError: [E117] The newly split tokens must match the text of the original token. New orths: coComo. Old text: Como.

[versae]

Thanks for reporting, @sontek!

This bug seems to be the same than #18. Prefix splitting can cause more trouble than is worth and we're actually in the process of thinking whether disabling prefixes by default.

When looking at the problematic Freeling rule (^co), the issues is that spaCy is not always reporting the correct PoS and thus some rules are triggered that should not. In the meantime, you could try only using suffixes rules (e.g., clitics) if that fits your case. We use something like this in other projects:

import spacy 
from spacy_affixes import AffixesMatcher
from spacy_affixes.utils import AFFIXES_SUFFIX
from spacy_affixes.utils import load_affixes

nlp = spacy.load("es") 

suffixes = {k: v for k, v in load_affixes().items()
            if k.startswith(AFFIXES_SUFFIX)} 
affixes_matcher = AffixesMatcher(nlp, split_on=["VERB"], rules=suffixes)
nlp.add_pipe(affixes_matcher, name="affixes", before="tagger")

Let us know if that helps in anyway!

[sontek]

Yeah, I've been having a lot of issues with spacy's parts of speech pick up. They are "90%" accurate, which means 1 and 10 of my words are wrong :)