Attribute ruler and Parser inconsistent state

[databill86]

How to reproduce the behaviour

This is a follow up on the issue 9782. It's closed but it's still unresolved in the new spaCy relase: 3.3.0

The fix mentioned in the link here by @adrianeboyd is working, but can be an issue if we need to use the dependencies tags on a pipeline later on the model.

Your Environment

• Operating System: Unix
• Python Version Used: 3.8
• spaCy Version Used: 3.3.1

[adrianeboyd]

This issue should be fixed in the v3.3.0 model releases. Double-check that you've also upgraded the models and not just spacy itself by using spacy validate?

With spacy v3.3.1 and en_core_web_sm v3.3.0, the test case from #9782 works on my end:

>>> import en_core_web_sm
>>> nlp = en_core_web_sm.load(disable=['parser'])
>>> text = 'Big  news. Great deals.'
>>> doc = nlp(text)
>>> print([(t.i, t.text) for t in doc])
[(0, 'Big'), (1, ' '), (2, 'news'), (3, '.'), (4, 'Great'), (5, 'deals'), (6, '.')]
>>> doc[4].sent_start = True
>>>

The v3.2.0 models are not going to change in terms of this behavior because it's a problem with the attribute ruler rules in the model rather than a bug within the spacy library.

[databill86]

Thank you for your response. I'm using the last version of spaCy, as well as the last version of the model.

Actually, it's more about using the Parser with the Attribute ruler. When "parser" is enabled (need to use "DEP" in some matcher other pipeline), along with the attribute ruler and some segmentation pipeline, I get this error:

...
File "spacy/tokens/token.pyx", line 501, in spacy.tokens.token.Token.is_sent_start.set
ValueError: [E043] Refusing to write to token.sent_start if its document is parsed, because this may cause inconsistent state.

If I add this to the attribute ruler:

{'attrs': {'DEP': '', 'HEAD': 0, 'SENT_START': 0},
                   'index': 0,
                   'patterns': [[{}]]}

I do not get the error anymore, but the document will not be parsed, and I can't get dependency tags anymore.
doc.has_annotation("DEP") will be always False.

I made a reproducible example here: https://github.com/databill86/spacy-tests

If you comment this line: https://github.com/databill86/spacy-tests/blob/3a33c03931e95d26195df10ed7fb7df9aebfae7a/src/my_pipelines.py#L38

You will see the error.

[adrianeboyd]

It's a restriction in spacy Doc objects that the sentence boundaries have to correspond to the dependency parses. So if your doc includes any dependency annotation (even a partial parse), the only way to modify the sentence boundaries is to modify the parses.

If you delete all the dependency annotation (as in your example rule), then you can set the sentence boundaries again with token.is_sent_start.

[databill86]

So we can't use the dependency parser annotations if we are changing the sentence boundaries at some point the pipelines.
The only way I see here is to use custom extensions to store some of the annotations. This way, I will be able to exploit them in my matcher.

[adrianeboyd]

That sounds like a good workaround!