Serialization of doc

[nicno90]

Serialization loss

After serializing a spacy doc with jsonpickle, the matcher has stopped working.

How to reproduce the behaviour:

nlp = spacy.load('./spacy_embedder/models/en_core_web_trf_finetuned_v1/')

doc = nlp('This is an example (keep this together)')
doc_bytes = jsonpickle.encode(doc.to_bytes())
vocab_bytes = jsonpickle.encode(nlp.vocab.to_bytes())

At this point doc_bytes and vocab bytes will be sent in a json to another service

vocab = Vocab().from_bytes(jsonpickle.decode(vocab_bytes))
doc = Doc(vocab).from_bytes(jsonpickle.decode(doc_bytes))

matcher = matcher = Matcher(doc.vocab)
matcher.add('PARENTH', [[ {'is_left_punct': True}, {'is_right_punct': False, 'OP': '*'}, {'is_right_punct': True}]])

matched_spans = matcher(doc, as_spans=True)
print(matched_spans)

There is no match:

>> []

UPDATE:

is_left_punct/is_right_punct are not included in the attributes which is causing the error. Is there any way to ensure that some attributes are ensured to be saved? lemma and lower attributes in a pattern do not work either. Is this an issue with the vocab?

Your Environment

• spaCy version: 3.1.2
• jsonpickle version: 2.0.0
• Platform: Windows-10-10.0.19041-SP0
• Python version: 3.8.6
• Pipelines: en_core_web_lg (3.0.0), en_core_web_trf (3.1.0)

[ljvmiranda921]

Hi @nicno90 !

Is there any way to ensure that some attributes are ensured to be saved?

Can you try saving your dataset as a DocBin? Ideally, this should store all attributes of your Doc. Of course, you can still use the standard pickle, but it will still require the same Vocab.

[adrianeboyd]

This line is the problem:

vocab = Vocab().from_bytes(jsonpickle.decode(vocab_bytes))

This is not creating the same English vocab as from the "en" pipeline, but a vocab for an unspecified language with no defaults for things like is_left_punct. To restore the same vocab, you want:

vocab = spacy.blank("en").vocab.from_bytes(...) # or English().vocab ...

If you're using en_core_web_trf and not customizing any lexical features (cluster, norm, prob, and sentiment are the main ones that would be saved with the vocab), you don't really need to save the vocab at all. spacy.blank("en").vocab will have the same lexical attributes asen_core_web_trf.vocab. Do be aware that the lexical attributes can change slightly across minor versions, so spacy v3.0.x might be slightly different from v3.1.x. We try to minimize the changes, but stop words in particular can get updated occasionally.

DocBin will have the same vocab issues as doc.to_bytes(), where you need a vocab for the same language to load the docs with the same lexical attributes.

Unlike doc.to_bytes(), pickle does include the vocab, but because of this it can be a lot more bloated than other options (and have other drawbacks), see: https://spacy.io/usage/saving-loading#pickle