Unpickling a pickled List of Doc objects.

[hrshdhgd]

I have referred to the following issues within the spaCy repo to get answers for my issue but I haven't found a solution:

• PhraseMatcher persistance issue #5493 (comment)
• serializing phrase matcher to_disk. #5096 (comment)
• Serialize PhraseMatcher for faster performance #4445 (comment)
• PhraseMatcher Silently Fails After Pickling #4333 (comment) [Old version of spaCy]
• Pickling of PhraseMatcher not yet resolved in 2.1.3 #3494 (comment) [Old version of spaCy]

How to reproduce the behaviour

Here's my code to generate terms_dict, pattern_dict and doc_obj [list of Doc objects]

def get_terms_patterns(self, *args):
        """Get dictionaries of terms and patterns,
        along with doc format of the term.

        :return: Dictionary of terms;
        Dictionary of patterns;
        doc object of the term
        """
        origin, object_id, matched_term, description, object_category = args[0]
        terms_dict = {}
        pattern_dict = {}
        object_match_field = ""

        if "[SYNONYM_OF:" in description:
            object_label = description.split("[SYNONYM_OF:")[-1].rstrip("]")
            object_match_field = "hasRelatedSynonym"
        else:
            object_label = matched_term
            # object_match_field = "isExactMatch"

        if matched_term is not None and matched_term == matched_term:
            terms_dict[matched_term.lower()] = {
                "object_id": object_id,
                "object_category": object_category,
                "object_label": object_label,
                "object_match_field": object_match_field,
                "origin": origin,
            }
            pattern_dict["id"] = object_id
            pattern_dict["label"] = origin.split(".")[
                0
            ]  # could be object_id/object_label
            pattern_dict["pattern"] = matched_term

    return terms_dict, pattern_dict, self.nlp(matched_term)

This is part of the code where this function is called:

        for (
                origin,
                object_id,
                matched_term,
                description,
                object_category,
            ) in df.to_records(index=False):
                terms, patterns, object_doc = self.get_terms_patterns(
                    (
                        origin,
                        object_id,
                        matched_term,
                        description,
                        object_category,
                    )
                )
                self.terms.update(terms)
                self.list_of_pattern_dicts.append(patterns)
                self.list_of_obj_docs.append(object_doc)

Here's the pickling code:

        with open(TERMS_PICKLED, "wb") as tp:
            pickle.dump(self.terms, tp)
        with open(PATTERN_LIST_PICKLED, "wb") as plp:
            pickle.dump(self.list_of_pattern_dicts, plp)
        with open(OBJ_DOC_LIST_PICKLED, "wb") as odlp:
            pickle.dump(self.list_of_obj_docs, odlp)

where the variables in UPPERCASE are just filenames.

When I use pickle.load() on self.terms and self.list_of_pattern_dicts , It works as expected. When I use the same to load self.list_of_obj_docs,

with open(input_fn, "rb") as f:
    test = pickle.load(f)

it gives me the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-44-9d28b4ea6fab> in <module>
     15 
     16 with open(input_fn, "rb") as f:
---> 17     test = pickle.load(f)

../lib/python3.9/site-packages/spacy/tokens/doc.pyx in spacy.tokens.doc.unpickle_doc()

ValueError: need more than 4 values to unpack

I looked at spacy.tokens.doc.unpickle_doc() but couldn't really find an issue. I did notice there is a free floating line in the code (Line 1739). I'm not sure if that's intended or an oversight, just thought of bringing it to your attention.

Any suggestions/guidance would be greatly appreciated.

Your Environment

• Operating System: macOS Monterey (12.2.1)
• Python Version Used: 3.9.2
• spaCy Version Used: v3.2.3
• Environment Information: I am using Anaconda as my virtual environment v4.12.0

[adrianeboyd]

Hmm, I can't reproduce this. That line in the code is what tells pickle how to handle Doc objects.

In general, you should use a DocBin to save your docs instead of pickle. It's much more efficient, safe, and portable: https://spacy.io/usage/saving-loading#docs

If you really want to use pickle: my initial guesses might be that the file is corrupted somehow or that there are differences in the environments where you're saving/loading?

If you install all the dev dependencies, you can run a few tests to make sure that nothing is wrong with your install (would be unlikely):

python -m pip install -r https://raw.githubusercontent.com/explosion/spaCy/v3.2.3/requirements.txt
python -m pytest --pyargs spacy.tests.doc.test_pickle_doc

[hrshdhgd]

Thank you for the suggestion, @adrianeboyd ! I ran the pytest command the test passed in both environments (where I run pickle.dump() and pickle.load())

If you really want to use pickle: my initial guesses might be that the file is corrupted somehow or that there are differences in the environments where you're saving/loading?

You are spot-on! I cloned environment1 and created environment2 and pickle.load() code works!

Having said that, I'm still interested in employing the DocBin strategy.

Just so that I understand correctly, you're suggesting to use DocBin to save docs. As per the example above, I should add the lines:

self.doc_bin = DocBin(store_user_data=True)

and later in the code instead of self.list_of_obj_docs.append(object_doc) add

self.doc_bin.add(object_doc)

and after the for loop is complete, export self.doc_bin using:

self.doc_bin.to_disk("doc_obj.spacy")

instead of pickle?

So I did that, and then tried to retrieve it using the following code:

import spacy
from spacy.tokens import DocBin

test = DocBin()
test.from_disk("doc_obj.spacy")

that gives me <spacy.tokens._serialize.DocBin at 0x7fa640723760> . How do I retrieve/view the actual data that was serialized and exported?

[adrianeboyd]

You just need to use DocBin.get_docs(), see the example in the same link: https://spacy.io/usage/saving-loading#docs

And the API docs (but there's not much else to it): https://spacy.io/api/docbin