Pickle loading errors (pickle saved on windows, loading attempt on linux)

[gilo-andreja]

The pickle is created (saved) on windows. I need to run this code (load the pickles) on Linux (Ubuntu).
The script successfully runs on the windows machine where the pickle is created.

### Code that causes this error

import platform
import pathlib
plt = platform.system()
if plt == 'Linux': pathlib.WindowsPath = pathlib.PosixPath
curdir = os.path.dirname(os.path.abspath(__file__)) + "/"
print ("curdir = " + curdir)
nlp = pickle.load(open(curdir + "nlp_3words.pickle", "rb"))
matcher = pickle.load(open(curdir + "matcher_3words.pickle", "rb"))

### Output

Loading Spacy matcher
curdir = /home/osboxes/Desktop/gilo-api-server/gilo-api/apis/KeywordExtractor/
OverflowError: value too large to convert to uint32_t
Exception ignored in: 'preshed.bloom.bloom_from_bytes'
Traceback (most recent call last):
  File "/home/osboxes/Desktop/gilo-api-server/gilo-api/./apis/KeywordExtractor/Keyword_Extractor_API.py", line 62, in <module>
    nlp = pickle.load(open(curdir + "nlp_3words.pickle", "rb"))
OverflowError: value too large to convert to uint32_t
OverflowError: value too large to convert to uint32_t
Exception ignored in: 'preshed.bloom.bloom_from_bytes'
Traceback (most recent call last):
  File "/home/osboxes/Desktop/gilo-api-server/gilo-api/gilo-api/lib/python3.10/site-packages/srsly/_pickle_api.py", line 23, in pickle_loads
    return cloudpickle.loads(data)
OverflowError: value too large to convert to uint32_t
ValueError: bytes length not a multiple of item size
Exception ignored in: 'preshed.bloom.bloom_from_bytes'
Traceback (most recent call last):
  File "/home/osboxes/Desktop/gilo-api-server/gilo-api/./apis/KeywordExtractor/Keyword_Extractor_API.py", line 62, in <module>
    nlp = pickle.load(open(curdir + "nlp_3words.pickle", "rb"))
ValueError: bytes length not a multiple of item size
OverflowError: value too large to convert to uint32_t
Exception ignored in: 'preshed.bloom.bloom_from_bytes'
Traceback (most recent call last):
  File "/home/osboxes/Desktop/gilo-api-server/gilo-api/./apis/KeywordExtractor/Keyword_Extractor_API.py", line 62, in <module>
    nlp = pickle.load(open(curdir + "nlp_3words.pickle", "rb"))
OverflowError: value too large to convert to uint32_t
OverflowError: value too large to convert to uint32_t
Exception ignored in: 'preshed.bloom.bloom_from_bytes'
Traceback (most recent call last):
  File "/home/osboxes/Desktop/gilo-api-server/gilo-api/./apis/KeywordExtractor/Keyword_Extractor_API.py", line 62, in <module>
    nlp = pickle.load(open(curdir + "nlp_3words.pickle", "rb"))
OverflowError: value too large to convert to uint32_t

[polm]

Is this related to #11475?

Can you share the code that creates the pickle?

If you're saving an nlp object, it's usually better to save it with nlp.to_disk than to pickle it. You can read about how to save and load nlp objects here.

[gilo-andreja]

#11475 is a different issue.

I don't have the code that creates the pickle, I will ask for it and copy it here.

[mahagilo]

Here is the code that creates the pickle:

import spacy
import pickle
import os
from spacy.matcher import Matcher
nlp = spacy.load( "en_core_web_md", exclude=["morphologizer, ner, senter, sentencizer", "entity_linker", "entity_ruler", "textcat_multilabel"])

import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from structure_db import *   # KnowledgeDB Python classes
from const import *          # DB connection details

engine = sqlalchemy.create_engine(f'postgresql://{DB_user}:{DB_pwd}@{DB_host}:5432/{DB_name}')
session_maker = sessionmaker() #create a session object
session = session_maker(bind=engine) #bind postgresql connection to session to get open DB connection
print("Session with", DB_host, DB_name, DB_user)

import spacy
import pickle
import os
from spacy.matcher import Matcher
nlp = spacy.load( "en_core_web_md", exclude=["morphologizer, ner, senter, sentencizer", "entity_linker", "entity_ruler", "textcat_multilabel"])
save_nlp = "nlp_keyword_acro_compact.pickle"
save_matcher = "matcher_keyword_acro_compact.pickle"

def make_into_pattern(keyword):
    # Checks if a token is the final token in a span or doc
    is_final_token = lambda token, span: token.i == len(span) - 1
    
    tmpdoc = nlp(keyword)
    has_hyphen = False
    pattern = []
    hyphenless_pattern = []
    for token in tmpdoc:
        is_hyphen = token.lower_ == "-"
        if is_hyphen: has_hyphen = True
        
        # The last word of a multi-word entity is the one which carries inflection, so we match on the lemma for that one
        tokentext = token.text
        
        #pat_piece = ({"LEMMA": token.lemma_.lower()} if is_final_token(token, tmpdoc)
        #        else {"LOWER": token.lower_})
        
        token.set_extension("lemma_lower", force=True, getter=lambda t: t.lemma_.lower())
        pat_piece = {"_": {"lemma_lower": token.lemma_.lower()}} if is_final_token(token, tmpdoc) else {"LOWER": token.lower_}
        
        if has_hyphen:
            if len(hyphenless_pattern) == 0:
                   hyphenless_pattern = pattern.copy()
            if not is_hyphen: hyphenless_pattern.append(pat_piece)
        pattern.append(pat_piece)
    full_pattern = [pattern]
    if has_hyphen: full_pattern.append(hyphenless_pattern)
    return full_pattern

def make_acronym_pattern(acronym):
    pattern = []
    plural_pattern = []
    for word in acronym.split():
        pattern.append({"TEXT": word})
        plural_pattern.append({"TEXT": word + "s"})
    full_pattern = [pattern]
    full_pattern.append(plural_pattern)
    return full_pattern

# -------------------------------------------------------------------------------------------------------
# ------------------------ Build Spacy Matcher from keywords SQL and save as Pickle ---------------------
# -------------------------------------------------------------------------------------------------------

# Step 1 Load keywords and acronyms from SQL
keylist = []
print("Loading keywords")
result = session.query(Keywords_distinct).distinct()
for row in result:
     keylist.append(row.keyword)

print("Loading acronyms")
acronymlist = []
result = session.query(Acronyms.acronym).distinct()
for row in result:
     acronymlist.append(row.acronym)

# Step 2 BUILD SPACY MATCHER
print("Building Spacy Matcher")
matcher = Matcher(nlp.vocab)

for term in keylist:
    matcher.add(term, make_into_pattern(term))
      
for term in acronymlist:
    matcher.add(term, make_acronym_pattern(term))

# Step 3 Save Matcher
with open(save_matcher, "wb") as f: pickle.dump(matcher, f)
with open(save_nlp, "wb") as f: pickle.dump(nlp, f)
print("saved as:", save_matcher, "-", save_nlp)

[polm]

Thanks for the code. That gives me a better idea of what you're trying to do, though I'm still confused about what's happening here.

Let me start by saying that for an nlp object, it is usually easier to save and load it using nlp.to_disk / spacy.load, as recommended here. I would strongly recommend you try that first if you haven't done so yet. If you can't use that for some reason, it would be helpful to know why.

In the initial post, in the code, the nlp object is loaded before the Matcher. But in the output, there's a line that says "loading spaCy Matcher" first, and then repeated errors. I'm confused about why the error happens multiple times and the order of output. That isn't critical, but just in case I'm missing something, could you check that the code and error actually match?

It would help to know what version of Python you're using in both environments, and the versions of spaCy and preshed. You can check versions with pip freeze (or conda equivalent).

I'll try reproducing this by saving a pipeline from Windows to Linux with pickle...

[polm]

I was able to reproduce this, so don't worry about my comments about the output or versions. This seems to be an issue with how the lookups are serialized by pickle. I'll look into it but it may take a while.

In the meantime, I suggest you use the to_disk method I mentioned above, or the to_bytes method covered on the same page.