ValueError: Unpack failed: incomplete input despite conda updates

[CharlyWargnier]

Heya,

I keep having the following error.

ValueError: Unpack failed: incomplete input

I followed previous recommendations and updated Spacy in my conda environment via the following commands:

python -m spacy download en
conda install -c conda-forge spacy

However, the issue is still there.

Thanks
Charly

[adrianeboyd]

It sounds like something may be corrupted in your cache. You do need to install spacy first before trying to download a model. In a new conda environment, can you try with:

conda install -c conda-forge spacy
python -m spacy download en_core_web_sm --no-cache-dir

The en shortcut no longer works in spacy v3.

[CharlyWargnier]

Thanks Adrian

I've reinstalled Spacy in a new Conda env, yet I still have the issue.

Here's the log trace I've got:

Traceback (most recent call last):
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\streamlit\script_runner.py", line 354, in _run_script
    exec(code, module.__dict__)
  File "C:\Users\Charly\Desktop\All new Apps\Text-Anonymizer\app.py", line 38, in <module>
    models = load_models()
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\streamlit\legacy_caching\caching.py", line 576, in wrapped_func
    return get_or_create_cached_value()
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\streamlit\legacy_caching\caching.py", line 556, in get_or_create_cached_value
    return_value = func(*args, **kwargs)
  File "C:\Users\Charly\Desktop\All new Apps\Text-Anonymizer\app.py", line 9, in load_models
    english_model = spacy.load("./models/en/")
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\__init__.py", line 51, in load
    return util.load_model(
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\util.py", line 422, in load_model
    return load_model_from_path(Path(name), **kwargs)  # type: ignore[arg-type]
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\util.py", line 489, in load_model_from_path
    return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\language.py", line 2039, in from_disk
    util.from_disk(path, deserializers, exclude)  # type: ignore[arg-type]
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\util.py", line 1300, in from_disk
    reader(path / key)
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\language.py", line 2033, in <lambda>
    deserializers[name] = lambda p, proc=proc: proc.from_disk(  # type: ignore[misc]
  File "spacy\pipeline\transition_parser.pyx", line 588, in spacy.pipeline.transition_parser.Parser.from_disk
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\spacy\util.py", line 1300, in from_disk
    reader(path / key)
  File "spacy\pipeline\transition_parser.pyx", line 584, in spacy.pipeline.transition_parser.Parser.from_disk.lambda7
  File "spacy\pipeline\_parser_internals\transition_system.pyx", line 228, in spacy.pipeline._parser_internals.transition_system.TransitionSystem.from_disk
  File "spacy\pipeline\_parser_internals\transition_system.pyx", line 243, in spacy.pipeline._parser_internals.transition_system.TransitionSystem.from_bytes
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\srsly\_msgpack_api.py", line 27, in msgpack_loads
    msg = msgpack.loads(data, raw=False, use_list=use_list)
  File "C:\Users\Charly\miniconda3\envs\SpacyNew\lib\site-packages\srsly\msgpack\__init__.py", line 79, in unpackb
    return _unpackb(packed, **kwargs)
  File "srsly\msgpack\_unpacker.pyx", line 203, in srsly.msgpack._unpacker.unpackb
ValueError: Unpack failed: incomplete input

[polm]

? Sorry, I don't understand what you mean. The issue you link to is very old and was just checking part of the model.

When you checked the hash of the file, what was the file you were checking the hash of? I'm not sure if you were checking the whole .tar.gz or the files for individual components.

Regardless of that, maybe you can try downloading a newer version of spaCy / the models? We're up to 3.2 now.

[minylora]

That post just gave us the idea that a corrupted file was possible, so I tried multiple ways to verify if I had a bad version or corrupted download, and after trying a number of different things to check parser and also the rest of en_core_web_sm, I am pretty confident its not the files themselves at this point.

The issue is really that we get this ValueError when trying to load en_core_web_sm-3.0.0 with spacy.load(), and it only loads successfully if the excluded_pipes parameter contains the parser.

[polm]

I'm sorry, I still don't know what's going on.

msgpack is a binary format that prefixes some kinds of data with a length and then writes them. The "incomplete data" error should only happen if there are fewer bytes than expected, which can only happen in a truncated file. It's possible there's a bug in msgpack or something, but without access to the files you're working with or other reports of the problem it's hard to say what's going on.

Since the issue seems to be affecting that particular file, I would again recommend you try installing a different version of the same model.

[minylora]

I have very good news! It works now! It was a git-windows issue with the model file as described in your docs here https://spacy.io/usage#unhashable-list which means, just as you expected, there was a problem with the model file.

Added this to a gitattributes file in the repo:
baserepopath/en_core_web_sm/en_core_web_sm-3.0.0/parser/** text eol=lf

And now spacy.load() has no issues.

[polm]

Glad you figured it out, and thanks for letting us know - if anyone else has the same issue this will be a big help.