Unable to load model on VM, getting error 'utf-8' codec can't decode 0x86 in position 0: UnicodeDecodeError

[Anand195]

Hello,guys

I have successfully trained the model using GPU enabled system and now I want this model to be used on my VM.

While performing spacy.load("/Users/home/djnago/model-last")

getting the error

/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/util.py:877: UserWarning: [W095] Model 'en_pipeline' (0.0.0) was trained with spaCy v3.4 and may not be 100% compatible with the current version (3.5.0). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
  warnings.warn(warn_msg)
Traceback (most recent call last):
  File "/home/ubuntu/ResumeParser/manage.py", line 22, in <module>
    main()
  File "/home/ubuntu/ResumeParser/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
    utility.execute()
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/management/__init__.py", line 440, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/management/base.py", line 402, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/management/base.py", line 443, in execute
    self.check()
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/management/base.py", line 475, in check
    all_issues = checks.run_checks(
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/checks/registry.py", line 88, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/checks/urls.py", line 14, in check_url_config
    return check_resolver(resolver)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/core/checks/urls.py", line 24, in check_resolver
    return check_method()
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/urls/resolvers.py", line 494, in check
    for pattern in self.url_patterns:
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/urls/resolvers.py", line 715, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/utils/functional.py", line 57, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/django/urls/resolvers.py", line 708, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ubuntu/ResumeParser/cv_parser_api/urls.py", line 18, in <module>
    from resume_parser.views import home
  File "/home/ubuntu/ResumeParser/resume_parser/views.py", line 21, in <module>
    entityNlp = spacy.load(os.path.join(os.path.dirname(os.path.dirname(os.path.realpath(__file__))),"model/model-last-ner-18"))
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/__init__.py", line 54, in load
    return util.load_model(
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/util.py", line 434, in load_model
    return load_model_from_path(Path(name), **kwargs)  # type: ignore[arg-type]
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/util.py", line 514, in load_model_from_path
    return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/language.py", line 2125, in from_disk
    util.from_disk(path, deserializers, exclude)  # type: ignore[arg-type]
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/util.py", line 1352, in from_disk
    reader(path / key)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/language.py", line 2119, in <lambda>
    deserializers[name] = lambda p, proc=proc: proc.from_disk(  # type: ignore[misc]
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy_transformers/pipeline_component.py", line 419, in from_disk
    util.from_disk(path, deserialize, exclude)  # type: ignore
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy/util.py", line 1352, in from_disk
    reader(path / key)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy_transformers/pipeline_component.py", line 393, in load_model
    self.model.from_bytes(mfile.read())
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/thinc/model.py", line 619, in from_bytes
    return self.from_dict(msg)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/thinc/model.py", line 657, in from_dict
    node.shims[i].from_bytes(shim_bytes)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/spacy_transformers/layers/hf_shim.py", line 89, in from_bytes
    msg = srsly.msgpack_loads(bytes_data)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/srsly/_msgpack_api.py", line 27, in msgpack_loads
    msg = msgpack.loads(data, raw=False, use_list=use_list)
  File "/home/ubuntu/ResumeParser/rParserVenv/lib/python3.10/site-packages/srsly/msgpack/__init__.py", line 79, in unpackb
    return _unpackb(packed, **kwargs)
  File "srsly/msgpack/_unpacker.pyx", line 191, in srsly.msgpack._unpacker.unpackb
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x86 in position 0: invalid start byte

Kindly provide the solution, or do let me know how to get rid of this error!

Info about spaCy

• spaCy version: 3.5.0
• Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.31
• Python version: 3.10.10
• Pipelines: en_core_web_sm (3.5.0)

• Operating System: Ubuntu 22.04 LTS
• Python Version Used: Python 3.10.10
• spaCy Version Used: 3.5.0
• Environment Information: venv

[svlandeg]

Hi! Sorry to hear you're running into this.

First off, you mention that both systems use spaCy 3.5.0 but that doesn't seem true from the warning that is printed right up front:

UserWarning: [W095] Model 'en_pipeline' (0.0.0) was trained with spaCy v3.4 and may not be 100% compatible with the current version (3.5.0). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate

To debug this, you could try again after updating both versions to 3.5 and run a quick training, e.g. by setting max_steps to 100 in the config so you don't have to wait long. Then, could you train packaging the model and load it on the other system?

If you still run into issues, it would be great if you can share all of the console commands & outputs.