ERROR when loading spacy model from local file

[LGTiscar]

Hello, I am trying to load the en_core_web_sm 3.5.0 from its folder, but I keep getting an error E912:

poetry run app-start
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/luisgt/.asdf/installs/ivm-python/3.8.11/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/__init__.py", line 1, in <module>
    from mecseourlgen import __main__
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/__main__.py", line 1, in <module>
    from mecseourlgen.application.LaunchRestServerUseCase import LaunchRestServerUseCase
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/application/LaunchRestServerUseCase.py", line 4, in <module>
    from mecseourlgen.infrastructure.rest.server.AppRouter import AppRouter
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/infrastructure/rest/server/AppRouter.py", line 1, in <module>
    from mecseourlgen.infrastructure.rest.server import seo_generator, status
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/infrastructure/rest/server/seo_generator.py", line 5, in <module>
    from mecseourlgen.application.GenerateSeoComponentUseCase import (
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/application/GenerateSeoComponentUseCase.py", line 5, in <module>
    from mecseourlgen.domain.services.SeoComponentGeneratorService import (
  File "/home/luisgt/dev/mic-mecseourlgen/code/mecseourlgen/domain/services/SeoComponentGeneratorService.py", line 13, in <module>
    nlp = spacy.load(spacy_model_path)
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/__init__.py", line 54, in load
    return util.load_model(
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/util.py", line 434, in load_model
    return load_model_from_path(Path(name), **kwargs)  # type: ignore[arg-type]
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/util.py", line 514, in load_model_from_path
    return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/language.py", line 2125, in from_disk
    util.from_disk(path, deserializers, exclude)  # type: ignore[arg-type]
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/util.py", line 1352, in from_disk
    reader(path / key)
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/language.py", line 2119, in <lambda>
    deserializers[name] = lambda p, proc=proc: proc.from_disk(  # type: ignore[misc]
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/pipeline/lemmatizer.py", line 304, in from_disk
    self._validate_tables()
  File "/home/luisgt/.cache/pypoetry/virtualenvs/mecseourlgen-qGBwjy5r-py3.8/lib/python3.8/site-packages/spacy/pipeline/lemmatizer.py", line 173, in _validate_tables
    raise ValueError(
ValueError: [E912] Failed to initialize lemmatizer. Missing lemmatizer table(s) found for mode 'rule'. Required tables: ['lemma_rules']. Found: [].

Also, as I am working with Poetry, I'll include the environment info for spacy using poetry show spacy.

How to reproduce the behaviour

Just passing a path to spacy.load() method.

import spacy
spacy_model_path = (
    get_st_files_path() + "/en_core_web_sm-3.5.0/en_core_web_sm/en_core_web_sm-3.5.0"
)
nlp = spacy.load(spacy_model_path)

Your Environment

• Operating System: WSL2
• Python Version Used: 3.8.11
• spaCy Version Used: 3.5.0
• Environment Information:

poetry show spacy
 name         : spacy                                                           
 version      : 3.5.0                                                           
 description  : Industrial-strength Natural Language Processing (NLP) in Python 

dependencies
 - catalogue >=2.0.6,<2.1.0
 - cymem >=2.0.2,<2.1.0
 - jinja2 *
 - langcodes >=3.2.0,<4.0.0
 - murmurhash >=0.28.0,<1.1.0
 - numpy >=1.15.0
 - packaging >=20.0
 - pathy >=0.10.0
 - preshed >=3.0.2,<3.1.0
 - pydantic >=1.7.4,<1.8 || >1.8,<1.8.1 || >1.8.1,<1.11.0
 - requests >=2.13.0,<3.0.0
 - setuptools *
 - smart-open >=5.2.1,<7.0.0
 - spacy-legacy >=3.0.11,<3.1.0
 - spacy-loggers >=1.0.0,<2.0.0
 - srsly >=2.4.3,<3.0.0
 - thinc >=8.1.0,<8.2.0
 - tqdm >=4.38.0,<5.0.0
 - typer >=0.3.0,<0.8.0
 - wasabi >=0.9.1,<1.2.0

Thing is, I don't know if this is something related to Poetry, because I tried the same workflow in a jupyter notebook and worked as intended.

[EDIT 1]

Seems that excluding "lemmatizer" from the .load() method let's the code run smoothly but, of course, now my NLP pipeline does not have a lemmatizer...

[danieldk]

I have converted this to a discussion first, since it is not clear yet whether this is an issue with your installation or an actual bug. For an issue we need to shake out some more specific reproducible issue.

[danieldk]

Could you verify that your download directory didn't get corrupted somehow? I can't reproduce this locally:

$ curl -LO https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.5.0/en_core_web_sm-3.5.0.tar.gz
$ tar zxf en_core_web_sm-3.5.0.tar.gz
$ pip install "spacy==3.5.0"
$ python
>>> nlp = spacy.load("./en_core_web_sm-3.5.0/en_core_web_sm/en_core_web_sm-3.5.0")
>>>