v3.5 models in conda-forge?

[hpcpony]

Should there be a new set of v3.5.0 models in conda-forge to match the release of spaCY v.3.5?

[polm]

Sorry this is confusing. The conda-forge models are unofficial mirrors and we don't control them. We recommend installing pretrained pipelines using spacy download, as outlined in the install widget.

[hpcpony]

Thanks. I didn't realize that the spacy stuff wasn't handled by the same group of people. I'll keep an eye out. I assume they'll show up there eventually.

[adrianeboyd]

Just to clarify, we do manage all the spacy-related python packages on conda-forge, just not any of spacy model conda packages. (We don't have model packages on pypi, either, since they're too large. We just have them hosted on github where you can download them through spacy download.)

[hpcpony]

Just to follow up...

I tried using spaCy to install the models and ran into trouble. As soon as I installed the 'en_core_web_trf' model my installation broke. My site may be bit unconventional in terms of configuration, or maybe I'm just doing something stupid, but I'll document here just in case it can help someone in the future.

Here's the basic steps. I had to elide much of the outputs to keep it managable.

[user@sys4 spaCy]$ uname -a
Linux sys4 5.4.206-200.el7.x86_64 #1 SMP Thu Jul 28 14:58:01 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[user@sys4 spaCy]$ cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)

[user@sys4 spaCy]$ /home/user/Companies/Anaconda/Miniconda3-py39_4.12.0-Linux-x86_64.sh -b -p /public/opt/NLP/spacy-3.5.0 

[user@sys4 spaCy]$ cat pkgs.350
spacy=3.5.0 
jellyfish
editdistance
jupyter
matplotlib
pandas
datasets
cython

[user@sys4 spaCy]$ conda config --set channel_priority flexible
[user@sys4 spaCy]$ conda install -q --file pkgs.350
  .
  .
  .

[user@sys4 spaCy]$ python -m spacy download en_core_web_sm
  .
  .
  .
[user@sys4 spaCy]$ python -m spacy download en_core_web_md
  .
  .
  .
[user@sys4 spaCy]$ python -m spacy download en_core_web_lg
  .
  .
  .

[user@sys4 spaCy]$ python -m spacy info

============================== Info about spaCy ==============================

spaCy version    3.5.0                         
Location         /public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy
Platform         Linux-5.4.206-200.el7.x86_64-x86_64-with-glibc2.17
Python version   3.9.12                        
Pipelines        en_core_web_sm (3.5.0), en_core_web_md (3.5.0), en_core_web_lg (3.5.0)

[user@sys4 spaCy]$ python -m spacy download en_core_web_trf
  .
  .
  .

[user@sys4 spaCy]$ python -m spacy info
Traceback (most recent call last):
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy/__init__.py", line 14, in <module>
    from . import pipeline  # noqa: F401
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy/pipeline/__init__.py", line 1, in <module>
    from .attributeruler import AttributeRuler
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy/pipeline/attributeruler.py", line 6, in <module>
    from .pipe import Pipe
  File "spacy/pipeline/pipe.pyx", line 1, in init spacy.pipeline.pipe
  File "spacy/vocab.pyx", line 1, in init spacy.vocab
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy/tokens/__init__.py", line 1, in <module>
    from .doc import Doc
  File "spacy/tokens/doc.pyx", line 22, in init spacy.tokens.doc
  File "/public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy/tokens/_dict_proxies.py", line 7, in <module>
    from .span_group import SpanGroup
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /public/opt/NLP/spacy-3.5.0/lib/python3.9/site-packages/spacy/tokens/span_group.cpython-39-x86_64-linux-gnu.so)

So, everything installs fine until the 'trf' model. Then the library paths for loading shared objects gets "corrupted". It's interesting to note that if you're on a new enough system the system /lib64/libstdc++ might be new enough that it will work. You'll be using the wrong library (i.e. not the one in the Python space) and never even know it. But, because I'm on an older CentOS box things break.

This is getting beyond my area of expertise, but I tried seeing what the loader was doing and it looked to me like after loading the trf model there are some cuda libraries that maybe corrupt where things get looked for (or what gets cached). I don't have a GPU and don't really need CUDA libraries but it seems to want to load a bunch of needless stuff anyway. I don't now how to just turn of those libraries and see if it fixes the problem.

Since I started debugging this the 3.5 models have made it into conda-forge. (Thanks to whoever does that ;^)) I did try installing things from there (which is my usual way of doing things) and everything works fine (so far). It's kinda hard to make apples-to-apples comparison because the spaCy install brings in torch 1.13.1 from pypi whereas the conda-forge install gets 1.12.1 from conda-forge.

Anyway hope that helps someone in the future.

[adrianeboyd]

Installing the correct version of torch for your setup can be tricky. In general, we recommend following the torch quickstart based on your setup (conda/pip/etc.) and then installing spacy: https://spacy.io/usage/embeddings-transformers#transformers-installation

We realize that this doesn't work well in practice when the dependencies are pulled in from the en_core_web_trf pip package because spacy download defaults to pip install torch in the background, but it's tricky for us to do anything more complicated within spacy download (which is itself pip install underneath and has all the limitations of pip and wheel packages).

[hpcpony]

Thanks for the additional info. I seem to have a knack for finding the worst ways to install software. Since I started with the fragile Python/conda/pip ecosystem my luck's only gotten worse. Anyway, I do try to stay away from pip if I can so I can avoid things like setting up a spaCy 3.5.0 installation then using pip to install spacy-experimental only to have pip go and downgrade all the 3.5.0 stuff to 3.4.4 (except of course the models which it leaves as 3.5.0). sigh