Installing spacy-transformers on mac with m1 chip fails

[JeffryCA]

How to reproduce the behaviour

pip install spacy-transformers

and also failing (no error it just gets stuck every time):

python -m spacy download de_dep_news_trf

Your Environment

• Operating System: OS 11.2.3
• Python Version Used: 3.9 (conda)
• spaCy Version Used: 3.0.5

[adrianeboyd]

Hi, it looks like the current spacy-transformers (v1.0.1) was released before there was a version of tokenizers that can use a version of rust with support for the M1 architecture, so there's not a version of tokenizers+transformers that can be installed according to the spacy-transformers v1.0.1 requirements.

If you want to use it in the meanwhile, you can install from the current master branch in the github repo.

First, you will need to install rust (for tokenizers and spacy-alignments) and it's best to install pytorch separately first (probably: conda install pytorch -c conda-forge).

Then I think pip install https://github.com/explosion/spacy-transformers/archive/master.zip will work from the current master branch. There aren't many changes beyond extending the supported transformers version.

We can hopefully release the next version of spacy-transformers soon (will be v1.0.2) with support for newer versions of tokenizers.

[KTRosenberg]

Has anything changed with regard to this issue? I tried installing
python3 -m spacy download en_core_web_trf in a condo environment, but get a large failure message saying that rust's environment variables need to be set:

  ERROR: Command errored out with exit status 1:
   command: /Users/username/miniforge3/envs/TEST/bin/python /Users/username/miniforge3/envs/TEST/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/c6/gmbh3q_56nd078hcmng9z64h0000gn/T/tmpsi64l5j9
       cwd: /private/var/folders/c6/gmbh3q_56nd078hcmng9z64h0000gn/T/pip-install-7hurf5tb/spacy-alignments_edc726e2c8da4eef86fbfad8191f38af
  Complete output (21 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-11.0-arm64-3.9
  creating build/lib.macosx-11.0-arm64-3.9/spacy_alignments
  copying spacy_alignments/__init__.py -> build/lib.macosx-11.0-arm64-3.9/spacy_alignments
  creating build/lib.macosx-11.0-arm64-3.9/spacy_alignments/tests
  copying spacy_alignments/tests/__init__.py -> build/lib.macosx-11.0-arm64-3.9/spacy_alignments/tests
  copying spacy_alignments/tests/test_main.py -> build/lib.macosx-11.0-arm64-3.9/spacy_alignments/tests
  running egg_info
  writing spacy_alignments.egg-info/PKG-INFO
  writing dependency_links to spacy_alignments.egg-info/dependency_links.txt
  writing top-level names to spacy_alignments.egg-info/top_level.txt
  reading manifest file 'spacy_alignments.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE'
  writing manifest file 'spacy_alignments.egg-info/SOURCES.txt'
  running build_ext
  running build_rust
  error: Don't know the correct rust target for system type arm64-apple-darwin20.0.0. Please set the CARGO_BUILD_TARGET environment variable.
  ----------------------------------------
  ERROR: Failed building wheel for spacy-alignments

The same goes for spacy-tokenizers.

Is there any way to resolve this?

[adrianeboyd]

For reference, this problem first occurs with setuptools-rust==1.0.0, which was released about a week ago. Earlier versions of setuptools-rust did work as expected, which is why we hadn't seen this problem before. I submitted a bug report and my guess is that it's not going to be difficult to fix (although I'm not familiar enough with rust to know for sure), but if there's no an easy fix, we can consider restricting the version of setuptools-rust in the package setup instead.

[KTRosenberg]

@adrianeboyd Is that CUDA warning related? Is it something that can be ignored, or does it disable any use of the GPU, in which case, what else do I need to do?

[jamesonquinn]

@KTRosenberg I managed to get the warning to go away by reinstalling conda from miniforge. I could tell I needed to do that because activity monitor was showing that my python threads were running under intel emulation. Once I'd done that (and finished re-adding all the deps) it does in fact run faster (about 5x speedup on my task).

[KTRosenberg]

@jamesonquinn So you restarted? What is your full installation procedure? (If you could provide a script, that would be great.) I have to use spacy 3.1.4. Is 3.2 required? Do I just start the minigorge installation script?

Do you also set the rust arm flag?

I guess another problem could be the need for some tensorflow installation for another library, which might introduce some doengrades.

[adrianeboyd]

There's now a new version of setuptools-rust that should fix this particular issue. If you're having other problems, it would be better to open a new discussion thread.