Error when installing spacy with cupy

[sunhaozhepy]

Hi,

This is a bug report concerning using GPU with spaCy and Anaconda.
I'm new to spaCy and I tried to follow the documentation to install it. More specificially, I used:

conda create --name spacy
conda activate spacy
conda install -c conda-forge spacy
conda install -c conda-forge cupy
python -m spacy download en_core_web_trf
python -m spacy download en_core_web_sm

when I opened Python in this environment, I saw:

C:\Users\Sun\anaconda3\envs\spacy\lib\site-packages\cupy\_environment.py:213: UserWarning: CUDA path could not be detected. Set CUDA_PATH environment variable if CuPy fails to load.
  warnings.warn(

and then I tried to use GPU by typing spacy.prefer_gpu(), and:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Sun\anaconda3\envs\spacy\lib\site-packages\thinc\util.py", line 204, in prefer_gpu
    require_gpu(gpu_id=gpu_id)
  File "C:\Users\Sun\anaconda3\envs\spacy\lib\site-packages\thinc\util.py", line 214, in require_gpu
    set_current_ops(CupyOps())
  File "C:\Users\Sun\anaconda3\envs\spacy\lib\site-packages\thinc\backends\__init__.py", line 140, in set_current_ops
    set_torch_tensor_type_for_ops(ops)
  File "C:\Users\Sun\anaconda3\envs\spacy\lib\site-packages\thinc\util.py", line 548, in set_torch_tensor_type_for_ops
    torch.set_default_tensor_type("torch.cuda.FloatTensor")
  File "C:\Users\Sun\anaconda3\envs\spacy\lib\site-packages\torch\__init__.py", line 323, in set_default_tensor_type
    _C._set_default_tensor_type(t)
TypeError: type torch.cuda.FloatTensor not available. Torch not compiled with CUDA enabled.

• Operating System: Windows 10
• Python Version Used: 3.10.4 (which comes with spacy when installing)
• spaCy Version Used: 3.3.0
• Environment Information: It is a an empty environment when I installed spaCy.

[sunhaozhepy]

I noticed this when installing, if this helps:

1. cupy comes with cudatoolkit=11.7 (as the environment was empty);
2. PyTorch was installed when trying to install en_core_web_trf.

[sunhaozhepy]

I think I got it. Apparently both CuPy and PyTorch come with an abridged CUDA Toolkit, but they are not enough for spaCy to use. One must install the standalone CUDA Toolkit on NVIDIA website.

[sunhaozhepy]

I installed CUDA Toolkit and it didn't work. Guess I'll have to install it first before installing CuPy...

[sunhaozhepy]

This time with CUDA I found that without en_core_web_trf, the warning is still there, but everything else works perfectly fine: prefer_gpu() and require_gpu() returns True and en_core_web_sm works well. However, with en_core_web_trf joining the party, neither gpu calling and pipeline usage is working anymore. The error message is the same as above, even for en_core_web_sm which doesn't use PyTorch.

[sunhaozhepy]

The first issue is understandable: spaCy and CuPy uses the smaller CUDA Toolkit and creates a "sandbox" so as to isolate their runtime from the global CUDA. However, the second issue is not clear for me: it seems that I can't use en_core_web_trf anyway.

[polm]

Sorry you're having trouble with this. It looks like since you installed Torch without CUDA, it may have used a CPU-only version, and ended up caching that in conda somewhere. For example this issue seems to be the same.

I would suggest you create a new conda environment and try reinstalling everything. If that doesn't work check what version of pytorch you have installed - it shouldn't have cpu in the name.

[sunhaozhepy]

Hi Polm,

I did check and it turns out that PyTorch is indeed a CPU-only version. I will try to redo it. Thank you very much.

However, The PyTorch package was in fact a dependency of the en_core_web_trf module, so I didn't mean to install any specific version of it, it comes with the transformer module. Maybe the spaCy team should avoid using pytorch from PyPI as it automatically downloads a CPU version. Otherwise they should try to add --extra-index-url https://download.pytorch.org/whl/cu113 like on PyTorch website.

On Anaconda, using the conda install pytorch -c pytorch command is fine; but another thing is the cudatoolkit version is 11.7 from CuPy, but PyTorch needs 11.3 to download a correct CUDA version otherwise it downloads a CPU version too. Maybe when downloading PyTorch with en_core_web_trf they could try to reinstalling cudatoolkit=11.3 to ensure that a correct CUDA-enabled PyTorch version is installed.

[sunhaozhepy]

I tried conda install cudatoolkit (11.3) and conda install pytorch -c pytorch and now it works.
Another note: this is all isolated in a venv so there's no need to use the CUDA Toolkit from NVIDIA website. (At least for Anaconda)

[polm]

I think we don't mention this because the assumption is that you'll install CUDA outside of Python, since it interfaces with the kernel and is pretty heavy. But thanks for the suggestion, we'll see if there's a more convenient way to package this for conda.

[Premshay]

I have the same issue, and I installed all packages by order. It doesn't matter if I install trf or sm, the same issue persists.
CUDA path was OK, but same error when running prefer_gpu(): "TypeError: type torch.cuda.FloatTensor not available. Torch not compiled with CUDA enabled."

Solution: update pytorch after everything else.
Reason: it seems that the spacy installations installed pytorch 1.10.2, and this didn't work.
updating brought it to pytorch-1.11.0-py3.10_cuda11.3_cudnn8_0.

[sunhaozhepy]

Try to add conda install cudatoolkit and conda install pytorch -c pytorch and see if it works, it worked for me.

[Premshay]

As mentioned, I did that from the start. It brings a series of problems - 1. overrides some python scripts that lead to a total jupyter meltdown (the kernel fails to start).
After fixing that, the aforementioned issue persists, until enough updates make it work.

[sunhaozhepy]

In that case I am afraid that I don't what to do since I was trying spaCy for a very short time and with no other utility libraries like jupyter. This is another evidence to my point of view that spaCy team needs to do something about this...

[pmbaumgartner]

@Premshay @sunhaozhepy If you can keep your issues in fewer relevant threads, that will help us in responding since we're able to review more information in context.

I've just recreated a virtual environment with spaCy running using a GPU from scratch to see what the stumbling blocks were. If you can replicate the process I outline below, I think you'll be able to get spaCy working with a GPU. Note that I'm not using conda installs because it's not clear to me how conda handles extras and the specific cuda versions. That shouldn't really matter for your case though, there's no reason that you should need to use conda and not pip here.

One more caveat before I share these steps is that I notice you're on Windows. This means the results you see from using the spaCy install widget or the pytorch install widget might be different, so don't just copy the commands that I have here without using those widgets yourself.

Here we go!

---

Check your installed cuda version:

nvcc --version

You should see something like

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_Mar__8_18:18:20_PST_2022
Cuda compilation tools, release 11.6, V11.6.124
Build cuda_11.6.r11.6/compiler.31057947_0

I'm at 11.6. That means I want anything I install that uses CUDA to be version <=11.6.

Use the widget here to figure out the install command you should run. For me, it was the following:

pip install -U 'spacy[cuda116,transformers,lookups]'

But when I tried that, I got a warning:

WARNING: spacy 3.3.1 does not provide the extra 'cuda116'

I can see how this would be confusing - personally I think this should fail more loudly than it does and not install anything, but that's more of a pip thing.

So if I rerun with the cuda115 extra, everything installs correctly.

pip install -U 'spacy[cuda115,transformers,lookups]'

Now I install the transformers model:

python -m spacy download en_core_web_trf

No issues. I start up a python REPL:

import spacy
nlp = spacy.load('en_core_web_trf')

doc = nlp("This is a doc")

Yay, it works!

Now, I personally get a warning with spacy.prefer_gpu because of my specific GPU:

>>> spacy.prefer_gpu()
/home/peter/miniconda3/envs/spacy-gpu-test/lib/python3.10/site-packages/torch/cuda/__init__.py:145: UserWarning: 
NVIDIA GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

Even though things are working, I'm going to try and fix this warning. I go to the pytorch site and use their installer widget, which tells me I should use this command:

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113

But I already have torch installed, so I'll need to pass the --force-reinstall flag to pip as well. So I execute the following command:

pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113 --force-reinstall

Now, running spacy.prefer_gpu() returns True and I don't see the warning and the model seems to be working. ✔️

[Premshay]

Thank you very much, @pmbaumgartner, and sorry for the unsupervised commenting:)
This is a good reference for anyone facing similar issues.

[dave-espinosa]

Nice post @pmbaumgartner !

I think the part that definitely does the trick, is using nvcc --version instead of nvidia-smi to check the correct CUDA version (i.e., the one that you should refer to, when using the widget⚠). There are some other unofficial containment solutions accross the web, where the CUDA version is found by running nvidia-smi, which could eventually address the reader in the wrong direction 😕. Allow me then, to share "my two cents" on that matter...

In my case, I usually use Vertex AI Workbench to train my spaCy models, and when I created a new VM with a NVIDIA GPU, Google provided me some wizard, which installed all the NVIDIA drivers for me. After creating the VM, turning it on, creating and enabling a venv inside, and running the command nvidia-smi, I used to get:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   59C    P0    27W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Leading me to believe that my CUDA version was '11.4' (which of course I used for the remaining steps of the answer). When running spacy.prefer_gpu(), I used to end with a nice "cupy library was not found" error (or some other similars). Troubleshooting used to lead me into way more errors, so after a while I decided to just delete the old venv and create a new one, for newer trials. After a while and seeing no success, I just decided to use nvcc version instead of nvidia-smi (thinking I was gonna get the same '11.4') but surprisingly, what I got was:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0

Which made me realize about the right CUDA version to use, which was 11.0. I then just repeated your steps again and magic! 🧙‍♂️: The answer of this thread worked flawlessly 😎.

Just as a side note (to be %200 percent sure you won't see any errors afterwards 😅), after importing spacy and running either spacy.prefer_gpu() or spacy.require_gpu(), I ran afterwards (from CLI):

import cupy
a = cupy.zeros((1,1))
a

To get a array([[0.]]) with no errors ✅.

I think this small finding is somehow related to your answer, and of course can be of use to future readers, so they save themselves a headache, and some hours of troubleshooting 😁...

Once again, thanks for your answer!