Unable to pretrain with Spacy on AWS GPU (Tesla k80) - ValueError: No GPU devices detected

[Kobena-Afful]

Hi,
I am trying to pretrain some tok2vec vectors using a GPU environment in AWS and I keep getting ValueError: No GPU devices detected. Pytorch is able to detect the GPU.
I am using

• spacy 3.4.2
• python 3.8.12

The environment has CUDA11.4 installed. I installed:

• pip install cupy-cuda11x
• pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu114
• pip install -U spacy[cuda114]

I also tried:

• pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu11x
• pip install -U spacy[cuda11x]

I also tried the solution specified in [https://github.com//discussions/11224]. Trying to install cupy-wheel, threw up a series of errors. example below:

ERROR: Command errored out with exit status 1:
     command: /opt/conda/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-qa3fok0q/cupy-wheel_1fd6e04d909443e0ba2a7050b89efa87/setup.py'"'"'; __file__='"'"'/tmp/pip-install-qa3fok0q/cupy-wheel_1fd6e04d909443e0ba2a7050b89efa87/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-tzupvkue
         cwd: /tmp/pip-install-qa3fok0q/cupy-wheel_1fd6e04d909443e0ba2a7050b89efa87/
    Complete output (26 lines):
    [cupy-wheel] Trying to detect CUDA version from libraries: ['libnvrtc.so.11.2', 'libnvrtc.so.11.1', 'libnvrtc.so.11.0', 'libnvrtc.so.10.2']
    [cupy-wheel] Looking for library: libnvrtc.so.11.2
    [cupy-wheel] Failed to open libnvrtc.so.11.2: libnvrtc.so.11.2: cannot open shared object file: No such file or directory
    [cupy-wheel] Looking for library: libnvrtc.so.11.1
    [cupy-wheel] Failed to open libnvrtc.so.11.1: libnvrtc.so.11.1: cannot open shared object file: No such file or directory
    [cupy-wheel] Looking for library: libnvrtc.so.11.0
    [cupy-wheel] Failed to open libnvrtc.so.11.0: libnvrtc.so.11.0: cannot open shared object file: No such file or directory
    [cupy-wheel] Looking for library: libnvrtc.so.10.2
    [cupy-wheel] Failed to open libnvrtc.so.10.2: libnvrtc.so.10.2: cannot open shared object file: No such file or directory
    [cupy-wheel] No more candidate library to find
    [cupy-wheel] Looking for library: libamdhip64.so
    [cupy-wheel] Failed to open libamdhip64.so: libamdhip64.so: cannot open shared object file: No such file or directory
    [cupy-wheel] No more candidate library to find
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-qa3fok0q/cupy-wheel_1fd6e04d909443e0ba2a7050b89efa87/setup.py", line 295, in <module>
        main()
      File "/tmp/pip-install-qa3fok0q/cupy-wheel_1fd6e04d909443e0ba2a7050b89efa87/setup.py", line 276, in main
        package = infer_best_package()
      File "/tmp/pip-install-qa3fok0q/cupy-wheel_1fd6e04d909443e0ba2a7050b89efa87/setup.py", line 252, in infer_best_package
        raise AutoDetectionFailed(
    __main__.AutoDetectionFailed:
    ============================================================
    Unable to detect NVIDIA CUDA or AMD ROCm installation.
    ============================================================

However, when I install cupy-cuda11x, the install is successful, and I can run the following pytorch commands and get the corresponding responses:

 import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.current_device()
0
>>> torch.cuda.device(0)
<torch.cuda.device object at 0x7fc28f5c88b0>
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'Tesla K80'

nvidia-smi brings up:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:00:1E.0 Off |                    0 |
| N/A   34C    P8    28W / 149W |      0MiB / 11441MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I have been researching this for the past couple of weeks and can't seem to find a solution. If anyone has new things I can try, I will be most grateful.
thanks!

[polm]

Thanks for the report, that's an issue we haven't seen before.

As the error indicates, cupy-wheel checks a number of locations, but can't find your cuda installation. Do you know where cuda is actually installed, for example, where a file named libnvrtc.so is? It might be in a non-standard location that cupy-wheel can't find for some reason.

[Kobena-Afful]

Hi Polm,
Thanks for your response. I tried to check where cuda was installed using the command line:
dpkg --listfiles cuda

and the response I got was:

dpkg-query: package 'cuda' is not installed
Use dpkg --info (= dpkg-deb --info) to examine archive files,
and dpkg --contents (= dpkg-deb --contents) to list their contents.

Yet, when I run nvidia-smi I get:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla K80 On | 00000000:00:1E.0 Off | 0 |
| N/A 34C P8 29W / 149W | 0MiB / 11441MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

what am I missing? I have asked our server admin to see if cuda isn't installed properly.

[polm]

Try this command:

find /usr /lib /opt -name libnvrtc.so

You may get errors about not having permission on some directories, you can ignore them. But you should see at least one file listed. If not it's possible you installed your libraries in some unusual location - if you have a server administrator they should be aware of this or able to find out if it's the case.

[Kobena-Afful]

Hi Polm,
I tried that command and it didn't return anything. I have asked our admin. I think cuda is installed in an unusual location. I had him rooting around the server today and he said he will research some more and get back to me. I will provide you with more information once he gets back to me

[Kobena-Afful]

I did some digging around starting with the usr and this is what I found. I found two directories with cuda; cuda and cuda-11.2

ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/mnt$ cd /usr
ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/usr$ ls
bin games include lib lib32 local sbin share src
ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/usr$ cd local
ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/usr/local$ ls
bin cuda cuda-11.2 etc games include lib man sbin share src
ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/usr/local$ cd cuda-11.2
ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/usr/local/cuda-11.2$ ls
compat lib64 targets
ubuntu@run-6362d983fec1b307aca51c9b-76zsh:/usr/local/cuda-11.2$

[Kobena-Afful]

Hello,
So I finally figured out where cuda has been installed (or maybe it was just installed recently in this location). This is what I found:

ubuntu@run-63750cd8fec1b307aca52874-bsxvv:/mnt$ ls /usr/local
bin cuda cuda-11.2 etc games include lib man sbin share src
ubuntu@run-63750cd8fec1b307aca52874-bsxvv:/mnt$ ls /usr/local/cuda
compat lib64 targets
ubuntu@run-63750cd8fec1b307aca52874-bsxvv:/mnt$ ls /usr/local/cuda-11.2
compat lib64 targets
ubuntu@run-63750cd8fec1b307aca52874-bsxvv:/mnt$

so it appears it is installed in /usr/local. However, I still get the same error when I try to install cupy-wheel. Any suggestions will be very much appreciated.

[polm]

That doesn't seem like that weird a location, but it looks like you have CUDA 11.2 and not CUDA 11.4. Can you try installing cupy-cuda112 and Torch for CUDA 11.2 instead of 11.4, or try upgrading CUDA?

It also sounds like you might have multiple versions of CUDA installed, which would just cause problems. In that case you should uninstall one.

[Kobena-Afful]

Thanks Polm! I will try that and let you know how it goes.