Unable to train NER model using A100 GPU

[Anand195]

Hi guys
I had A100 gpu and trying to train NER model using custom dataset.

Got exception

ℹ Saving to output directory: /home/ubuntu/NER/model
ℹ Using GPU: 0
Traceback (most recent call last):
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/home/ubuntu/.local/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
setup_cli()
File "/home/ubuntu/.local/lib/python3.8/site-packages/spacy/cli/_util.py", line 71, in setup_cli
command(prog_name=COMMAND)
File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ubuntu/.local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/ubuntu/.local/lib/python3.8/site-packages/typer/main.py", line 532, in wrapper
return callback(**use_params) # type: ignore
File "/home/ubuntu/.local/lib/python3.8/site-packages/spacy/cli/train.py", line 45, in train_cli
train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
File "/home/ubuntu/.local/lib/python3.8/site-packages/spacy/cli/train.py", line 67, in train
setup_gpu(use_gpu)
File "/home/ubuntu/.local/lib/python3.8/site-packages/spacy/cli/_util.py", line 571, in setup_gpu
require_gpu(use_gpu)
File "/home/ubuntu/.local/lib/python3.8/site-packages/thinc/util.py", line 195, in require_gpu
raise ValueError("No GPU devices detected")
ValueError: No GPU devices detected
Command exited with non-zero status 1
1.46user 0.90system 0:02.07elapsed 114%CPU (0avgtext+0avgdata 1142240maxresident)k
0inputs+0outputs (2major+122131minor)pagefaults 0swaps

spacy version : 3.4
Python version : 3.8

Kindly share the solution.

[polm]

It looks like your GPU is not found. I'm not sure if the A100 requires special support or not, but to start, did you use the install widget to install spaCy with dependencies? What's the output of spacy info?

[polm]

If you could provide the output of the following that would be helpful for debugging:

python -c 'import cupy; print(cupy.__version__); print(cupy.cuda.runtime.getDeviceCount())'

[mhillebrand]

After upgrading, I had the same problem with my A6000 GPUs. I had cupy-cuda117 installed, but the above cupy test claimed No module named 'cupy'. Then I went to the cupy website and installed cupy-cuda11x. Spacy worked again, but it complained that I had two cupy packages installed.

cupy-cuda117           10.6.0
cupy-cuda11x           11.0.0

Then I uninstalled cupy-cuda117, got the No GPU devices detected error again, reinstalled cupy-cuda117, and now it works great with no errors or warnings—even though I once again have two cupy packages installed. What's going on here?

[adrianeboyd]

cupy usually doesn't work with more than one package installed, so we're not sure what's going on.

Our general advice would be to start in a fresh venv and install exactly one version of cupy, where the current newest version for CUDA 11.2+ would be cupy-cuda11x.

Alternatively you could try pip install cupy-wheel which should auto-detect the best wheel for your setup.

[mhillebrand]

Uninstalling both cupy packages and doing pip install cupy-wheel worked great. Thank you.

[rs-pawanmethre]

pip install cupy-wheel gives me this error :

Collecting cupy-wheel
Using cached cupy-wheel-12.0.0.tar.gz (3.1 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.6.0.tar.gz (3.1 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.5.0.tar.gz (3.1 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.4.0.tar.gz (3.1 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.3.0.tar.gz (3.0 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.2.0.tar.gz (3.0 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.1.0.tar.gz (3.1 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-11.0.0.tar.gz (3.1 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-10.6.0.tar.gz (2.9 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-10.5.0.tar.gz (2.9 kB)
Preparing metadata (setup.py) ... done
Using cached cupy-wheel-10.4.0.tar.gz (2.5 kB)
Preparing metadata (setup.py) ... done
ERROR: Cannot install cupy-wheel==10.4.0, cupy-wheel==10.5.0, cupy-wheel==10.6.0, cupy-wheel==11.0.0, cupy-wheel==11.1.0, cupy-wheel==11.2.0, cupy-wheel==11.3.0, cupy-wheel==11.4.0, cupy-wheel==11.5.0, cupy-wheel==11.6.0 and cupy-wheel==12.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
cupy-wheel 12.0.0 depends on cupy-cuda102==12.0.0
cupy-wheel 11.6.0 depends on cupy-cuda102==11.6.0
cupy-wheel 11.5.0 depends on cupy-cuda102==11.5.0
cupy-wheel 11.4.0 depends on cupy-cuda102==11.4.0
cupy-wheel 11.3.0 depends on cupy-cuda102==11.3.0
cupy-wheel 11.2.0 depends on cupy-cuda102==11.2.0
cupy-wheel 11.1.0 depends on cupy-cuda102==11.1.0
cupy-wheel 11.0.0 depends on cupy-cuda102==11.0.0
cupy-wheel 10.6.0 depends on cupy-cuda102==10.6.0
cupy-wheel 10.5.0 depends on cupy-cuda102==10.5.0
cupy-wheel 10.4.0 depends on cupy-cuda102==10.4.0

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

[adrianeboyd]

Please try following their instructions in a new virtual environment: https://docs.cupy.dev/en/stable/install.html#installing-cupy-from-pypi

[rs-pawanmethre]

thank you very much for your kind help . okay will check .
Also i had one more doubt :

I am using Nvidia Tesla T4 GPU has 16bg vram, when i train spacy model on 1000 data points i get memory error :

":warning: Aborting and saving the final best model. Encountered exception:
OutOfMemoryError('CUDA out of memory. Tried to allocate 354.00 MiB (GPU 0; 14.62
GiB total capacity; 12.11 GiB already allocated; 153.38 MiB free; 13.55 GiB
reserved in total by PyTorch) If reserved memory is >> allocated memory try
setting max_split_size_mb to avoid fragmentation. See documentation for Memory
Management and PYTORCH_CUDA_ALLOC_CONF')"

should i have more RAM for training or is it got something with batch size etc...i tried to reduce batch size still gives me issue . what should i do ?

[adrianeboyd]

Please open a new thread for this kind of question with more details about your config and what you are trying to do.