Finetuned spacy-transformers Inference on CPU

[azeem-scienaptic]

Hello there,

To detect entities, I fine-tuned a Spacy NER model using distilbert-base-uncased with the help of a GPU that had CUDA dependencies. However, I now need to use the trained model for production purposes on a CPU. The issue I am facing is that in order to download spacy-transformers package, multiple Nvidia packages are being downloaded, which is taking up a lot of space. I am not sure if these Nvidia packages are necessary for running the model on a CPU. Can you please confirm if my assumption is correct? I have listed the packages being downloaded by spacy-transformers below.

nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)
nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)
nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)
nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)
nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)
nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)
nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)

Version:

spacy==3.5.2
spacy-alignments==0.9.0
spacy-legacy==3.0.12
spacy-loggers==1.0.4
spacy-transformers==1.2.5

[danieldk]

That's a really an interesting question! Since spacy-transformers uses PyTorch, it requires the torch package. The default build of PyTorch of PyTorch includes GPU support and for that it installs these additional CUDA packages. You can avoid this by installing a CPU-only build of PyTorch. You can find more information about how to install the CPU version on the PyTorch download page. But it amounts to installing Torch from their CPU package index:

$ pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

If you install the CPU package before installing spacy-transformers, it should not download the GPU version of Torch, since the torch dependency is already fulfilled.

[azeem-scienaptic]

That worked like a charm, thanks for the help, @danieldk