When does a GPU Matter?

[KTRosenberg]

I am trying to figure out if a GPU / CUDA are beneficial if I only want to use Spacy to parse text, and not train models. Is there a benefit, or should I instead worry about getting the most CPU power? (The idea is to run spacy on a compute server.)

Thanks!

[polm]

A GPU should always give increased performance (in terms of words per second processed). For CPU models this difference is noticeable but not major, for Transformers it's very significant. For Transformers particularly the difference is more exaggerated in training, but also present even if you're just doing inference.

It should be easy to run the Transformers models on CPU and see if the speed is acceptable for you - for training I assume it's basically unusable. Some people have apparently been able to use it for inference, but I would assume that's the exception rather than the rule.

[KTRosenberg]

That makes sense. One last thing: has anyone been able to get this working with the M1 macs, and on Windows, where can I find the documentation for enabling GPU acceleration after I've installed a version of Spacy with CUDA? I assume I need to figure out which version of CUDA a given GPU supports. e.g. an old NVIDIA 1080.

[polm]

I think it works with M1s now; there's several threads on it if you have trouble, or feel free to open a new one. On Windows it should just work.

You need to install libraries that match the version of CUDA you have installed, please see the install widget.

[BramVanroy]

As an fyi, as @polm noted the speed-ups you cam get with a GPU are very noticeable, also during inference. If you only have little data to parse, it is probably not necessary. But it adds up quickly.

from thinc.api import require_gpu  # OR prefer_gpu, which will not error if no CUDA
from thinc.backends import use_pytorch_for_gpu_memory  # If you are working with transformer models

use_pytorch_for_gpu_memory()
require_gpu()
nlp = spacy.load(...)

[KTRosenberg]

Thanks! Is there also an example somewhere of enabling GPU use on M1, or is the process the same with thinc?

[BramVanroy]

I'm not at all sure about this but I think if you install with

pip install spacy[apple]

you get those speed improvements out of the box.