Transformer inference on CPU

[kanayer]

As you know, the documentation recommends having GPU and running spacy.prefer_gpu() before loading transformer models. On the machine where I don't have access to GPU I keep getting the following error:

ValueError: Gradient scaling is not supported, requires capable GPU and torch>=1.9.0

I was wondering if it is possible to spacy.load() transformer on the CPU?

[polm]

Transformers should run on CPU, and if they don't that could be a bug, though in most cases they will be too slow to be useful.

To be clear, are you getting that ValueError when running prefer_gpu() on a machine with no GPU? Or when running spacy.load()? Please provide the actual code / configuration you are using so we can understand the issue.

[kanayer]

The ValueError occurs with and without prefer_gpu(), it always occurs when running spacy.load() on a machine without GPU. The code is the following:

if model == "ko_dep_nikl_trf":
    spacy.prefer_gpu()

nlp: spacy = spacy.load(model)

You can also re-create the error by running the following in Google Colabs without choosing GPU as runtime:

!python -m pip install https://huggingface.co/explosion/en_udv25_englishewt_trf/resolve/main/en_udv25_englishewt_trf-any-py3-none-any.whl
import spacy
nlp = spacy.load("en_udv25_englishewt_trf")

[danieldk]

We are overly strict in enforcing support for mixed-precision in the gradient scaler. This PR should fix this issue: explosion/thinc#624.

You could also try temporarily downgrading to Thinc 8.0.14 as a workaround:

pip install thinc==8.0.14