spacy train without GPU: CURAND_STATUS_INITIALIZATION_FAILED

[kormilitzin]

How to reproduce the behaviour

I'm trying to use spacy train without Transformers. When I use GPU it quickly runs out-of-memory (I have RTX 2080Ti with 12GB). To train in CPU, I put the --gpu-id=-1, expecting it to not to use GPUs, but got this error:

(med7_v3__venv) kormilitzin@doraemon7:/mnt/sdf/Andrey/projects/med7_v3$ python -m spacy train ./ner_config/config_cpu.cfg --output ./output_cpu --paths.train ./data/spacy_format/train.spacy --paths.dev ./data/spacy_format/dev.spacy --gpu-id=-1
ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

=========================== Initializing pipeline ===========================
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/spacy/cli/train.py", line 56, in train_cli
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/spacy/training/initialize.py", line 34, in init_nlp
    fix_random_seed(config["training"]["seed"])
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/thinc/util.py", line 76, in fix_random_seed
    cupy.random.seed(seed)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/cupy/random/_generator.py", line 1240, in seed
    get_random_state().seed(seed)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/cupy/random/_generator.py", line 1272, in get_random_state
    rs = RandomState(seed)
  File "/mnt/sdf/Andrey/virtual_envs/med7_v3__venv/lib/python3.9/site-packages/cupy/random/_generator.py", line 57, in __init__
    self._generator = curand.createGenerator(method)
  File "cupy_backends/cuda/libs/curand.pyx", line 93, in cupy_backends.cuda.libs.curand.createGenerator
  File "cupy_backends/cuda/libs/curand.pyx", line 97, in cupy_backends.cuda.libs.curand.createGenerator
  File "cupy_backends/cuda/libs/curand.pyx", line 86, in cupy_backends.cuda.libs.curand.check_status
cupy_backends.cuda.libs.curand.CURANDError: CURAND_STATUS_INITIALIZATION_FAILED

Any ideas please?

my config file:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 2000
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = "en_core_web_lg"
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Info about spaCy

• spaCy version: 3.0.6
• Platform: Linux-5.4.0-72-generic-x86_64-with-glibc2.27
• Python version: 3.9.5
• Pipelines: en_core_web_trf (3.0.0), en_core_web_sm (3.0.0), en_core_web_lg (3.0.0)

[polm]

To train without GPU, just omit the --gpu-id argument - --gpu-id=-1 is not supported.

[kormilitzin]

@adrianeboyd @polm 's reply didn't address the issue.

[polm]

Ah, sorry, you can use -1 for CPU, I'd never done that before since it's the default.

I'll look at what's causing this. It's strange that you're loading cupy even though you're not using the GPU though.

[polm]

It looks like Thinc sets the GPU random seed if your GPU is usable, even if you're not training with GPU (see here). It's not clear to me why that would cause an error unless there was a problem with your cuda install.

Does this error happen if you start training with GPU? Could you tell us your cupy and CUDA versions?

[kormilitzin]

Thanks @polm.

My CUDA:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

I installed spaCy with CUDA enabled:

pip install -U spacy[cuda110]

It all worked well and I was able to train NER models with Transformers on GPUs (yesterday). My (v rough guess) is that GPU is enable automatically for spacy training, even though I pass GPU-flag -1 because I installed with spaCy with GPU support. But I may be wrong.

[polm]

Thanks for the details.

GPU should not used for training unless explicity specified in spaCy. You can even see this in your output:

ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

However, the seed initialization code I linked to does call GPU code if possible, even when training with CPU. This might be triggering some kind of weird state.

A simple workaround would be to uninstall cupy - if spaCy can't load the GPU libraries it won't make the call that's causing this error.

One thing I notice is that you have Cuda 10.2 but you installed spacy[cuda110]. Could you start with a clean environment and install spacy[cuda102] instead? The Cuda version mismatch might be your issue.