Using Spacys en_core_web_trf not working

[AnnemarieWittig]

Hi!

I want to train models based on some of spacy's given language models. For that, I do not wish to use in-code solutions as they have been suggested fairly often but I'd rather like to use a solution via the config file.

Now, I have made this work nicely for spacy's en_core_web_sm model using CPU, but cannot seem to make it work with the en_core_web_trf model as spacy does not seem to find my GPU. Starting the training in my code with mainly this snippet (it's part of a bigger system that I start using docker compose):

spacy.prefer_gpu()
try:
    train(config_file, output_dir, overrides={"paths.train": training_docbin_file, "paths.dev": testing_docbin_file})
except:
    raise FailedTrainingException()

returns the following output (I had to save it to a txt file so the format is a little.. off):

#16 22.65 Successfully installed en-core-web-trf-3.4.0
#16 22.66 WARNING: You are using pip version 22.0.4; however, version 22.2.2 is available.
#16 22.66 You should consider upgrading via the '/opt/venv/bin/python -m pip install --upgrade pip' command.
#16 22.76 �[38;5;2m✔ Download and installation successful�[0m
#16 22.76 You can now load the package via spacy.load('en_core_web_trf')
#16 22.76 �[38;5;4mℹ Saving to output directory: /code/app/spacy_model/output�[0m
#16 22.76 �[38;5;4mℹ Using CPU�[0m
#16 22.76 �[1m
#16 22.76 =========================== Initializing pipeline ===========================�[0m
#16 24.94 �[38;5;2m✔ Initialized pipeline�[0m
#16 24.94 �[1m
#16 24.94 ============================= Training pipeline =============================�[0m
#16 24.95 �[38;5;4mℹ Pipeline: ['transformer', 'tagger', 'parser', 'ner']�[0m
#16 24.95 �[38;5;4mℹ Initial learn rate: 0.0�[0m
#16 24.95 E    #       LOSS TRANS...  LOSS TAGGER  LOSS PARSER  LOSS NER  TAG_ACC  DEP_UAS  DEP_LAS  SENTS_F  ENTS_F  ENTS_P  ENTS_R  SCORE 
#16 24.95 ---  ------  -------------  -----------  -----------  --------  -------  -------  -------  -------  ------  ------  ------  ------
#16 25.06 /opt/venv/lib/python3.8/site-packages/torch/amp/autocast_mode.py:198: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
#16 25.06   warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')
#16 34.63   0       0         314.90         0.00         0.00    398.72     0.00     0.00     0.00     0.00    0.00    0.00    0.00    0.00

and just, doesn't do anything else. It does still load but nothing actually happens as I am assuming my CPU does not suffice even though it is fairly good(?).

I can't start it via console because afaik my GPU ID would be 0 and Spacy tells me there is no GPU there (ValueError: No GPU devices detected). On console it also gives some NUMA warnings before that error:

2022-08-30 14:09:56.600666: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-30 14:09:56.618808: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-08-30 14:09:56.618932: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

The other models I used had the pipeline = ["tok2vec","tagger","parser","ner"] so this is my first time using Transformers. In case my config file is wrong / needs adjustments, I'll add that, too:

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = null
dev = null
vectors = null
[system]
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["transformer","tagger","parser","ner"]
batch_size = 128

[components]

[components.transformer]
source = "en_core_web_trf"

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.tagger]
source = "en_core_web_trf"

[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null

[components.tagger.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.tagger.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.parser]
source = "en_core_web_trf"

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = false
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.parser.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.ner]
source = "en_core_web_trf"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 2000
initial_rate = 5e-5

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[initialize]
vectors = ${paths.vectors}

If it's relevant, my requirements.txt for my docker is:

fastapi==0.79.0
pandas==1.4.3
spacy==3.4.1
spacy-alignments==0.8.5
spacy-legacy==3.0.9
spacy-loggers==1.0.3
spacy-transformers==1.1.8
starlette==0.19.1
pydantic==1.8.2
uvicorn==0.17.6
uvloop==0.16.0
python-dotenv==0.20.0
python-multipart==0.0.5

Now I assume I might need to install some spacy-cuda package but I'm unsure if that's the error and what to install exactly. I've done my research but it seems like every page states something else and I'd rather find a solution here (where it is trustworthy) than somewhere else.

[polm]

It looks like the issue is that you don't have cupy or torch (or possibly other GPU requirments) installed.

You can check the version of CUDA you have using nvcc --version. After you determine that you can install a compatible version of Torch by following directions on their site, and you can install everything needed for spaCy by selecting the GPU option in the quickstart widget. Let me know if that doesn't work.

[AnnemarieWittig]

Well. I really should think of reinstalling things more often. I installed your click version and it runs now, that will definitely be a lesson for future situations, too (rather new to Linux so I am still hesitant). I will try sometime else if it works with my usual locales (all UTF-8). The kernel error still persists though.

python3.8 -m spacy train bert_en_config2.cfg --paths.train ./train.spacy --paths.dev ./test.spacy --gpu-id 0                                          1 ✘  31s   RasaChatbot  
2022-09-01 08:20:42.273490: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-01 08:20:42.273805: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-09-01 08:20:42.273894: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:939] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
ℹ No output directory provided
ℹ Using GPU: 0
/home/annemarie/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:146: UserWarning: 
NVIDIA GeForce RTX 3070 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3070 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

=========================== Initializing pipeline ===========================
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/cli/_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "/home/annemarie/.local/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/annemarie/.local/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/annemarie/.local/lib/python3.8/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/annemarie/.local/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/annemarie/.local/lib/python3.8/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/annemarie/.local/lib/python3.8/site-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/cli/train.py", line 72, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/training/initialize.py", line 41, in init_nlp
    nlp = load_model_from_config(raw_config, auto_fill=True)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 551, in load_model_from_config
    nlp = lang_cls.from_config(
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/language.py", line 1833, in from_config
    source_nlps[model] = util.load_model(
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 429, in load_model
    return load_model_from_package(name, **kwargs)  # type: ignore[arg-type]
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 465, in load_model_from_package
    return cls.load(vocab=vocab, disable=disable, enable=enable, exclude=exclude, config=config)  # type: ignore[attr-defined]
  File "/home/annemarie/.local/lib/python3.8/site-packages/en_core_web_trf/__init__.py", line 10, in load
    return load_model_from_init_py(__file__, **overrides)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 646, in load_model_from_init_py
    return load_model_from_path(
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 511, in load_model_from_path
    return nlp.from_disk(model_path, exclude=exclude, overrides=overrides)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/language.py", line 2115, in from_disk
    util.from_disk(path, deserializers, exclude)  # type: ignore[arg-type]
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 1340, in from_disk
    reader(path / key)
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/language.py", line 2109, in <lambda>
    deserializers[name] = lambda p, proc=proc: proc.from_disk(  # type: ignore[misc]
  File "spacy/pipeline/trainable_pipe.pyx", line 343, in spacy.pipeline.trainable_pipe.TrainablePipe.from_disk
  File "/home/annemarie/.local/lib/python3.8/site-packages/spacy/util.py", line 1340, in from_disk
    reader(path / key)
  File "spacy/pipeline/trainable_pipe.pyx", line 333, in spacy.pipeline.trainable_pipe.TrainablePipe.from_disk.load_model
  File "spacy/pipeline/trainable_pipe.pyx", line 334, in spacy.pipeline.trainable_pipe.TrainablePipe.from_disk.load_model
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/model.py", line 618, in from_bytes
    msg = convert_recursive(is_xp_array, self.ops.asarray, msg)
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/util.py", line 321, in convert_recursive
    value = convert_recursive(is_match, convert_item, value)
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/util.py", line 325, in convert_recursive
    return [convert_recursive(is_match, convert_item, item) for item in obj]
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/util.py", line 325, in <listcomp>
    return [convert_recursive(is_match, convert_item, item) for item in obj]
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/util.py", line 321, in convert_recursive
    value = convert_recursive(is_match, convert_item, value)
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/util.py", line 313, in convert_recursive
    return convert_item(obj)
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/backends/cupy_ops.py", line 78, in asarray
    array = self.xp.array(data)
  File "/home/annemarie/.local/lib/python3.8/site-packages/cupy/_creation/from_data.py", line 46, in array
    return _core.array(obj, dtype, copy, order, subok, ndmin)
  File "cupy/_core/core.pyx", line 2266, in cupy._core.core.array
  File "cupy/_core/core.pyx", line 2290, in cupy._core.core.array
  File "cupy/_core/core.pyx", line 2422, in cupy._core.core._array_default
  File "cupy/_core/core.pyx", line 171, in cupy._core.core.ndarray.__init__
  File "cupy/cuda/memory.pyx", line 698, in cupy.cuda.memory.alloc
  File "cupy/cuda/memory.pyx", line 1375, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1396, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1076, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc
  File "cupy/cuda/memory.pyx", line 1097, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
  File "cupy/cuda/memory.pyx", line 1312, in cupy.cuda.memory.SingleDeviceMemoryPool._try_malloc
  File "cupy/cuda/memory.pyx", line 1047, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc
  File "/home/annemarie/.local/lib/python3.8/site-packages/thinc/backends/_cupy_allocators.py", line 38, in cupy_pytorch_allocator
    torch_tensor = torch.zeros(
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

I guess there's still unhappiness with my pytorch?

[polm]

Yes, looks like a pytorch issue.

Looking at this thread, if you pip uninstall torch can you still import torch? That might happen if you installed pytorch via pacman, for example.

Also, are you just installing things with pip as a user? If so you might want to use a virtualenv instead. Here's a simple guide to using a virtualenv:

# if you don't have virtualenv installed, you can run `pip install virtualenv` first
virtualenv env # env is the directory where the environment is saved
source env/bin/activate # this will modify your prompt and start using the virtualenv
pip install whatever # install all your stuff

This will isolate the versions of libraries your using, which may help with version interference.

[AnnemarieWittig]

I haven't installed pytorch via pacman, only using the command i showed in an earlier comment

python3.8 -m pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116

Before uninstalling, running the snippet from that thread

import torch
torch.tensor(1).cuda()

gave me the same/similar error:

/home/annemarie/.local/lib/python3.8/site-packages/torch/cuda/__init__.py:146: UserWarning: 
NVIDIA GeForce RTX 3070 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA GeForce RTX 3070 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))

I uninstalled it anyways for testing using

python3.8 -m pip uninstall torch torchvision torchaudio

and confirmed that it's not around with pip freeze, now it tells me I can't import it because it's not there, so that is as expected / as it should be, I suppose:

Traceback (most recent call last):
  File "teset.py", line 6, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'

And I am using a virtualenv already, so that should not be an issue. I always activate it before I run pip (un)install or run any of the scripts / training calls etc. It's my first time using it though.

Is there another Pytorch version I should install? I figured this one was the way to go with my CUDA version

[polm]

I am running out of ideas but there is one more thing to try. What is the output of this?

import torch
print(torch.version.cuda)
print(torch.cuda.get_arch_list())

If the version is 10.2, try running the last command you used to install torch, but add --update.

If that doesn't work I'm out of ideas, unfortunately. I am also on Arch Linux and have seen this error before, but it was like a year ago, and vanilla venv installs work for me now.

Since this error seems Torch specific at the moment, you can probably get better help on the Torch forums. If you figure that out and still have trouble with spaCy feel free to follow up here though.

[AnnemarieWittig]

I'm afraid the output is correct and as expected.

11.6
['sm_37', 'sm_50', 'sm_60', 'sm_70', 'sm_75', 'sm_80', 'sm_86']

But! I followed your comment and simply created a new virtual environment. Looking around, I found this fella describing issues with the venv and it seems mine has a similar issue that I simply haven't realized before. (I will now create a new venv for each subproject I think to make sure things go better from now on)

With the new venv and only cupy and torch installed the way you previously explained (+ later additions), it seems to finally work with the GPU 🥳

python3.8 -m spacy train bert_en_config2.cfg --paths.train ./train.spacy --paths.dev ./test.spacy --output ./output --gpu-id 0 

ℹ Saving to output directory: output
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-09-01 09:54:02,568] [INFO] Set up nlp object from config
[2022-09-01 09:54:02,575] [INFO] Pipeline: ['transformer', 'tagger', 'parser', 'ner']
[2022-09-01 09:54:02,575] [INFO] Resuming training for: ['ner', 'parser', 'tagger', 'transformer']
[2022-09-01 09:54:02,581] [INFO] Created vocabulary
[2022-09-01 09:54:02,581] [INFO] Finished initializing nlp object
[2022-09-01 09:54:02,581] [INFO] Initialized pipeline components: []
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'tagger', 'parser', 'ner']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS TAGGER  LOSS PARSER  LOSS NER  TAG_ACC  DEP_UAS  DEP_LAS  SENTS_F  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  -----------  -----------  --------  -------  -------  -------  -------  ------  ------  ------  ------
  0       0         298.25         0.00         0.00    391.88     0.00     0.00     0.00     0.00    0.00    0.00    0.00    0.00
 45     200       32415.32         0.00         0.00  58398.71     0.00     0.00     0.00     0.00   23.60   23.56   23.64    0.08
 93     400        1061.56         0.00         0.00  39753.98     0.00     0.00     0.00     0.00   60.79   63.28   58.48    0.20
138     600        1883.56         0.00         0.00  11118.12     0.00     0.00     0.00     0.00   89.59   89.19   90.00    0.30

It's still slow, obviously, but I haven't gotten to see four rows in training since a while so I'm very happy with this already!
I will later make a separate single comment explaining the errors I encountered and the solutions as it might be beneficial for others (I think)

---

Again, I really appreciate the help. I have found a minimal docker gpu example so I will try to apply that and update if anything else doesnt work. Thank you! Updating in here: All examples I can find are more or less deprecated, so I will ask my team and might come back with more question, but I'll start a separate ticket for that then to keep it organised. Again, thank you!