Unsupported Type Error when running Relation Extraction Tutorial on GPU

[AashishTiwari]

Hi, I am trying the relation extraction component tutorial via spaCy projects, Ref: Rel Component on a similar BioMedical relation extraction challenge.
I have been able to replicate results from tutorial codebase + data combination on CPU & GPU.

However, when replaced with another relation extraction challenge data, exactly similar to this data, it works only on CPU (with decent results) and seems to run into issue on a GPU. spaCy code is GPU agnostic, so it shouldn't have been a problem, here is the error snippet.

Stack Trace

ℹ Running workflow 'all_gpu'

================================= train_gpu =================================
Running command: /home/ubuntu/anaconda3/envs/bionlp/bin/python -m spacy train configs/rel_trf.cfg --output training --paths.train data/rel_train.spacy --paths.dev data/rel_dev.spacy -c ./scripts/custom_functions.py --gpu-id 0
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-03-18 07:53:44,773] [INFO] Set up nlp object from config
[2021-03-18 07:53:44,786] [INFO] Pipeline: ['transformer', 'relation_extractor']
[2021-03-18 07:53:44,790] [INFO] Created vocabulary
[2021-03-18 07:53:44,790] [INFO] Finished initializing nlp object
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/spacy/cli/train.py", line 56, in train_cli
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/spacy/training/initialize.py", line 71, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/spacy/language.py", line 1246, in initialize
    proc.initialize(get_examples, nlp=self, **p_settings)
  File "/home/ubuntu/workspace/learn_bionlp/rel_component/scripts/rel_pipe.py", line 177, in initialize
    self.model.initialize(X=doc_sample, Y=label_sample)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/layers/chain.py", line 88, in init
    layer.initialize(X=curr_input)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ubuntu/workspace/learn_bionlp/rel_component/scripts/rel_model.py", line 109, in instance_init
    tok2vec.initialize(X)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/model.py", line 296, in initialize
    self.init(self, X=X, Y=Y)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/layers/chain.py", line 90, in init
    curr_input = layer.predict(curr_input)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/model.py", line 312, in predict
    return self._func(self, X, is_train=False)[0]
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/spacy_transformers/layers/trfs2arrays.py", line 28, in forward
    output, get_d_dst = pooling(dst, is_train)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/model.py", line 288, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/layers/reduce_mean.py", line 18, in forward
    Y = model.ops.reduce_mean(cast(Floats2d, Xr.data), Xr.lengths)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/backends/cupy_ops.py", line 113, in reduce_mean
    return _custom_kernels.reduce_mean(X, lengths)
  File "/home/ubuntu/anaconda3/envs/bionlp/lib/python3.6/site-packages/thinc/backends/_custom_kernels.py", line 102, in reduce_mean
    reduce_sum_kernel((num_blocks,), (threads_per_block,), (out, X, lengths, B, T, O))
  File "cupy/core/raw.pyx", line 82, in cupy.core.raw.RawKernel.__call__
  File "cupy/cuda/function.pyx", line 182, in cupy.cuda.function.Function.__call__
  File "cupy/cuda/function.pyx", line 153, in cupy.cuda.function._launch
  File "cupy/cuda/function.pyx", line 121, in cupy.cuda.function._pointer
TypeError: Unsupported type <class 'numpy.ndarray'>

It looks like simple tok2vec initialization using transformers on GPU has issues, however on CPU works fine.
Environment is a new fresh spaCy installation, with below mentioned versions, also for sake of completion, I tested my NER model training and it worked fine on GPU with good results.

Your Environment

Info about spaCy

• spaCy version: 3.0.5

• Platform: Linux-5.4.0-1039-aws-x86_64-with-debian-buster-sid

• Python version: 3.6.13

• Pipelines: en_core_web_trf (3.0.0)

• Operating System: Ubuntu 18.04

• Environment Information: Fresh Anaconda Env with below spaCy install commands:
  CUDA toolkit version 10.2 (confirmed via nvcc --version)

Installation Steps

pip install -U pip setuptools wheel
pip install -U spacy[cuda102,transformers,lookups]
python -m spacy download en_core_web_trf

Any ideas/pointers as to what components should be looked at? It might also be helpful to know, if this is a cupy / thinc specific issue.
I have tried the latest CUDA 11.0 version as well and error remains the same.

Thanks!

[svlandeg]

I have been able to replicate results from tutorial codebase + data combination on CPU & GPU.
However, when replaced with another relation extraction challenge data, exactly similar to this data, it works only on CPU (with decent results) and seems to run into issue on a GPU.

I'm a little at a loss here, because you mention the tutorial works as-is on both CPU & GPU. When you replaced the data, did you make any other changes to the code base, config file, or project yml?

[AashishTiwari]

Hi @svlandeg, Thanks for looking into this, indeed the behaviour is a bit confusing. There is no change in the code base, the project.yml is updated, since the parse data step is not needed, as I have my own data generation process, paths to the training data are updated likewise.

I am adding few details about the data generation process. My training data is a list of doc objects, since I have entity labels and offsets, I just add the relations to the doc objects and serialize them to binary format (.spacy) as required by the code.

Sample Data Generation Code

spacy.require_gpu()
nlp = spacy.load("en_core_web_trf", disable=["ner", "pos"])
vocab = Vocab()

From the training data, set entities and relations on the doc object:

Doc.set_extension("rel", default={})
doc.ents = entities
doc._.rel = rels

where rels is a dict similar to the tutorial code and entities is list of training entities, created via doc.char_span(start, end, label)
Example:
doc._.rel :
{
(0, 2): {'R1': 1.0, 'R2': 0.0, 'R3': 0.0},
(0, 0): {'R1': 0.0, 'R2': 0.0, 'R3': 0.0},
(2, 0): {'R1': 0.0, 'R2': 0.0, 'R3': 0.0},
(2, 2): {'R1': 0.0, 'R2': 0.0, 'R3': 0.0}
}

Once the training & test docs are generated, they are serialized for use in the pipeline.

docbin = DocBin(docs=training_data, store_user_data=True)
docbin.to_disk("rel_train.spacy")

The results on CPU are decent, below is the full run with custom data on CPU:

CPU Pipeline Output

ℹ Running workflow 'all'

================================= train_cpu =================================
Running command: /home/ubuntu/anaconda3/envs/bionlp/bin/python -m spacy train configs/rel_tok2vec.cfg --output training --paths.train data/rel_train.spacy --paths.dev data/rel_dev.spacy -c ./scripts/custom_functions.py
ℹ Using CPU
ℹ To switch to GPU 0, use the option: --gpu-id 0

=========================== Initializing pipeline ===========================
[2021-03-24 18:45:20,823] [INFO] Set up nlp object from config
[2021-03-24 18:45:20,837] [INFO] Pipeline: ['tok2vec', 'relation_extractor']
[2021-03-24 18:45:20,842] [INFO] Created vocabulary
[2021-03-24 18:45:20,842] [INFO] Finished initializing nlp object
[2021-03-24 18:45:39,074] [INFO] Initialized pipeline components: ['tok2vec', 'relation_extractor']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'relation_extractor']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS RELAT...  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  ------------  -------------  -----------  -----------  -----------  ------
  0       0          0.27           2.83         3.31        36.67         6.06    0.06
  0     500          3.30         140.93        69.33        29.63        41.51    0.42
  1    1000          4.40         103.19        66.67        42.95        52.24    0.52
  3    1500          7.14          74.30        65.22        51.43        57.51    0.58
  6    2000          9.88          51.87        66.98        54.96        60.38    0.60
 12    2500         11.16          35.65        61.43        58.59        59.98    0.60
 17    3000          9.77          26.16        62.89        58.04        60.37    0.60
 23    3500          8.71          20.95        62.59        59.14        60.82    0.61
 28    4000          8.03          17.52        65.55        55.95        60.37    0.60
 34    4500          7.03          15.37        67.01        56.83        61.50    0.62
 40    5000          7.01          14.33        68.32        54.63        60.71    0.61
 45    5500          6.55          13.32        65.92        58.37        61.92    0.62
 51    6000          6.47          12.22        67.65        58.04        62.48    0.62
 57    6500          5.84          11.72        64.58        59.03        61.68    0.62
 62    7000          6.26          10.98        67.41        56.72        61.60    0.62
 68    7500          5.86          10.61        63.22        59.25        61.17    0.61
 73    8000          5.92          10.45        64.70        57.93        61.13    0.61
 79    8500          5.75           9.93        66.88        57.38        61.77    0.62
 85    9000          5.83           9.86        66.67        59.69        62.99    0.63
 90    9500          5.82           9.54        63.62        59.69        61.59    0.62
 96   10000          5.49           9.42        63.54        57.38        60.30    0.60
✔ Saved pipeline to output directory
training/model-last

Threshold @ 0.5 gives about 60% f1-score

The results (although decent) could definitely benefit from using transformers.
From the error logs, it looks like I am not able to initialize the relation extraction model, and error seems to be within Cupy reduce_mean operation, which doesn't like the data format. I have tried couple of fresh installations, with different versions. My initial thought was issue with custom data generation, but since it works on CPU, it might be an issue with sending data to GPU.

Thanks!

[AashishTiwari]

Hi @svlandeg, Thanks for checking on this earlier, any thoughts/suggestions would be really helpful here. Should I change the data generation strategy, since I don't have prodigy style annotations? Does the current data generation process makes sense?
Would it make sense to change the configuration and try other readers?
Thanks!

[AashishTiwari]

Just for posterity, leaving the patch I applied to solve this problem;

File: thinc/backends/_custom_kernels.py View on GitHub
Under the following method:

def reduce_mean(X, lengths, out=None, threads_per_block=128, num_blocks=128):

I forced conversion to cupy array, so that GPU device is used:

 X = cupy.array(X)

This will convert the input array into cupy.core.ndarray type which is required by Cupy to compute the pooling (thinc offloads the calculations to backend cupy kernels), otherwise the type will be <class 'numpy.ndarray'> which is not supported.

cc: @svlandeg, Hope this was one-off error, but if further investigation is required, and similar errors are reported in future, a bug can be reported on the main issue log and can be investigated in detail.

Thanks,
--Aashish

[AashishTiwari]

This issue might be fixed with PR #7626
Also, this might be part of upcoming v3.0.6 release, which should fix similar errors.
cc: @svlandeg

[svlandeg]

Hi @AashishTiwari, thanks for reporting back, and sorry that I haven't been replying as I was looking into this. Great to hear that the recent PR fixes things on your end though!