Cuda Error While Training NER+RELATIONAL model

[karndeepsingh]

Hi,
I am trying to train NER+ RELATIONAL model and I am facing CUDA error. And If use different training.spacy file it doesn't show any CUDA Error. Please help me to understand how to resolve this kind of issue and what parameters I need to play with in relation_trf.config file.

[karndeepsingh]

How do I train 2000 labelled documents using Transformer for NER + RELEATION problem. I am facing cuda error as shown above. What parameters setting should change to get it running on my GPU? I have changed the Batchsize from 32 to 2 but still it is showing cuda error.

Any help would be greatful.

[adrianeboyd]

In general, please paste output as text in a code block instead of as a screenshot. It's easier to read and search for when related errors come up.

There are two batch sizes to adjust, see: #9451 (comment)

[karndeepsingh]

Please guide me to fit my 5K documents or sentences for NER + RELATIONS training into 12GB of GPU. Below are the configurations I am using for Transformers. Let me know what parameters I could play with to get my all data fit into GPU ram and start training. Thanks.

[paths]
train = null
dev = null
raw = null
init_tok2vec = null

[system]
seed = 342
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}
batch_size = 4

[components]

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "roberta-base"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 64
stride = 48

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor]
factory = "relation_extractor"
threshold = 0.5

[components.relation_extractor.model]
@architectures = "rel_model.v1"

[components.relation_extractor.model.create_instance_tensor]
@architectures = "rel_instance_tensor.v1"

[components.relation_extractor.model.create_instance_tensor.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.relation_extractor.model.create_instance_tensor.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.pooling]
@layers = "reduce_mean.v1"

[components.relation_extractor.model.create_instance_tensor.get_instances]
@misc = "rel_instance_generator.v1"
max_length = 10

[components.relation_extractor.model.classification_layer]
@architectures = "rel_classification_layer.v1"
nI = null
nO = null

[initialize]

[initialize.components]

[corpora]

[corpora.dev]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.dev}

[corpora.train]
@readers = "Gold_ents_Corpus.v1"
file = ${paths.train}

[training]
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600000
max_epochs = 0
max_steps = 2000
eval_frequency = 10
frozen_components = []
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
before_to_disk = null
logger = {"@Loggers":"spacy.ConsoleLogger.v1"}

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.score_weights]
rel_micro_p = 0.0
rel_micro_r = 0.0
rel_micro_f = 1.0

[karndeepsingh]

In general, please paste output as text in a code block instead of as a screenshot. It's easier to read and search for when related errors come up.
There are two batch sizes to adjust, see: #9451 (comment)

Here is the error I am getting when I train it on 3500 sentences. Please help to understand how I can train it with large dataset. I have trained it on 400 sentences, it started training. But as soon as I increase the data it throws CUDA ERROR. Is it a bug? I saw many people are reporting the similar issue while training NER TRANSFORMER model on LARGE data.

================================= train_gpu =================================
Running command: /home/gsatis/anaconda3/envs/spacy/bin/python -m spacy train configs/rel_trf.cfg --output training --paths.train data/relations_test_ida.spacy --paths.dev data/relations_test_ida.spacy -c ./scripts/custom_functions.py --gpu-id 0
ℹ Saving to output directory: training
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2021-11-26 15:20:31,101] [INFO] Set up nlp object from config
[2021-11-26 15:20:31,107] [INFO] Pipeline: ['transformer', 'ner']
[2021-11-26 15:20:31,109] [INFO] Created vocabulary
[2021-11-26 15:20:31,110] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[2021-11-26 15:20:58,947] [INFO] Initialized pipeline components: ['transformer', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  REL_MICRO_P  REL_MICRO_R  REL_MICRO_F  SCORE 
---  ------  -------------  --------  ------  ------  ------  -----------  -----------  -----------  ------
⚠ Aborting and saving the final best model. Encountered exception:
RuntimeError('CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 10.91 GiB
total capacity; 8.95 GiB already allocated; 38.31 MiB free; 9.88 GiB reserved in
total by PyTorch) If reserved memory is >> allocated memory try setting
max_split_size_mb to avoid fragmentation.  See documentation for Memory
Management and PYTORCH_CUDA_ALLOC_CONF')
Traceback (most recent call last):
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/cli/_util.py", line 69, in setup_cli
    command(prog_name=COMMAND)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/cli/train.py", line 63, in train_cli
    train(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/training/loop.py", line 122, in train
    raise e
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/training/loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/training/loop.py", line 226, in train_while_improving
    score, other_scores = evaluate()
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/training/loop.py", line 281, in evaluate
    scores = nlp.evaluate(dev_corpus(nlp))
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/language.py", line 1377, in evaluate
    for doc, eg in zip(
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy/util.py", line 1490, in _pipe
    yield from proc.pipe(docs, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy_transformers/pipeline_component.py", line 204, in pipe
    self.set_annotations(subbatch, self.predict(subbatch))
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy_transformers/pipeline_component.py", line 220, in predict
    activations = self.model.predict(docs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/thinc/model.py", line 315, in predict
    return self._func(self, X, is_train=False)[0]
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/spacy_transformers/layers/transformer_model.py", line 145, in forward
    tensors, bp_tensors = transformer(wordpieces, is_train)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/thinc/model.py", line 291, in __call__
    return self._func(self, X, is_train=is_train)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/thinc/layers/pytorchwrapper.py", line 134, in forward
    Ytorch, torch_backprop = model.shims[0](Xtorch, is_train)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/thinc/shims/pytorch.py", line 56, in __call__
    return self.predict(inputs), lambda a: ...
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/thinc/shims/pytorch.py", line 66, in predict
    outputs = self._model(*inputs.args, **inputs.kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 844, in forward
    encoder_outputs = self.encoder(
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 522, in forward
    layer_outputs = layer_module(
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 449, in forward
    layer_output = apply_chunking_to_forward(
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/transformers/modeling_utils.py", line 2186, in apply_chunking_to_forward
    return forward_fn(*input_tensors)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 461, in feed_forward_chunk
    intermediate_output = self.intermediate(attention_output)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 363, in forward
    hidden_states = self.intermediate_act_fn(hidden_states)
  File "/home/gsatis/anaconda3/envs/spacy/lib/python3.8/site-packages/torch/nn/functional.py", line 1556, in gelu
    return torch._C._nn.gelu(input)
RuntimeError: CUDA out of memory. Tried to allocate 40.00 MiB (GPU 0; 10.91 GiB total capacity; 8.95 GiB already allocated; 38.31 MiB free; 9.88 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[svlandeg]

Have you tried changing the batch size parameter(s) as Adriane suggested?

[karndeepsingh]

@svlandeg I have tried everything. Found one bug in the spacy code. When we load the dev dataset of more than 400 sentences, this whole dev dataset is kept in GPU memory for calculating the score and it is there until the run is complete instead of taking whole dev dataset in GPU memory for calculating the score it could have been done in batches. Here is the issue mentioned properly #9602 and discussed in detail but no update has been done on this. Please look into this problem of training on large dataset with dev dataset more than 400 sentences.

Thanks