Error while Training NER using transformers

[shrinidhin]

Hi. I have been trying to train my custom NER component with the Transformer embeddings (albert-base-v2) configuration. however, when I am trying to run the training using config files via the terminal on my GPU, it throws a Runtime OOM error:

RuntimeError('CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 2.00 GiB
total capacity; 781.73 MiB already allocated; 0 bytes free; 880.00 MiB reserved
in total by PyTorch) If reserved memory is >> allocated memory try setting
max_split_size_mb to avoid fragmentation.  See documentation for Memory
Management and PYTORCH_CUDA_ALLOC_CONF')

Traceback (most recent call last):
  File "C:\Python\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Python\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\cli\_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1128, in __call__    return self.main(*args, **kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\typer\main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\cli\train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\cli\train.py", line 75, in train    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\training\loop.py", line 122, in train
    raise e
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\training\loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\training\loop.py", line 203, in train_while_improving
    nlp.update(
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\language.py", line 1156, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])  # type: ignore
  File "spacy\pipeline\transition_parser.pyx", line 398, in spacy.pipeline.transition_parser.Parser.update
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\model.py", line 309, in begin_update
    return self._func(self, X, is_train=True)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy\ml\tb_framework.py", line 33, in forward
    step_model = ParserStepModel(
  File "spacy\ml\parser_model.pyx", line 216, in spacy.ml.parser_model.ParserStepModel.__init__
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\model.py", line 291, in __call__    return self._func(self, X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\model.py", line 291, in __call__    return self._func(self, X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\model.py", line 291, in __call__    return self._func(self, X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\layers\chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\model.py", line 291, in __call__    return self._func(self, X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\spacy_transformers\layers\transformer_model.py", line 185, in forward
    model_output, bp_tensors = transformer(wordpieces, is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\model.py", line 291, in __call__    return self._func(self, X, is_train=is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\layers\pytorchwrapper.py", line 134, in forward
    Ytorch, torch_backprop = model.shims[0](Xtorch, is_train)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\shims\pytorch.py", line 54, in __call__
    return self.begin_update(inputs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\thinc\shims\pytorch.py", line 80, in begin_update
    output = self._model(*inputs.args, **inputs.kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\models\albert\modeling_albert.py", line 721, in forward
    encoder_outputs = self.encoder(
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\models\albert\modeling_albert.py", line 467, in forward
    layer_group_output = self.albert_layer_groups[group_idx](
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\models\albert\modeling_albert.py", line 419, in forward
    layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\models\albert\modeling_albert.py", line 389, in forward
    ffn_output = apply_chunking_to_forward(
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\modeling_utils.py", line 2360, in apply_chunking_to_forward
    return forward_fn(*input_tensors)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\models\albert\modeling_albert.py", line 401, in ff_chunk
    ffn_output = self.activation(ffn_output)
  File "C:\Users\Shrinidhi\Desktop\Sentiment Analysis\sentiment\lib\site-packages\transformers\activations.py", line 42, in gelu_new
    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
RuntimeError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 2.00 GiB total capacity; 781.73 MiB already allocated; 0 bytes free; 880.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

The error pops up at the start itself, even before executing for 1 epoch. I found some answers online which spoke about clearing cache via code, but since I am training via config files, how do I resolve this?I even tried decreasing the batch size but it didn't have any effect. Any help would be much appreciated. Thank you!

[thomashacker]

Hello,
If I see it correctly, the machine that you're currently using has only 2GB of memory available for training. This is not suited for training with transformers embeddings. You might get it working by using very small batch sizes and small transformer models, but I wouldn't recommend that.