Span finder CUDA out of memory

[monWork]

Hi I am trying to use Spanfinder but i keep getting into cuda out of memory error.

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'span_finder', 'spancat', 'doc_cleaner']
ℹ Set annotations on update for: ['span_finder']
ℹ Initial learn rate: 0.0
E    #       LOSS TRANS...  LOSS SPAN_...  LOSS SPANCAT  SPAN_FINDE...  SPAN_FINDE...  SPAN_FINDE...  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE
---  ------  -------------  -------------  ------------  -------------  -------------  -------------  ----------  ----------  ----------  ------
  0       0       37570.11          24.69        857.84           0.00           0.00           0.00        0.28        0.15        3.27    0.00
  0     100      239592.97        1258.19       6011.99           0.00           0.00           0.00        0.00        0.00        0.00    0.00
  0     200          20.31         416.32        353.05           0.00           0.00           0.00       19.71       93.10       11.02    0.10
  0     300          34.33         443.65        382.60           0.00           0.00           0.00       34.08       80.30       21.63    0.17
Epoch 1:  11%|████████████████████████▌                                                                                                                                                                                                      | 11/100 [00:02<00:15,  5.84it/s]⚠ Aborting and saving the final best model. Encountered exception:
RuntimeError('CUDA out of memory. Tried to allocate 1.61 GiB (GPU 0; 16.00 GiB
total capacity; 13.84 GiB already allocated; 289.00 MiB free; 13.90 GiB reserved
in total by PyTorch) If reserved memory is >> allocated memory try setting
max_split_size_mb to avoid fragmentation.  See documentation for Memory
Management and PYTORCH_CUDA_ALLOC_CONF')
Traceback (most recent call last):
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\cli\_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\typer\main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\cli\train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\cli\train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\training\loop.py", line 122, in train
    raise e
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\training\loop.py", line 105, in train
    for batch, info, is_best_checkpoint in training_step_iterator:
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\training\loop.py", line 203, in train_while_improving
    nlp.update(
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\language.py", line 1164, in update
    proc.update(examples, sgd=None, losses=losses, **component_cfg[name])  # type: ignore
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\spacy\pipeline\spancat.py", line 346, in update
    backprop_scores(d_scores)  # type: ignore
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\layers\chain.py", line 61, in backprop
    dX = callback(dY)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\layers\chain.py", line 61, in backprop
    dX = callback(dY)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\layers\concatenate.py", line 72, in backprop
    gradient = bwd(dY)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\layers\reduce_mean.py", line 26, in backprop
    return Ragged(model.ops.backprop_reduce_mean(dY, lengths), lengths)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\backends\cupy_ops.py", line 261, in backprop_reduce_mean
    return _custom_kernels.backprop_reduce_mean(d_means, lengths)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\backends\_custom_kernels.py", line 636, in backprop_reduce_mean
    out = _alloc((T, O), dtype=d_means.dtype, zeros=False)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\backends\_custom_kernels.py", line 139, in _alloc
    return cupy.empty(shape, dtype)
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\cupy\_creation\basic.py", line 22, in empty
    return cupy.ndarray(shape, dtype, order=order)
  File "cupy\_core\core.pyx", line 171, in cupy._core.core.ndarray.__init__
  File "cupy\cuda\memory.pyx", line 698, in cupy.cuda.memory.alloc
  File "cupy\cuda\memory.pyx", line 1375, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy\cuda\memory.pyx", line 1396, in cupy.cuda.memory.MemoryPool.malloc
  File "cupy\cuda\memory.pyx", line 1076, in cupy.cuda.memory.SingleDeviceMemoryPool.malloc
  File "cupy\cuda\memory.pyx", line 1097, in cupy.cuda.memory.SingleDeviceMemoryPool._malloc
  File "cupy\cuda\memory.pyx", line 1312, in cupy.cuda.memory.SingleDeviceMemoryPool._try_malloc
  File "cupy\cuda\memory.pyx", line 1047, in cupy.cuda.memory.SingleDeviceMemoryPool._alloc
  File "C:\Users\pilanisp\AppData\Local\miniforge3\envs\tas_model_new\lib\site-packages\thinc\backends\_cupy_allocators.py", line 38, in cupy_pytorch_allocator
    torch_tensor = torch.zeros(
RuntimeError: CUDA out of memory. Tried to allocate 1.61 GiB (GPU 0; 16.00 GiB total capacity; 13.84 GiB already allocated; 289.00 MiB free; 13.90 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Here is my config.

train = null
dev = null
vectors = null
init_tok2vec = null

[vars]
spans_key = "sc"

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","span_finder","spancat", "doc_cleaner"]
batch_size = 32
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.doc_cleaner]
factory = "doc_cleaner"
silent = true

[components.doc_cleaner.attrs]
tensor = null
_.trf_data = null

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "roberta-base"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[components.span_finder]
factory = "experimental_span_finder"
threshold = 0.35
predicted_key = "sc"
training_key = ${vars.spans_key}
min_length = 0
max_length = 0

[components.span_finder.scorer]
@scorers = "spacy-experimental.span_finder_scorer.v1"
predicted_key = ${components.span_finder.predicted_key}
training_key = ${vars.spans_key}

[components.span_finder.model]
@architectures = "spacy-experimental.SpanFinder.v1"

[components.span_finder.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO=2

[components.span_finder.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.span_finder.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.spancat]
factory = "spancat"
max_positive = null
spans_key = ${vars.spans_key}
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 32

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.spancat.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[components.spancat.suggester]
@misc = "spacy-experimental.span_finder_suggester.v1"
candidates_key = ${components.span_finder.predicted_key}

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 2000
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 2000
max_epochs = 0
max_steps = 20000
eval_frequency = 100
frozen_components = []
annotating_components = ["span_finder"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

I have tried reducing batch_size(from 128-64-32) and eval_frequency (500-300-100) and doc_cleaner
I have 16gb GPU memory, not sure what wrong i am doing. How can i avoid getting CUDa out of memory.

[thomashacker]

Hello,
from the first look, your config looks fine. I think the issue might be more about the data than the SpanFinder component. How long are the individual documents you're using for training?