Ner + GPU + Accuracy = Out of memory (Nvidia 3090)

[MagiCsito]

Hello,

I am trying to train a custom NER model with my own anotations, I have like 13000 texts annoted. Using https://spacy.io/usage/training I created the .cfg for the training but when I try to train it, I never can do it because it always go Out Of Memory.

I decided to try it in colab and it works but it required 30 gb of VRAM to train the model (using the default model name = "roberta-base").

In the past I asked about this error and I receive the recomendation of reduce the [nlp] batch_size but it doesn't work. Someone could help me?

The .spacy train file size is 40mb, the config that i am using is the following:

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner","tagger","morphologizer"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.morphologizer]
factory = "morphologizer"
extend = false
overwrite = true
scorer = {"@scorers":"spacy.morphologizer_scorer.v1"}

[components.morphologizer.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[components.morphologizer.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.tagger]
factory = "tagger"
neg_prefix = "!"
overwrite = false
scorer = {"@scorers":"spacy.tagger_scorer.v1"}

[components.tagger.model]
@architectures = "spacy.Tagger.v2"
nO = null
normalize = false

[components.tagger.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null
before_update = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_f = 0.33
ents_p = 0.0
ents_r = 0.0
ents_per_type = null
tag_acc = 0.33
pos_acc = 0.17
morph_acc = 0.17
morph_per_feat = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Commands used:

• python -m spacy init fill-config base_config.cfg config.cfg
• python -m spacy train config.cfg --output ./NER_v12 --paths.train ./train.spacy --paths.dev ./train.spacy -g 0

I trained the model using tok2vec but it doesn't work properly (It doesnt "extract" the entities that are not in the original list), I am a little bit desesperated about this (months working on it) so any help will be appreciated!

Thank you :)

[adrianeboyd]

Please see this related discussion: #8600 (comment)

[MagiCsito]

Hello @adrianeboyd !
Yesterday I even tried with nlp.batch_size = 1 and training.batcher.size = 1, however it fails too. Then I tried to use distilbert-base-uncase but it failed too so I suspect that the problem could be the anotated text (I am parsing urls, extracting the text and anotating it directly).

What I am doing right now is, from the list of 13679 urls that I have, I am extracting the text from each url, spliting the text in sentenceS using nltk.tokenize and if len(each_sentence) > 60, anotating it and adding it to the doc.

Maybe the problem could be related that I was anotating using en_core_web_lg instead of trf, I changed the code and anotating everything again with this fixes.

I will return with the result! Thank you so much for your help.

[MagiCsito]

Hello again @adrianeboyd !

First of all I would like to say thank you again for the help :) With your recommendations now everything works and I can train the model using Roberta !

Just a summary for other people.

I went OOM all the time training NER with any AI model and the config was this:

• Annotations done with "en_core_web_lg"
• The annotated text was so long (full text extracted from news websites)
• With this annotated text, it didn't matter what Batcher size I used (training size or nlp batcher size), it always went OOM.

Solution:

• Annotate the text using "en_core_web_trf"
• Split the text extracted from news urls in sentences and annotate it instead the full text
• Experiment the with batcher size

Now I have one last question, when I will use the model to extract entities, should I use it over the full text of news? Or should I divide the text in sentences and then use it over the sentences?

Thank you again and I hope that this post help other people like me!

[adrianeboyd]

Now I have one last question, when I will use the model to extract entities, should I use it over the full text of news? Or should I divide the text in sentences and then use it over the sentences?

This is a good question! Because the statistical models mostly rely on local context while predicting (usually a window of ~1-2 surrounding sentences), the amount of surrounding text beyond that doesn't matter too much (although you can still run out of memory with very long texts).

So it's typically fine to train on shorter texts due to memory constraints but then predict on longer texts. One important exception here is the dependency parser, which by default in spacy also predicts sentence boundaries. If you only train it on single sentences, it won't ever learn where to annotate sentence boundaries, so paragraphs or larger are needed for training.

There will probably be slight differences in the predictions for sentences in isolation vs. sentences in context, so if you're concerned you should just test this out with your data. Remember to use nlp.pipe to speed things up when predicting over lots of short texts.

[MagiCsito]

Hello again!
Oki I will do some tests and share the results here before close the topic.

Thank you again!

[MagiCsito]

Hello,

Now I can train NER using bigger AI models and it works like a CHARM! I did the anotations over 25000 texts extracted from pdfs and news URLs splitted in sentences (I dont know how much sentences). It was training the model during 34 hours [training.batcher] size = 1500, [nlp] batcher_size = 128.

The results after training are really good!

@adrianeboyd Thank you again for the support :)