Preventing CUDA Out of Memory #10664
-
|
Hi all, Like a lot of the discussions/issues here, I've been dealing with CUDA OOM errors when fine tuning my NER model. For example: Some details:
In the config file, if I set a max_epochs in [training], then I'm not able to get to a single eval step before running out of memory. If I stream the data in by setting max_epochs to -1 then I can get through ~4 steps (with an eval_frequency of 200) before running OOM. I've tried adjusting a wide variety of settings in the config file, including:
I thought about setting and run train with Finally, I created my train set .spacy files with the following, which takes a Pandas DF's 'spacy' col: Here's my full config file: |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 9 replies
-
You are correct you need to edit your config to add the component, and that pipeline description looks fine. Your custom code file is not correct, you shouldn't be creating a pipeline there - remove the lines with "nlp" on them. You just need to declare the component, which in this case is basically just a function that takes in a Doc, modifies it, and returns the modified Doc. After you fix that you should be able to add the component to your pipeline. If you get an error doing so please share it. |
Beta Was this translation helpful? Give feedback.
-
|
You shouldn't need to write a custom component to remove If you've checked that you're removing And Paul already mentioned this, but one unusually long train or dev doc can also cause problems, in particular with |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, Adriane and Paul! I'll look into using Also, not sure how much this might affect things, but I have 85k unique entity labels in my train set. I added label data files using |
Beta Was this translation helpful? Give feedback.
-
|
Still hitting that error even after adding the Also, my baseline eval set has 4380 docs, and the longest doc has 292 tokens and just 1503 characters. Here's the full error: |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, we've never tested with anywhere near that many labels, and it's probably too many labels to be practical with any spacy components. The typical use cases for NER are for 5-20 labels. For textcat we have other suggestions for cases with a large number of labels, but I don't know of one for NER off the top of my head. |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense! The way my data is organized is that we have ~85k entities with unique IDs. These entities are grouped into 6 categories/types. Our goal is to train two models, one for entity types and the other for the individual entity IDs. (The reason for the entity ID model is that it's important for us to know that "George Washington University", "GWU", and "The George Washington University" all refer to the same entity instead of just identifying each of these mentions as a "College", for example.) We've already trained a model to identify the 6 entity types (using the same training data as for the entity ID model, but just swapping out the labels), and we've also trained an entity ID model (with the 85k labels) on the CPU with tok2vec. I might try just training the entity ID model in Transformers directly to see if that works any differently and then bring it back into spacy afterwards, because those two models (entity type and ID) will be combined with an entity ruler in a pipeline for inference. Thanks again for your help on this. |
Beta Was this translation helpful? Give feedback.
You are correct you need to edit your config to add the component, and that pipeline description looks fine.
Your custom code file is not correct, you shouldn't be creating a pipeline there - remove the lines with "nlp" on them. You just need to declare the component, which in this case is basically just a function that takes in a Doc, modifies it, and returns the modified Doc.
After you fix that you should be able to add the component to yo…