Spancat - help understanding training memory usage

[Onyoursix]

I'm trying to understand what is going on with a strange (or not so strange) memory issue I'm having while training the spancat portion of a model.

I have a training set of 2,000 examples with 4 labels that I annotated in Prodigy. When I train on a CPU I see a major spike in memory usage then it goes down to almost nothing.

When I export using data-to-spacy with no base model and train via GPU I am getting out of memory errors with a 15GB GPU. Using watch -n0.1 nvidia-smi I can see the same memory spike as I do on the CPU. However, if I export the data with a base model like en_core_web_sm the memory usage is lower during GPU training I still see a massive jump in memory usage but it only uses 13GB out of the 15GB available (it seems to stay at 13GB and doesn't drop during training).

I don't understand why the memory usage spikes on CPU and drops during training as well as why do I get higher memory usage when I don't use a base model?

[svlandeg]

The base model influences the type of network architecture that is used for training, so it can certainly influence memory usage. A transformer-based model for instance will have a different memory usage than a small model. So if you're using en_core_web_sm as base model, the config will source components from that and the architecture will be less complex.

To verify - can you share the exact commands you used to train with Prodigy, and to export from Prodigy and then train?