What is the "corpora.dev limit" behavior?

[info2000]

My .dev file had 15000 items, and reach cuda memory limit, so, if I set the corpora.dev limit to 6000, what 6000 items will try to test on each evaluation?
a)always the same 6000 items in all evaluations
b)will randomize the 6000 items on each evaluation?
other?

Thanks

[polm]

The items will be the same on each evaluation.

The corpus is reloaded on each call to evaluate, which triggers a new shuffling, but the call to evaluate also fixes the random seed first so the shuffle is the same each time.

Randomizing the items on each evaluation wouldn't make sense.

[polm]

Actually, the above explanation is wrong, but the point is the same - you don't need to worry about this.

Shuffling is not something you turn on at the corpus level (for spacy.Corpus.v1 anyway), rather, for non-streaming corpora, examples are automatically shuffled during training for the training data only.

That said, it's a little unusual to use limit on the dev set. Dev sets are usually small enough to fit in memory. It may make sense for you to preprocess your dev set to make it smaller.

[info2000]

Thanks @polm
my issue is that with 23 categories, 400 examples per category (not enough to some of them), reach the 15gb gpu ram, because the corpus size is around 8000 chars per example
and with the limit, some of the categories less represented was omitted

some alternative to let increase the number of examples with 15gb gpu memory?

[polm]

I think you just have too much data for your evaluation set - those documents are quite long.

Words / tokens are a better reference for memory usage than characters. For Transformers the wordpiece unit is actually important, which is a little smaller, but assuming an average of 5 chars per word you have 1600 words per example, so a total of something like 14 million words in your dataset.

I would start by reducing the number of docs per category. I understand that's undesirable but your examples are just too big.

[adrianeboyd]

Another thing to try is to add the doc_cleaner component to your pipeline. This deletes doc._.trf_data from the docs, which removes the transformer tensors that are otherwise kept in memory on the GPU. This is mainly helpful for the dev set, where the docs are all kept in memory at the same time for the evaluation. A related comment: #10664 (reply in thread)