Training, dev and test data

[NixBiks]

It is pretty common to have three datasets. Some for training, some for dev for parameter tuning and finally a set for testing/evaluation.

I'm a little confused about the best practice split for spacy. In docs you usually have train.spacy and dev.spacy - I assume those are intended as I just mentioned? But I don't really see any docs on how to integrate test data (to be able to see how new models perform on the same unseen data as previous models).

Currently I have split some prodigy datasets into training and evaluation. Then I use prodigy train on the training data with 20% eval split e.g.. Is that reasonable? When I use prodigy train then my config is auto generated as a good starting point, but I'd like to integrate the test data as well. I.e. generate a classification report based on the test data.

Any pointers would be much appreciated

[thomashacker]

Hello,
you can use the spacy evaluate command for your test dataset. Deciding a "good" split percentage is very empirical and depends on the quantity of your data, but 20% is probably fine.