Train data vs dev data for spacy #8395
-
|
How do spacy model training uses "train" data and "dev" data? In the config.cfg file I can specify the two params |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
|
When you train with training and dev data, the training data is actually used for training (to update your model), while the dev data is used for evaluation that occurs every however-many steps. The dev data is referred to as "evaluation data" at the start of the training docs. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @polm! So, just to be sure I understood correctly, if I don't specify the dev data spacy will use the train data also for evaluation? |
Beta Was this translation helpful? Give feedback.
-
|
You could easily try this out: if you run You could specify the same training set also for the "dev" portion, and then the ML algorithm will be trained AND evaluated on the same corpus. Do note that this is not an ideal scenario though, because you won't know whether or not your algorithm is overfitting, and you won't have a realistic view on the performance of your model. |
Beta Was this translation helpful? Give feedback.
-
|
Awesome! Thanks a lot! |
Beta Was this translation helpful? Give feedback.
When you train with training and dev data, the training data is actually used for training (to update your model), while the dev data is used for evaluation that occurs every however-many steps.
The dev data is referred to as "evaluation data" at the start of the training docs.