What was the test set for the accuracy scores reported for Dutch spacy pipelines for NER?

[StellaVerkijk]

Hi,
On spaCy's main website, accuracy scores for each pipeline are reported, but it is not always stated what test sets were used. I need to know which test set was used to measure the accuracy of the Dutch models on NER (reported here: https://spacy.io/models/nl).

Does anyone know this?
Thanks in advance.

[svlandeg]

Hi!

In general you can find the corpora used for training & evaluation detailed on that models page. For Dutch, we worked with NLP Town to augment the Dutch UD corpus LassySmall with NER annotations, and that's the corpus it's trained and evaluated on. I think we haven't made that corpus publicly available yet though (its license is CC BY-SA 4.0). But we could look into that.

[StellaVerkijk]

Thanks for the info!

[StellaVerkijk]

Hi, thank you for the answer!
Does that mean the train/develop/test split of LassySmall was maintained (so 6641, 350, and 350 sentences, respectively)?

[adrianeboyd]

It's using the same split as in the UD corpus and the numbers reported on the models page are for the dev set. (So far, we reserve the test sets for future use and don't touch them at all.)

The internal corpus was created before I started working on spacy and unfortunately I'm not entirely sure exactly which version of UD Dutch LassySmall was annotated (we could track down the details but it would take a bit of time), but I would guess that it was probably v2.4.

[StellaVerkijk]

Alright. I think I know enough. Thanks!