Split data to perform NER

[mandar-avhad]

Hi, I want to perform NER on a data.
What is the proper way to split the data?
Are there any stratified or other methods to do this?

[ljvmiranda921]

Hi @mandar-avhad , in general, spaCy accepts the standard train / dev / test split during training and evaluation. This means that you have to split your data, preferably in a serialized DocBin format, beforehand. You can check a sample conversion script here.

If you want to do stratified splits, you can implement a custom Corpus that gives you the correct batch during training. You can also check out an example project that does cross-validation (note: it may not be the most efficient solution).

[mandar-avhad]

Yes, I want to split my NER json data into train/dev/test before converting it to spacy format.
Could you share some specific references, please?
I want to know the technique which can be used for NER

[ljvmiranda921]

Hi @mandar-avhad , have you taken a look at the sample project in the initial response already? You can split your NER JSON using Python programmatically. Load the dataset and split the resulting data structure (you can use libraries like json or srsly for that). Once you've partitioned them into train / test / split, then you can convert them into a DocBin object. Use this ner_demo project conversion script as a starting point.