do we include non entities in training data for custom spaCy NER model? What does the structure of the .spacy training data file look like?

[fariiaakh]

Hi Everyone,
I am fairly new to the spaCy community and in training customs models in general so I need some help please.
I am trying to figure out how to train a custom spaCy model for NER and I am using this dataset from kaggle for a test run:
https://www.kaggle.com/code/lenguyendinhkhanh/named-entity-recognition/data

The words which are entities have special tokens corresponding to them and the words which are not entities have a "O" token corresponding to them.

My question is, when I am creating the training data in the spacy format, Should I include both entity and non entity tags in it to give the model an 'better' understanding of what an entity vs a non entity looks like? Or should I only include the labels which actually point to an entity.

Please note that I am not talking about fine tuning a pretrained NER model. I am talking about creating a fully custom model for a specialized task. I am just using a toy dataset from Kaggle to understand what to look out for when I start training my actual model.

this is what the toy dataset looks like:

[thomashacker]

Hello,
you can use the spacy convert command to convert IOB files to the .spacy training format.
Here are some example files which you can use as a starting point for your data.

[rmitsch]

If you want to look at the contents of your .spacy file, I recommend using a DocBin however:

import spacy
from spacy.tokens import DocBin

nlp = spacy.blank("en") 
doc_bin = DocBin()
doc_bin.from_disk("nermini.spacy")
for doc in doc_bin.get_docs(nlp.vocab):
    print(doc)

[fariiaakh]

hey Raphael,

thanks alot for your response! Is there anyways to see the ner tags in the .spacy file, like how i can see them in the json format? like for example:

this is the original iob file:

this is how it looks in the json format from spacy v2:

and when i run the code you provided, i see the words arranged into sentences without any of the ner tags:

is there a way to see the ner tags in the .spacy file, the way i can see them in the json file?
Basically i want to know what is the format of the data that is contained in the .spacy file?

[rmitsch]

You can access the NER tags via the label_ property - see also the spaCy docs on Named Entity Recognition. For more information on the .spacy file I recommend the docs on the Binary Training Format and DocBin.

[fariiaakh]

thank you very much Raphael and Edward!

[fariiaakh]

i also found the spacy course! i think the tile for this should be higher on the main website😅
https://course.spacy.io/