IOB2 format for .spacy data conversion

[m-nlp-q]

Hello,

I want to convert my data to BIO2 format like the penultimate data sample presented in here. In the converters description says that sentences are separated by blank lines. Just to make sure, the file given to the converter should have a format like this:

Stop O
the O
Bombings O
. O

They O
marched O
from O
the O

"They" starts a new sentence. I am asking because I want to exclude possible bug reasons.

[polm]

Yes, using blank lines to separate sentences is the format described as conll 2003, or the ner option for the converter. The example file from the repo you link to is here.

There's slightly more detail about the optional -DOCSTART- tag, which can be used to separate documents, if you look at the source here:

https://github.com/explosion/spaCy/blob/master/spacy/training/converters/conll_ner_to_docs.py#L20

I guess by the "penultimate" example you mean ner-token-per-line.iob? That doesn't have blank lines and so doesn't seem to distinguish sentences.

[m-nlp-q]

You are right @polm, I was talking about ner-token-per-line.iob. I used the converter with my dataset having a blank line between sentences and it worked. I have errors when I run spacy train so this might be a reason.

b.t.w. my dataset initially is in biluo format. Can I use the converter with this?

[polm]

So the terminology around them is used very inconsistently, but BILOU and IOB are not really data formats, they're conventions for how you use named entity tags. Whether spaCy can use them or not depends more on the actual data formats, like whether it's one sentence or token per line, and how the fields are structured.

As far as the actual tags in BILOU, spaCy supports all of them, so assuming the data format is otherwise supported they will just work. (The docs do not make this clear, so that's something that can be improved.)

[polm]

Note that if you want to check the output, you can convert to the JSON output format and check that everything got picked up. That would look like this:

spacy convert --converter ner -t json input.iob out/

And then out/input.json would have your entity data.

[m-nlp-q]

Thank you @polm you have been very helpful. I ran

spacy convert --converter ner -t json input.iob out/

with my BIO2 dataset having and not having blank lines, and creates the same .json file, so I guess that it also creates the same .spacy file.

I also noticed that in .json file the tags have turned into bilou, even though my dataset had BIO2 tags.

[m-nlp-q]

Thank you @polm, you have been really helpful.