Converting custom NER.tsv to NER.spacy format

[shina181]

I have performed NER on scientific literature using some custom labels like "Operation", "target", "Amount", etc. I tried using the following command to convert my file into a corresponding json file to be further converted into a binary spacy file:

python -m spacy convert NER.tsv ./ -t json -n 1

But it is not working on my file. I have 2 columns in my file viz.tokens in the 1st column and their labels in the 2nd. Is it because my labelled tags are not in the IOB/BILOU format?

In that case how should I go about converting my file into a binary spacy file?
Should I write a code to convert the file into a json file first and then convert it into a spacy file or is there a way to directly convert my NER.tsv file to a NER.spacy file??
Thanks

[ljvmiranda921]

Hi @shina181, it's most likely due to your labels not being in the IOB format. How did you obtain these annotations? If it's through some software, perhaps there's an option to have it output IOB tags? Alternatively, you can rewrite the tags in place with some script (either update the TSV file itself or via Doc.set_ents)

[shina181]

I am thinking of writing a script to convert the tsv file to corresponding json file, which I then think can be converted into a spacy file. Was just wondering if there was an easier way

[ljvmiranda921]

Are you suggesting to replace say tag "Operation" with "B-Operation" in all places? I am not sure if that will be the correct approach or not.

For the tags, you need to convert the tag of the first token to B-Operation and the later contiguous ones as I-Operation. I don't think converting it to an intermediary format like JSON will make it easier since you can use the TSV directly (you just need to format the labels right). 🤔

I'm also not sure if you've seen or used these brat conversion scripts, they might already be doing the steps I mentioned above.

[shina181]

Why I thought it may work is because if you observe the entities in the final json file they are stripped of the IOB in the tag:

And in the final task to be performed with these input files, I plan to extract the entities and their associated span.

[shina181]

Also I tried using not the scripts that you have mentioned in the last reply, but this one to convert into CoNLL format. It just converts the format without inserting the BIO ahead of the tag.
But I can see the script you have shared does that too. I will try that once too. Thanks!

[shina181]

For the tags, you need to convert the tag of the first token to B-Operation and the later contiguous ones as I-Operation.

This worked !
The brat conversion scripts did not give the desired output. So I wrote a small lil script to convert the file. And converting the file in IOB format did the magic. Thanks for the help!

Cheers,
Sheena :)