Converting Ontonotes ( .gold_conll file format) to DocBins ( .spacy file format)

[emiltj]

Hi!

TLDR; How to convert .gold_conll (as seen in table below) to .spacy?

#begin document (bc/cctv/00/cctv_0001); part 000
bc/cctv/00/cctv_0001   0    6               a    DT      (NP(NP*        -    -   -   Speaker#1       *             *    (ARG1*   -
bc/cctv/00/cctv_0001   0    7         special    JJ            *        -    -   -   Speaker#1       *             *         *   -
bc/cctv/00/cctv_0001   0    8         edition    NN            *)       -    -   -   Speaker#1       *             *         *   -
bc/cctv/00/cctv_0001   0    9              of    IN         (PP*        -    -   -   Speaker#1       *             *         *   -
bc/cctv/00/cctv_0001   0   10          Across   NNP         (NP*        -    -   -   Speaker#1   (ORG*             *         *   -
bc/cctv/00/cctv_0001   0   11           China   NNP      *)))))))       -    -   -   Speaker#1       *)            *)        *)  -

#end document

My plan is to work with the OntoNotes data in spaCy. I have retrieved the OntoNotes dataset, however, I am struggling to convert my data to a format. I see that the format is similar, but not identical, to two of the sample data formats that may be converted using the python -m spacy converter CLI-command, both trying the --converter auto, conllu, and conll (ner-token-per-line.iob and ner-token-per-line-conll2003.iob). Above is a table of what my data looks like, and this is where I have retrieved it from. I have been unable to find the entire dataset anywhere else, and I see that none of the HuggingFace datasets are available with 1) all the data, and 2) in the right format.

[rmitsch]

Hi @emiltj, have you tried using the -c option to specify the conll converter? See the docs here.
E. g.: spacy convert nameofyourfile.conll -c conll ./output/

[emiltj]

Backing up a bit: what are you attempting to do with this data?

I want to load the Ontonotes dataset as spacy Doc objects, so I can work with it in spaCy.
I only want to use the NE tags and the text itself.

After looking into this, the conll converter supports the the conll-2003 format, which is different from the one you're using. This means you'll have to convert this into Doc objects (or some other supported format) yourself.

I also suspected that the format I am using is not the format that the convert expects. However, I don't know how to manually convert this file to a .spacy file.

I see that many of spaCys models (e.g. en_core_web_sm are trained using the Ontonotes dataset. Before this, I have therefore gained access to the dataset using the link provided in spaCy documentation, e.g. under the en_core_model_sm
. However, these files do not look like the conll-2003 format that is documented in the converters documentation, either.

This might not be the correct forum for this, but is there any way of getting the Ontonotes dataset in any sort of spaCy format?

[adrianeboyd]

This is the official release of OntoNotes that is used for the trained pipelines: https://catalog.ldc.upenn.edu/LDC2013T19

Unfortunately our OntoNotes license does not permit us to redistribute any of the full texts or annotations. I don't know the details for sure, but I don't think that the version in that link is allowed to redistribute the texts or release it under a CC BY 4.0 license. The annotation columns were made available for the CoNLL 2012 shared task (https://conll.cemantix.org/2012/data.html), but not the texts.

With the same idea as in that shared task, we have considered developing scripts for users to re-create our internal en_core_web_* training corpus from the original OntoNotes release, but it would be a big effort on our side without a huge benefit, since most users don't have access to OntoNotes.

In any case, just looking at the data format, you want to convert the text column (column 4) and the NE column (column 11) from this to something spacy can use directly or convert. I'd recommend converting to CoNLL-2003 format, which is what spacy convert -c conll supports. For the spacy converter you can skip the middle two columns and just provide the first and last columns.

This annotation:

a       *
b       (PERSON*
c       *)
d       *
e       *

would become:

a       O
b       B-PERSON
c       I-PERSON
d       O
e       O

[emiltj]

Hi @adrianeboyd ,

Thank you very much for your thorough answer. I have also gained access using the legitimate channel, but since these files seemingly weren't in the right format either, I have tried this more untraditional way of accessing the data, since it looked more akin to the conll format.

But just to clarify:
The documentation states that the conll-2003 format should be formatted like this:

Which matches the examples given in the documentation that looks like this:

-DOCSTART- -X- O O

When WRB _ O
Sebastian NNP _ B-PERSON
Thrun NNP _ I-PERSON
started VBD _ O
working VBG _ O

However, the format you suggest matches another example, but does not match the description of conll-2003 from the documentation.

a       O
b       B-PERSON
c       I-PERSON
d       O
e       O

• Q1: Is the latter format (right above this text), also a case of conll-2003? And may it be converted using the convert -c conll, despite it not matching the documentation?
• Q2: You mentioned that "[...] You want to convert the text column (column 4) and the NE column (column 11) from this to something spacy can use directly or convert". and "You can skip the middle two columns and just provide the first and last columns.". Does this mean I can filter to only keep col 4 (as is), and col 11 (converted), and then the convert will work?

Best

[adrianeboyd]

The format quoted above is not exactly CoNLL-2003, but the converter is written to be a little more flexible for similar formats and includes CoNLL-2003 in what it supports. It works as described in the docs. Note that the doc start line in the spacy converter is only that exact string (spaces not tabs, no alternate -X-/O values) and both the doc and sentence breaks are optional. If your data doesn't have doc or sentence breaks, you could use -s or -n to automatically add them.

Please just try it out and double-check by loading your .spacy files with DocBin to check that the annotation and sentence/document boundaries are where you'd expect so you can catch any problems before training.

[emiltj]

Okay,

Thank you very much for the help.

Good day to you!