Import data from Doccano json for custom Relation Extraction problem

[daemonkillerr]

Hi

I want to import the Entities and Relations data from doccano which is of the following format:

{
"text": "Google was founded on September 4, 1998, by Larry Page and Sergey Brin.",
"entities": [
{
"id": 0,
"start_offset": 0,
"end_offset": 6,
"label": "ORG"
}
],
"relations": [
{
"id": 0,
"from_id": 0,
"to_id": 1,
"type": "foundedAt"
}
]
}

Please guide on how to convert it to spacy format and use it for custom ner and relation extraction.

Thank you

[svlandeg]

Hi,

In spaCy terminology, you want to store these annotations on a Doc object. You can create a Doc by specifying the words and spaces in your text, ensuring your tokenization will correspond to the gold annotations you've created. In the constructor, you can also set ents, which would look something like this, using the IOB scheme:

    import spacy
    from spacy.tokens import Doc
    nlp = spacy.blank("en")
    words = ["Sarah", "'s", "sister", "flew", "to", "Silicon", "Valley", "via", "London", "."]
    spaces = [False, True, True, True, True, True, True, True, False, False]
    ents = ["B-PERSON", "I-PERSON", "O", "", "O", "B-LOC", "I-LOC", "O", "B-GPE", "O"]
    doc = Doc(nlp.vocab, words=words, spaces=spaces, ents=ents)
    print([(e.text, e.label_) for e in doc.ents])
    # [("Sarah's", 'PERSON'), ('Silicon Valley', 'LOC'), ('London', 'GPE')]

You can add these Doc objects in a [DocBin](https://spacy.io/api/docbin) and store them to a .spacy file to use in training. You can find more information on this general process & data formats in the docs: https://spacy.io/api/data-formats#training

With respect to relation extraction - there is currently no builtin functionality in spaCy for this, but you can look at the example project we have compiled, that uses a custom data format. There is also a generic script for converting Prodigy annotations to the spaCy format, that you can use as inspiration. The video explains it in more detail.