How to convert .spacy training / testing data and use inside existing python code

[milos-cuculovic]

Hi,

I have an existing code in Python that worked with SpaCy v2 by using JSON format datasets:

examples = []
    for text, annotations in TEST_DATA:
        examples.append(Example.from_dict(nlp.make_doc(text), annotations))

Now that I converted the JSON files to .spacy binary files, this code is no longer working.
I am trying to fix it this way:

nlp = spacy.load("my-model")
    doc_bin = DocBin().from_disk(path_test_data)
    examples = []
    for doc in doc_bin.get_docs(nlp.vocab):
        spans = [{"start": ent.start_char, "end": ent.end_char, "label": ent.label_} for ent in doc.ents]
        examples.append(Example.from_dict(nlp.make_doc(doc.text), spans))

But this is returning an error:

TypeError: Argument 'example_dict' has incorrect type (expected dict, got list)

[ljvmiranda921]

Hi @milos-cuculovic ,

The second argument, example_dict should receive a dictionary, but we're passing a list of spans. You can always check the data formats here to be sure (for example, {"entities": [...] ...}). However, you can do something like this instead:

Example(nlp.make_doc(doc.text), doc)

Another thing to keep in mind is that you might run into some problems if the vocabs aren't the same, so just be aware of that.

[milos-cuculovic]

Thanks @ljvmiranda921 , I was able to solve it by using this code:

doc_bin = DocBin().from_disk(path_test_data)
examples = []
for doc in doc_bin.get_docs(nlp.vocab):
    entities = []
    for ent in doc.ents:
        entities.append((ent.start_char, ent.end_char, ent.label_))

    spacy_entry = (doc.text, {"entities": entities})
    examples.append(spacy_entry)