Cannot convert json data to spacy format

[nleguillarme]

dev.json.txt

How to reproduce the behaviour

python -m spacy convert dev.json . --converter json

Your Environment

• spaCy version: 3.0.0rc2
• Platform: Linux-5.4.0-52-generic-x86_64-with-Ubuntu-18.04-bionic
• Python version: 3.6.9

Hi, I'm trying to convert some json training data into spacy format using the convert command, so that I can try fine-tuning a pre-trained transformer model with the nightly version of spaCy.

I used this data successfully for fine-tuning a scispaCy NER model. So I know the format is valid for previous versions of spaCy.

When running the command above, I get the following result :
Generated output file (0 documents): dev.spacy`

You can find attached the file dev.json

[adrianeboyd]

The JSON converter is for a specific spacy JSON training format described here: https://spacy.io/api/annotation#json-input

See this answer for how to convert from the simple training format directly to DocBin (.spacy) for v3: https://stackoverflow.com/a/64677899/461847

[nleguillarme]

Thank you for your answer.

I've just tried on my data and got : ValueError: [E973] Unexpected type for NER data

Any idea ?

[adrianeboyd]

Make sure your data is in the simple training format as shown in the example here: https://github.com/explosion/spaCy/blob/45c9a688285081cd69faa0627d9bcaf1f5e799a1/examples/training/train_ner.py

I think you needs tuples instead of lists for the entity spans so the annotations are detected correctly by Example.from_dict.

[nleguillarme]

It works like a charm. Thank you very much.

[xsallowed]

test_logs_auth.txt
I am having a problem that my files are not being processed. Can you please help me understand what am I doing wrong?

[polm]

@xsallowed Your data is not in the spaCy JSON format mentioned in this thread. It looks like you should be able to use the example code in the training data docs with a little modification though - it looks like you can probably use the process described in Adriane's answer.

Also, this thread is pretty old, if you'd like to discuss this more please open a new Discussion.