Converting Huggingface dataset.datasets to DocBin

[emiltj]

How do I convert a HF dataset to DocBins?

I tried:

from datasets import load_dataset
import spacy
from spacy.tokens import DocBin, Doc

datasets = load_dataset("chcaa/DANSK")["train"] # This is a Dataset
train = [Doc(nlp.vocab).from_json(dataset_row) for dataset_row in datasets["train"]]
db = DocBin()
for doc in train:
    db.add(doc)

But it is rather slow. I wonder if there's a more optimal way of doing it?

[thomashacker]

Hey, thanks for your post!
Doc.from_json uses a specific JSON format and won't support whatever format load_dataset is returning. You'll need to convert the annotation to spaCy's format. If you'd like, we can provide some help with the conversion if you let us know how your annotation is structured.