Data set conversion and cross validation with spacy

[venti07]

I used prodi.gy to annotate a NER dataset. After that, I exported the data as .jsonl file.

I now want to create datasets to be able to perform cross validation. To do this, I convert the files to a dataframe and then back to a .jsonl.

I have a few questions about this:

How can I convert the individual dataframes or jsonl files to .spacy format?
How can I create a loop to train with the individual datasets? Spacy can be used primarily via CLI.
How can I process the corresponding outputs (F1, Acc) from the CLI in Python, since they only appear as output?

Many thanks for your help!

[ljvmiranda921]

Hi venti07!

How can I convert the individual dataframes or jsonl files to .spacy format?

You can check this documentation on how to perform serialization. In summary, you need to convert your texts into a spaCy Doc object, then collate them into a DocBin object, then save to disk. Converting to Doc depends on what kind of data you already have.

Since you annotated via Prodigy, you can export your dataset straight to spaCy by using the data-to-spacy command

How can I create a loop to train with the individual datasets? Spacy can be used primarily via CLI.

You have the following options:

1. Write a bash script that loops through your dataset and constantly call spacy train.
2. In Python, you can import the train command and pass different files for each loop. This means you need to write an external loop that calls this command.
3. If you're interested in cross-validation, you can write a custom reader with an additional option of returning a specific fold. A good starting point would be adapting spacy.Corpus.v1. Instead of returning the full set of Examples, it returns "chunks" of your Example objects.

How can I process the corresponding outputs (F1, Acc) from the CLI in Python, since they only appear as output?

You can check the meta.json file in your training folder. Assuming that you're saving your model in a directory called training, you will find said file in training/model-best/meta.json. There's also an equivalent with model-last.

[venti07]

Since you annotated via Prodigy, you can export your dataset straight to spaCy by using the data-to-spacy command

Yes, I know the command. However, I have to split the data for cross validation, so unfortunately I can't use this one. Is there a code sample directly from explosion to convert the data from a dataframe to a .spacy file? Or from .jsonl to .spacy?

1. Write a bash script that loops through your dataset and constantly call spacy train.
2. In Python, you can import the train command and pass different files for each loop. This means you need to write an external loop that calls this command.
   You can check the meta.json file in your training folder. Assuming that you're saving your model in a directory called training, you will find said file in training/model-best/meta.json. There's also an equivalent with model-last.

This means I can build an external loop and name each model differently. For further evaluation, I would then look at the meta.json of each model and evaluate the cross validation with the data?

[ljvmiranda921]

Is there a code sample directly from explosion to convert the data from a dataframe to a .spacy file? Or from .jsonl to .spacy?

You can look at this script as a starting point. When you run db-out and export your annotations to JSONL, you will be provided with the text and the span information. You can then access those values to create Doc objects (cf. this line of code) and assign the appropriate entities (i.e., doc.ents as shown in these lines of code). Note that the sample data in this project is a different JSONL file (it's the v2 format), but you should be able to access the same kind of information when you export from Prodigy.

This means I can build an external loop and name each model differently. For further evaluation, I would then look at the meta.json of each model and evaluate the cross validation with the data?

Yes, if you want to go with this option, you need to loop across your folds, train a model for that particular fold (by calling the train loop method ) and evaluate them via Language.evaluate.

[venti07]

I have implemented the whole thing. Input is a dataframe which was created from the .jsonl of prodigy. Maybe you can take a look at it again, if it looks good for you? I was able to use the SpaCy files to train a model. I am unsure if I have thought of everything though and look forward to feedback.

import srsly
import typer
import warnings
from pathlib import Path

import spacy
from spacy.tokens import DocBin

def convert(lang, dataframe, output_path):

  nlp = spacy.blank(lang)
  db = DocBin()
  
  for text, spans in dataframe[['text','spans']].itertuples(index=False):
    
      doc = nlp.make_doc(text)
      ents = []
      for dicts in spans:
        for k,v in dicts.items():
          if k == 'start':
            start = v
          if k == 'end':
            end = v
          if k =='label':
            label = v
      
            span = doc.char_span(start, end, label=label)
            if span is None:
                msg = f"Skipping entity [{start}, {end}, {label}] in the following text because the character span '{doc.text[start:end]}' does not align with token boundaries:\n\n{repr(text)}\n"
                warnings.warn(msg)
            else:
                ents.append(span)
      doc.ents = ents
      db.add(doc)
  db.to_disk(output_path)

[ljvmiranda921]

Hi @venti07 , iterating on a dataframe might be a bit slow and sub-optimal, so you might prefer converting it into another data structure (perhaps a list of tuples, etc.) before iterating on them. Lastly, we have an example project that does cross-validation here you can check it out.