Is there a way to extract the original annotation from .spacy file?

[vmatter]

I am trying to create a function to evaluate some annotation of my original test data, so I would like to know if is it possible to convert a .spacy file back to the annotation format.

Example of annotation format:

examples = [
    ('Who is Steve Gates?',
     {(7, 18, 'PERSON')})]

What I am able to do by now:

# loads the best model
nlp_test_annotated = spacy.load("./output/model-best")

# recovers a spacy file from disk:
`doc_bin_test = DocBin().from_disk("test.spacy")`

Afterward, I can access the entities text and label using:

list_of_annotated_documents = list(doc_bin_test.get_docs(nlp_test_annotated.vocab))

for index, document in enumerate(list_of_annotated_documents):
  # Classify the test data texts using the trained model
  doc = nlp_test_annotated(list_of_annotated_documents[index])
  # Get the labels from the classification
  list_of_annotations = [(ent.text, ent.label_) for ent in doc.ents]
  print(list_of_annotations)

Doing that I get this output:

[('Steve Gates', 'Name')

Is there any optimized way of transforming the .spacy file, back to its original annotation format?

I have one function that evaluates the training and data files which uses spacy's evaluate.

And the one that I am creating now is for simple outputs that the user wants to test (e.g., simple annotations like the one in the example above), in which the user adds the annotation and I am able to evaluate the output using the following function:

def evaluate_docs(ner_model, examples):
  scorer = Scorer()
  example = []
  for input_, annotations in examples:
      pred = ner_model(input_)
      print(pred,annotations)
      temp = Example.from_dict(pred, dict.fromkeys(annotations))
      example.append(temp)
  scores = scorer.score(example)
  return scores

Perhaps using the Example class, however, I could not think of a solution for getting the initial annotation from the .spacy file.

Is there a way to get the original annotation using the generated .spacy file?

Many thanks.

[polm]

I am trying to create a function to evaluate some annotation of my original test data, so I would like to know if is it possible to convert a .spacy file back to the annotation format.

The original annotation format is not saved anywhere in the Doc, so in the general case, no, the original annotations can't be generated right away. Depending on the format there might be third-party tools to produce it from Docs, like spacy_conll. All the relevant information should be saved in the Doc though, so generating it usually isn't hard.

Your input annotation format looks pretty simple, so you should just be able to create it from your Docs with short code like this.

out = []
for doc in docs:
    ents = [(ee.start_char, ee.end_char, ee.label) for ee in doc.ents]
    out.append( (doc.text, ents) )

I don't really understand how this relates to the evaluate function.

[vmatter]

Many thanks, I will test the code that you wrote.

The objective of doing something like that is to get one or many of my original documents in the test_dataset to use as the examples parameter for that custom evaluate function, just as a way for testing specific inputs with my trained model.

Then you might ask why I don't use the original test dataset and only use my conversion functions again? That would be a good question as well, however, I really wanted to know if there was a way to convert back to the annotation form.😅

In the end, I guess all my workaround would be the same as getting a specific quantity of documents from my test_dataset, converting it to a .spacy file, and running my original evaluate:

spacy evaluate output/model-best test.spacy

Perhaps this would be the better way in the end...

Well, I will try both ways and find out which is better, once again, many thanks.

[polm]

Ah, thanks for clarifying.

So an Example is just a pair of Docs, so if you already have the Docs there's no need to go back to raw annotations. The Example.from_dict is provided for convenience if you already have data in that format, but if you have Docs you can just do example = Example(predicted, reference).

[vmatter]

Many thanks, I did not know this way of using the Example class, but I will try it out for sure!