Spancat - How to append SpanCategorizer into a trained custom NER model ?

[Genco2]

Hello Spacy Community;
I have been working on a custom model for some time; I created a pipeline("tok2vec", "ner"), training data and relevant custom labels, prepare config file etc.
At this moment the requirement of obtaining confidence scores of spans have emerged. May you please guide me about how to use spancat in such a case ? Is there a way to inject the "spancat" to my model or do I have to retrain the model together with "spancat" ?
Thanks.

[polm]

You'll need to replace the NER model with a spancat component trained on the same data. Look at the spancat example project for details on how to prepare the data and config file.

[Genco2]

Thanks a lot.

[Jchandar]

You can use spacy init config -p ner,spancat to generate a config with both. The default spancat component config currently contains its own internal tok2vec rather than a tok2vec listener, but you can edit the config to create a pipeline where both ner and spancat listen to the same tok2vec component if you'd like.

Be aware that if you to train ner and spancat from the same doc.ents annotation, you'll need to copy that to doc.spans["sc"] as in the script in the example project:

https://github.com/explosion/projects/blob/24553c6786aeaa44d5f308be8a43504aec073c87/experimental/ner_spancat/scripts/add_ents_to_spans_dict.py

Hi adrianeboyd,
I converted doccano to spacy format and then did doc ents to doc.span["sc"] in the script. when I train the span score gives zero. I could not find any related answers to that. Could you help me out please.Thanks in advance.

[polm]

Please do not paste screenshots of code, they are not helpful. Copy and paste the code as text. In this case it would help to see the code you used to convert the entities to spans.

[Jchandar]

Sorry for that, This is the issue I am getting when I train the model the span score gives zero.

import pandas as pd
from tqdm import tqdm
import spacy
from spacy.tokens import DocBin

nlp = spacy.blank("en") # load a new spacy model
db = DocBin() # create a DocBin object
for text, annot in tqdm(ALL_DATA): # data in previous format
    doc = nlp.make_doc(text) # create doc object from text
    try:
      ents = []
      for start, end, label in annot["entities"]: # add character indexes
          span = doc.char_span(start, end, label=label, alignment_mode="strict")
          if span is not None:
              ents.append(span)            
      doc.ents = ents # label the text with the ents
    except Exception as e:
      doc.ents = []
      logging.exception("Unable to process, error = " + str(e))
    spacy.displacy.render(doc, style="ent", jupyter=True)
    db.add(doc)

db.to_disk("./spandata.spacy") # save the docbin object

#after getting ner data converting into doc.spans["sc"]
docbin = DocBin().from_disk("./spandata.spacy")
docs = list(docbin.get_docs(nlp.vocab))
for doc in docs:
    doc.spans["sc"] = list(doc.ents)
DocBin(docs=docs).to_disk("./spansdata.spacy")

#initializing Config
!python3 -m spacy init config configs.cfg --lang en --pipeline spancat --optimize efficiency

#Training the model
%%bash
#!/bin/bash
echo start: $(date "+%y%m%d.%H%M%S.%3N")
python3 -m spacy train configs.cfg --output ./output_ --paths.train ./spansdata.spacy --paths.dev ./spansdata.spacy
echo stop:  $(date "+%y%m%d.%H%M%S.%3N")

#while training the result like
start: 220318.120955.777
[2022-03-18 12:09:56,889] [INFO] Set up nlp object from config
[2022-03-18 12:09:56,895] [INFO] Pipeline: ['spancat']
[2022-03-18 12:09:56,899] [INFO] Created vocabulary
[2022-03-18 12:09:56,899] [INFO] Finished initializing nlp object
[2022-03-18 12:10:13,714] [INFO] Initialized pipeline components: ['spancat']
ℹ Saving to output directory: output_
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['spancat']
ℹ Initial learn rate: 0.001
E    #       LOSS SPANCAT  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  ----------  ----------  ----------  ------
  0       0     290015.16        0.01        0.00       32.90    0.00
 20     200    1066567.45        0.00        0.00        0.00    0.00
 40     400       6309.75        0.00        0.00        0.00    0.00
 60     600       6329.55        0.00        0.00        0.00    0.00
 80     800       6303.34        0.00        0.00        0.00    0.00
100    1000       5573.27       38.91       91.94       24.68    0.39

[polm]

Can you run the debug data command and share the output?

Your spancat component doesn't always have zero scores, so it looks like you have some data. But it seems like you only have 100 examples, so maybe something is a bit off with them. Also since your scores went up at the end you probably want to train a bit longer.