Unable to train SpanFinder model the scores are always zero

[vignesh-spericorn]

Hello

I'm trying to train a span_finder model using this as reference. My goal is to train a span_cat model using this span_finder model for extracting required data from resumes.
I have tried the sample project here and was able to successfully train a span_finder model.
I have annotated a small set of resumes using NER annotator.

I used this config file and modified only two things

1. Set batch size = 10
2. Changed spans_key = "sc"

I converted the annotated data to spacy format using the following function:

from spacy.tokens import SpanGroup
def get_spacy_doc(file,data, nlp):
  db = DocBin()

  num = 0
  for text,annot in tqdm(data):
    doc = nlp.make_doc(text)
    annot = annot["entities"]

    spans = []
    entity_indices = []
    counter = 0
    for start,end,label in annot:
      # print(start,end,label)
      skip_entity = False
      for idx in range(start,end):
        # print(idx)
        if idx in entity_indices:
          skip_entity = True
          # print("skip_entity worked")
          break
      if skip_entity == True:
        counter += 1
        print(counter)
        continue

      entity_indices = entity_indices + list(range(start,end))

      try:
        span = doc.char_span(start,end,label=label,alignment_mode="strict")
      except:
        print("span except worked")
        continue

      sl_no = 0
      if span is None:
        sl_no += 1
        write_text = f"{sl_no} : Text id = {num} --- {start}, {end} --- {text[start:end]} /---"
        err_data =  write_text + "\n"
        file.write(err_data)
      else:
        spans.append(span)

    try:
      group = SpanGroup(doc, name="sc", spans=spans)
      doc.spans["sc"] = group
      db.add(doc)
    except:
      pass

    num+=1
  return db

This is my output when i run spacy debug_data command:

============================ Data file validation ============================
✔ Pipeline can be initialized with data
✔ Corpus is loadable

=============================== Training stats ===============================
Language: en
Training pipeline: tok2vec, span_finder
39 training docs
6 evaluation docs
✔ No overlap between training and evaluation data
✘ Low number of examples to train a new pipeline (39)

============================== Vocab & Vectors ==============================
ℹ 20283 total word(s) in the data (4559 unique)
ℹ 514157 vectors (514157 unique keys, 300 dimensions)
⚠ 854 words in training data without vectors (4%)

================================== Summary ==================================
✔ 3 checks passed
⚠ 1 warning
✘ 1 error

When training the scores are always zero. I'm not sure what's the problem here.

ℹ Saving to output directory:
/content/drive/MyDrive/Resume-Analysis/Span-Finder/output-1
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_finder']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPAN_...  SPANS_ENTI...  SPANS_ENTI...  SPANS_ENTI...  SCORE 
---  ------  ------------  -------------  -------------  -------------  -------------  ------
  0       0          9.01         434.18           0.00           0.00           0.00    0.00
  5     200          0.62        9454.37           0.00           0.00           0.00    0.00
 10     400          0.00        9312.00           0.00           0.00           0.00    0.00
 15     600          0.00        9432.00           0.00           0.00           0.00    0.00

Aborted.

Kindly requesting the communities help.

Regards,

[adrianeboyd]

This is probably not enough examples for the model to generalize to new eval data.

If you want to test the training process with a very small number of examples, it can make sense to use the whole dataset both as the train and the dev data. Obviously just for testing, but you can make sure that the config is working and that the model can overfit and memorize training examples.

[vignesh-spericorn]

Hello,

Sorry for the late reply. There was a problem in the config file as per the comment here I modified the config file from this

[components.span_finder.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

to this

[components.span_finder.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = 96
upstream = "*"

[components.tok2vec]
source = "en_core_web_lg"

Now the span_finder model is training

ℹ Saving to output directory:
/content/drive/MyDrive/Resume-Analysis/Span-Finder/output-1
ℹ Using CPU

=========================== Initializing pipeline ===========================
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'span_finder']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPAN_...  SPANS_SC_F  SPANS_SC_P  SPANS_SC_R  SCORE 
---  ------  ------------  -------------  ----------  ----------  ----------  ------
  0       0          7.38         105.80        0.13        0.07       52.00    0.42
  3     200        126.92        4022.02       22.92       23.91       22.00    0.22
  6     400         75.28        1856.22       24.64       19.32       34.00    0.31
 10     600         51.59        1141.35       26.32       38.46       20.00    0.24
 13     800         34.07         681.66       19.35       20.93       18.00    0.19
 16    1000         26.64         499.02       17.65       33.33       12.00    0.16
 20    1200         18.85         387.71       19.75       25.81       16.00    0.18
 23    1400         14.31         319.55       22.78       31.03       18.00    0.21
 26    1600         11.85         284.37       19.51       25.00       16.00    0.18
✔ Saved pipeline to output directory
/content/drive/MyDrive/Resume-Analysis/Span-Finder/output-1/model-last

Any idea why this is happening ?

[adrianeboyd]

It's not a problem with your original config exactly, but sourcing tok2vec from en_core_web_lg instead of a randomly-initialized tok2vec lets you benefit from transfer learning from the English tagger and parser.

However from the overall scores, it still looks like the main issue is that there is not enough training data.

[vignesh-spericorn]

Can you tell me what is the minimum number required for training ? Currently i have around 600 resumes is this enough for training a Spanfinder and then a Spancat model ?