SpanCat returning training score as 0

[geetarajagopal]

Hello,

I am working on a NLP project using spacy where i am trying to use the SpanCat component. I already used ner but i am unable to get the desired performance. I prepared my training data for SpanCat using the method mentioned in this post and the several other posts including using SpanGroup, however when i train my model using these train.spacy and dev.spacy files, my model does not seem to provide any scores. PFB output while training my model:

Below is my code:

I tried different combinations by playing around with the default config file generated for spancat but none of them work. I tried using the spacy files from the projects provided by spacy(experimental/ner_spancat) and spancat seems to train properly for those spacy files.

Can someone please provide me pointers where i might be going wrong in preparing the training data? TIA

Originally posted by @geetarajagopal in #11351 (reply in thread)

[polm]

Couple of things:

• please do not post screenshots of code or terminal output, they are hard to read. Paste text as text, and use mardown code blocks where it makes sense.
• Please do not post the same question in multiple places.

About your problem:

• what is PFB?
• please share your training config.
• can you share some of your data for us to check?
• can you run spacy debug data to check if there are any issues with your data?

[geetarajagopal]

@ljvmiranda921 thank you for the pointers. I tried both approaches suggested by you with results as below:
1)When i changed config file to change suggester as @misc = "spacy-experimental.sentence_suggester.v1" while training, i got error:

I tried adding the "sentencizer" and "parser" components to the config cfg but still the error persisted. Below is the config file i tried with parser added:
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","parser","spancat"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@Tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.parser]
factory = "parser"
learn_tokens = false
min_action_freq = 30
moves = null
scorer = {"@scorers":"spacy.parser_scorer.v1"}
update_with_oracle_cut_size = 100

[components.parser.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "parser"
extra_state_tokens = false
hidden_width = 128
maxout_pieces = 3
use_upper = true
nO = null

[components.parser.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "grit"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy-experimental.sentence_suggester.v1"
#sizes = [10]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,1000,2500,2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@Loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
dep_uas = 0.25
dep_las = 0.25
dep_las_per_type = null
sents_p = null
sents_r = null
sents_f = 0.0
spans_grit_f = 1.0
spans_grit_p = 0.0
spans_grit_r = 0.0
spans_sc_f = null
spans_sc_p = null
spans_sc_r = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

2)For this approach i started with 36 and tried reducing the n-grams eventually. For all n-grams that i was setting i was seeing almost the same result as below with score still being 0:

[adrianeboyd]

Please use markdown code blocks with three backticks (```) for both code and output. It makes it much easier to read and search.

For the sentence suggester, to have the sentence boundaries added during training: add sentencizer as a component and then add sentencizer to [training.annotating_components].

[geetarajagopal]

Sure, will take care of that..Added sentencizer to the pipeline and below is the output

[2022-11-04 11:20:17,912] [INFO] Set up nlp object from config
INFO:spacy:Set up nlp object from config
[2022-11-04 11:20:17,923] [INFO] Pipeline: ['tok2vec', 'sentencizer', 'spancat']
INFO:spacy:Pipeline: ['tok2vec', 'sentencizer', 'spancat']
[2022-11-04 11:20:17,927] [INFO] Created vocabulary
INFO:spacy:Created vocabulary
[2022-11-04 11:20:17,928] [INFO] Finished initializing nlp object
INFO:spacy:Finished initializing nlp object
[2022-11-04 11:20:47,310] [INFO] Initialized pipeline components: ['tok2vec', 'sentencizer', 'spancat']
INFO:spacy:Initialized pipeline components: ['tok2vec', 'sentencizer', 'spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'sentencizer', 'spancat']
ℹ Set annotations on update for: ['sentencizer']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SENTS_F  DEP_UAS  DEP_LAS  SPANS_GRIT_F  SPANS_GRIT_P  SPANS_GRIT_R  SCORE 
---  ------  ------------  ------------  -------  -------  -------  ------------  ------------  ------------  ------
  0       0          4.59        317.31    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  1     200          2.14        192.34    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  2     400          0.00          0.10    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  4     600          0.00          0.05    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  5     800          0.00          0.03    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  6    1000          0.00          0.02    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  8    1200          0.00          0.02    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  9    1400          0.00          0.01    24.26     0.00     0.00          0.00          0.00          0.00    0.00
 10    1600          0.00          0.01    24.26     0.00     0.00          0.00          0.00          0.00    0.00
✔ Saved pipeline to output directory
train/model-last 

[adrianeboyd]

In your training corpus, do the sentencizer sentences line up with the annotated spans? Do you see any annotations at all if you lower the threshold a lot in the spancat config?

[geetarajagopal]

This is the kind of data i have in my training

{"id": 54279, "text": "Admission Date:  2009-05-31              Discharge Date:   2015-07-01\n\nDate of Birth:  2004-11-13             Sex:   M\n\nService: NEUROLOGY\n\nAllergies:\nBactrim\n\nAttending:Edwards\nChief Complaint:\naphasia, rightside plegia\n\n\nMajor Surgical or Invasive Procedure:\nnone\n\nHistory of Present Illness:\n Professor Moran is a 64-year-old gentleman with a history\nof atrial fibrillation last INR  1.2, who was last seen normal\nat\n11:30 to 12 am who presents with new onset aphasia and right\nhemiplegia.  Patient had spent the day playing with his\ngrandchildren.  He then was watching the Red Sox game on TV and\nit is unclear when he went to bed.  The son was 2845 Hogan Radial Suite 878\nNorth Stevenfort, IL 97737 a book\nand thinks he heard him around midnight.  At some point in the\nnight he woke up and went down stairs.  The wife also went down\nstairs and noted his speech was garbled.  The daughter came home\na little after 2 and noted he had a right facial droop and .......................DECREASED lisinopril to 10mg daily\nWe STARTED albuterol and ipratropium nebulisers as required for\nyour breathing difficulties\nWe STARTED laxatives\nPlease continue your other medications as prescribed\n\n\nFollowup Instructions:\nYou should follow-up with your PCP [**Name Initial (PRE) 176**] 1 week after discharge\nfrom rehab. Choi,Richard M. 900-138-2520\n\nYou also have the following neurology follow-up appointment.\n\nDepartment: NEUROLOGY\nWhen: TUESDAY 1978-12-14 at 1 PM\nWith: Joshua Figueroa Claudia Solomon, M.D. 942-874-8214\nBuilding: St. Mary Regional Medical Center 298 Johnson Rest\nSouth Jessicaside, SC 03271\nCampus: EAST     Best Parking: Oncology Garage\n\n\n\nCompleted by:2015-07-01", "spans": [{"document_id": 54279, "type": ["Appointment-related followup"], "start": 11767, "end": 11855}, {"document_id": 54279, "type": ["Medication-related followups"], "start": 12442, "end": 12554}, {"document_id": 54279, "type": ["Medication-related followups"], "start": 12729, "end": 12774}, {"document_id": 54279, "type": ["Procedure-related followup"], "start": 13018, "end": 13127}, {"document_id": 54279, "type": ["Procedure-related followup"], "start": 13351, "end": 13434}, {"document_id": 54279, "type": ["Other helpful contextual information"], "start": 13529, "end": 13694}, {"document_id": 54279, "type": ["Medication-related followups"], "start": 15118, "end": 15210}, {"document_id": 54279, "type": ["Procedure-related followup"], "start": 17530, "end": 17652}, {"document_id": 54279, "type": ["Other helpful contextual information"], "start": 17655, "end": 17814}, {"document_id": 54279, "type": ["Appointment-related followup"], "start": 18222, "end": 18349}, {"document_id": 54279, "type": ["Appointment-related followup"], "start": 18351, "end": 18625}]}

And i am using the code that i have given in my original post for creating the .spacy file out of this json data.

I reduced the threshold to 0.01 and still got the same output as before:

ℹ Saving to output directory: train
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-11-12 19:01:21,667] [INFO] Set up nlp object from config
INFO:spacy:Set up nlp object from config
[2022-11-12 19:01:21,678] [INFO] Pipeline: ['tok2vec', 'sentencizer', 'spancat']
INFO:spacy:Pipeline: ['tok2vec', 'sentencizer', 'spancat']
[2022-11-12 19:01:21,683] [INFO] Created vocabulary
INFO:spacy:Created vocabulary
[2022-11-12 19:01:21,684] [INFO] Finished initializing nlp object
INFO:spacy:Finished initializing nlp object
[2022-11-12 19:01:50,908] [INFO] Initialized pipeline components: ['tok2vec', 'sentencizer', 'spancat']
INFO:spacy:Initialized pipeline components: ['tok2vec', 'sentencizer', 'spancat']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'sentencizer', 'spancat']
ℹ Set annotations on update for: ['sentencizer']
ℹ Initial learn rate: 0.001
E    #       LOSS TOK2VEC  LOSS SPANCAT  SENTS_F  DEP_UAS  DEP_LAS  SPANS_GRIT_F  SPANS_GRIT_P  SPANS_GRIT_R  SCORE 
---  ------  ------------  ------------  -------  -------  -------  ------------  ------------  ------------  ------
  0       0          4.59        317.31    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  1     200          2.14        192.34    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  2     400          0.00          0.10    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  4     600          0.00          0.05    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  5     800          0.00          0.03    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  6    1000          0.00          0.02    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  8    1200          0.00          0.02    24.26     0.00     0.00          0.00          0.00          0.00    0.00
  9    1400          0.00          0.01    24.26     0.00     0.00          0.00          0.00          0.00    0.00
 10    1600          0.00          0.01    24.26     0.00     0.00          0.00          0.00          0.00    0.00
✔ Saved pipeline to output directory
train/model-last