meta.json labels for a custom NER using pretrained transformer

[geetarajagopal]

Hi

I am training a custom NER model using a hugging face pretrained transformer model. The transformer model provides 7 labels for annotating the text. When i look at the meta.json file of my created model, i see all the labels within the ner section instead of the transformer section, whereas all the labels are coming from the transformer section as well as my training data. Below are my config file and meta.json files. Please let me know if i am missing some step or if my meta.json is correct. Another reason for asking this question is, inspite of using a pretrained transformer, my scores are coming very low i i.e. in range of 0.2 - 0.3(as can be seen in the label scores in the meta.json)

train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "jamesmullenbach/CLIP_DNote_BERT_Context"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

  "lang":"en",
  "name":"pipeline",
  "version":"0.0.0",
  "spacy_version":">=3.4.3,<3.5.0",
  "description":"",
  "author":"",
  "email":"",
  "url":"",
  "license":"",
  "spacy_git_version":"Unknown",
  "vectors":{
    "width":0,
    "vectors":0,
    "keys":0,
    "name":null,
    "mode":"default"
  },
  "labels":{
    "transformer":[

    ],
    "ner":[
      "Activity",
      "Administration",
      "Age",
      "Appointment-related followup",
      "Area",
      "Biological_structure",
      "Case-specific instructions for patient",
      "Clinical_event",
      "Color",
      "Coreference",
      "Date",
      "Detailed_description",
      "Diagnostic_procedure",
      "Disease_disorder",
      "Distance",
      "Dosage",
      "Duration",
      "Family_history",
      "Frequency",
      "History",
      "Imaging-related followup",
      "Lab-related followup",
      "Lab_value",
      "Medication",
      "Medication-related followups",
      "Nonbiological_location",
      "Occupation",
      "Other helpful contextual information",
      "Outcome",
      "Personal_background",
      "Procedure-related followup",
      "Severity",
      "Sex",
      "Sign_symptom",
      "Subject",
      "Texture",
      "Therapeutic_procedure",
      "Time"
    ]
  },
  "pipeline":[
    "transformer",
    "ner"
  ],
  "components":[
    "transformer",
    "ner"
  ],
  "disabled":[

  ],
  "performance":{
    "ents_f":0.4229236899,
    "ents_p":0.4448742747,
    "ents_r":0.4030373832,
    "ents_per_type":{
      "Date":{
        "p":0.3835616438,
        "r":0.2666666667,
        "f":0.3146067416
      },
      "Clinical_event":{
        "p":0.62,
        "r":0.5961538462,
        "f":0.6078431373
      },
      "Sign_symptom":{
        "p":0.6728971963,
        "r":0.5853658537,
        "f":0.6260869565
      },
      "Detailed_description":{
        "p":0.4038461538,
        "r":0.3684210526,
        "f":0.3853211009
      },
      "Biological_structure":{
        "p":0.4705882353,
        "r":0.5839416058,
        "f":0.5211726384
      },
      "Diagnostic_procedure":{
        "p":0.387755102,
        "r":0.2878787879,
        "f":0.3304347826
      },
      "Lab_value":{
        "p":0.3953488372,
        "r":0.2615384615,
        "f":0.3148148148
      },
      "Time":{
        "p":0.1666666667,
        "r":0.125,
        "f":0.1428571429
      },
      "History":{
        "p":0.25,
        "r":0.2307692308,
        "f":0.24
      },
      "Duration":{
        "p":0.2666666667,
        "r":0.32,
        "f":0.2909090909
      },
      "Subject":{
        "p":0.303030303,
        "r":0.7692307692,
        "f":0.4347826087
      },
      "Lab-related followup":{
        "p":0.2666666667,
        "r":0.2222222222,
        "f":0.2424242424
      },
      "Medication-related followups":{
        "p":0.4651162791,
        "r":0.2631578947,
        "f":0.3361344538
      },
      "Appointment-related followup":{
        "p":0.3891891892,
        "r":0.4931506849,
        "f":0.4350453172
      },
      "Therapeutic_procedure":{
        "p":0.2051282051,
        "r":0.126984127,
        "f":0.1568627451
      },
      "Disease_disorder":{
        "p":0.2755102041,
        "r":0.3857142857,
        "f":0.3214285714
      },
      "Imaging-related followup":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Other helpful contextual information":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Nonbiological_location":{
        "p":0.4415584416,
        "r":0.4358974359,
        "f":0.4387096774
      },
      "Sex":{
        "p":0.8620689655,
        "r":0.9615384615,
        "f":0.9090909091
      },
      "Dosage":{
        "p":0.2352941176,
        "r":0.1025641026,
        "f":0.1428571429
      },
      "Medication":{
        "p":0.3103448276,
        "r":0.1363636364,
        "f":0.1894736842
      },
      "Procedure-related followup":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Age":{
        "p":1.0,
        "r":0.9090909091,
        "f":0.9523809524
      },
      "Severity":{
        "p":0.6666666667,
        "r":0.9411764706,
        "f":0.7804878049
      },
      "Activity":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Outcome":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Case-specific instructions for patient":{
        "p":0.0384615385,
        "r":0.0625,
        "f":0.0476190476
      },
      "Coreference":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Administration":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Color":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Personal_background":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Frequency":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      },
      "Family_history":{
        "p":0.0,
        "r":0.0,
        "f":0.0
      }
    },
    "transformer_loss":14753.4645913839,
    "ner_loss":22150.073728587
  }
}```

[polm]

spacy-transformers only uses Transformers as feature sources, it can't use NER or other task-specific heads. There's also no way for a transformer component to have labels for that reason.

Looking at per-label scores, several of them have zero, which might happen if you have no training examples of that score. What's the output of debug data for your training data? For the non-zero one, several of them seem to have pretty reasonable scores.

[polm]

Also:

[components.textcat]
factory = "textcat_multilabel"
scorer = {"@scorers":"spacy.textcat_scorer.v1"}
threshold = 0.5

You are using textcat_multilabel with the textcat_scorer instead of the textcat_multilabel_scorer, which is weird. The scorer should match the type of textcat you're using. How did you generate this config?

[geetarajagopal]

Hi polm

thanks a lot for all your guidance and help here.
I created the config file from the widget on spacy.io selecting textcat as the component to be used. It created the textcat configuration and factory for me by default, and since i had to use multilabel, i replaced the factory with textcat_multilabel. Right now, i replaced the textcat_scorer with textcat_multilabel_scorer and ran the evaluate CLI command. I could not see any difference in the scores. Do i need to retrain the model with this modified config file?

[polm]

You would need to run training again after changing the scorer config. However, the main problem is using the BOW architecture. If you're using the widget, you should select the "accuracy" config. Also make sure "exclusive categories" is not checked, which will use the multilabel config.

[geetarajagopal]

Thanks a lot, i was able to get the transformer loss with textcat_multilabel by creating config with Accuracy option and unchecking exclusive categories. Thanks for all your guidance

[polm]

Glad you got it working!