Help on Training SpanCat or NER model using Transformer

[egumasa]

I am trying to apply the Span Categorizer and/or NER architecture (https://github.com/explosion/projects/tree/v3/experimental/ner_spancat) into adverbial disambiguation task in English (adverbs of manner, temporal, etc.). I was able to successfully replicate the original SpanCategorizer project in Indonesian. I was also able to swap the dataset and language (that is, into English adverbial disambiguation task), and preliminarily train a model using Tok2Vec architecture. Then, I have been stuck in swapping the tok2vec to transformer architecture. Below, I describe the overall plan and specific problems I have. I appreciate if you could take a look at the .cfg file below and see if there is any errors.

Project plan and progress

Goal: Applying ner OR SpanCategorizer architecture into English Adverbial disambiguation task.
Steps:

• [ x] Replicate Indonesian spancategorizer project in my environment
• [ x] Swap dataset to English and train custom SpanCat and NER model for adverb disambiguation (the performance was not ideal, which encouraged me to use transformer).
• Change tok2vec to transformer model in Indonesian project and replicate the project using indolem/indobert-base-uncased (which I get recommended when I selected Indonesian in the following training tutorial). The specific issue is that I get 0s for evaluation metrics.
• Change tok2vec to transformer in original English adverb disambiguation project. The issue is the same; I get 0s for evaluation metrics.

Issue

The issue is that when I train the model using transformer models, I get 0s for evaluation metrics. I think this may mean that the no training or prediction is made (due to errors in cfg file). I would like to have help on figuring out the source of this issue and potential solutions.

=========================== Initializing pipeline ===========================
[2021-12-05 19:02:55,043] [INFO] Set up nlp object from config
[2021-12-05 19:02:55,049] [INFO] Pipeline: ['transformer', 'ner']
[2021-12-05 19:02:55,051] [INFO] Created vocabulary
[2021-12-05 19:02:55,052] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at roberta-base were not used when initializing RobertaModel: ['lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.dense.weight']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
/Users/masakieguchi/opt/anaconda3/envs/spacym1/lib/python3.9/site-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
  warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')
[2021-12-05 19:03:03,584] [INFO] Initialized pipeline components: ['transformer', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['transformer', 'ner']
ℹ Initial learn rate: 0.001
E    #       LOSS TRANS...  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE 
---  ------  -------------  --------  ------  ------  ------  ------
  0       0           0.00     61.67    0.00    0.00    0.00    0.00
  0     200         190.90   3070.96    0.00    0.00    0.00    0.00
  0     400         149.65   1465.70    0.00    0.00    0.00    0.00
  0     600         115.97   1676.03    0.00    0.00    0.00    0.00
  0     800          84.94   1437.86    0.00    0.00    0.00    0.00
  0    1000          71.95   1563.69    0.00    0.00    0.00    0.00
  0    1200          58.54   1394.03    0.00    0.00    0.00    0.00
  0    1400          50.30   1549.60    0.00    0.00    0.00    0.00
  1    1600          47.85   1590.71    0.00    0.00    0.00    0.00
✔ Saved pipeline to output directory
training/model-last

================================== evaluate ==================================
Running command: /Users/masakieguchi/opt/anaconda3/envs/spacym1/bin/python scripts/evaluate.py ./training/model-best ./corpus/dev.spacy --output ./metrics/ner_trf.json --gpu-id -1 --spans-key grit
ℹ Using CPU
/Users/masakieguchi/opt/anaconda3/envs/spacym1/lib/python3.9/site-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
  warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

================================== Results ==================================

TOK     -   
NER P   0.00
NER R   0.00
NER F   0.00
SPEED   131 


=============================== NER (per type) ===============================

              P      R      F
ARGM-ADV   0.00   0.00   0.00
ARGM-LOC   0.00   0.00   0.00
ARGM-TMP   0.00   0.00   0.00
ARGM-MNR   0.00   0.00   0.00
ARGM-NEG   0.00   0.00   0.00
ARGM-DIS   0.00   0.00   0.00
ARGM-EXT   0.00   0.00   0.00
ARGM-DIR   0.00   0.00   0.00
ARGM-GOL   0.00   0.00   0.00
ARGM-REC   0.00   0.00   0.00
ARGM-PRD   0.00   0.00   0.00
ARGM-ADJ   0.00   0.00   0.00

Spacy versions

spacy                     3.2.0                    pypi_0    pypi
spacy-alignments          0.8.4                    pypi_0    pypi
spacy-legacy              3.0.8                    pypi_0    pypi
spacy-loggers             1.0.1                    pypi_0    pypi
spacy-transformers        1.1.2                    pypi_0    pypi
thinc                     8.0.13                   pypi_0    pypi
tokenizers                0.10.3                   pypi_0    pypi
torch                     1.10.0                   pypi_0    pypi

Config file for Transformer

The following config file may contain errors, which results in the aforementioned 0s in the eval metrics. I appreciate any suggestions on how to make this work!!

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96


[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
upstream = "*"

[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"


[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 2000
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Config file for tok2vec (for comparison purpose)

This config worked fine. The performance was not ideal, but it successfully trained the model.
The F1 was 70.97.

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}
upstream = "*"

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 2000
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

Result of tok2vec model (To show it worked on tok2vec model)

================================== Results ==================================

TOK     -    
NER P   68.31
NER R   73.84
NER F   70.97
SPEED   18803


=============================== NER (per type) ===============================

               P       R       F
ARGM-NEG   77.95   98.07   86.86
ARGM-ADV   61.69   62.47   62.07
ARGM-DIR   55.71   22.03   31.58
ARGM-TMP   71.17   80.57   75.58
ARGM-LOC   49.53   34.19   40.46
ARGM-DIS   59.91   60.34   60.13
ARGM-MNR   60.53   72.71   66.06
ARGM-EXT   25.00    2.70    4.88
ARGM-CAU    0.00    0.00    0.00
ARGM-PRD    0.00    0.00    0.00
ARGM-ADJ    0.00    0.00    0.00
ARGM-GOL    0.00    0.00    0.00
ARGM-MOD    0.00    0.00    0.00

✔ Saved results to metrics/ner.json
(spacym1) masakieguchi@dyn-10-109-32-188 manner_adv_span2 % python -m spacy project run all
ℹ Running workflow 'all'

[polm]

warnings.warn('User provided device_type of \'cuda\', but CUDA is not available. Disabling')

It looks like your GPU isn't configured correctly. It's technically possible to train a Transformer on CPU, but it will take an extremely long time, and is not recommended. Are you training on CPU intentionally? I wouldn't expect it to train without error but produce meaningless results like this, but I wanted to check.

[egumasa]

Thank you for your follow-up question. The reason for this is I am on Mac OS, (more specifically M1 Mac mini), so I have not been able to figure out how to utilize the GPU yet. It's on my list to learn how to configure Tensorflow using metal and use it (but I think many NLP library is on Pytorch rather than Tensorflow, and Pytorch does not have M1 native alternative right now as I understand it correctly). I understand that Transformer is very slow on CPUs, but that's what I got so far.
I appreciate any suggestion :)

[polm]

Another thing - for the NER version, rather than working from the spancat project, have you tried using the quickstart to generate a config? Looking at your config, there are a few settings related to batching, learning rate, gradient, etc. that are different from the quickstart recommendations. It's not clear to me they would cause the specific issue you're seeing, but if nothing is wrong with the data the quickstart config should just work.

[egumasa]

Thanks for the further question!

That part, I got it from the ner.cfg under the SpanCategorizer project.
The following is the copy from that cfg file.

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

Should I swap this chunk to the version on Quickstart like below? Is this something I should try first?

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[polm]

I would definitely try the quickstart config first.

However, while I understand that GPU support on the M1 isn't ready yet, I really can't recommend training a transformer without a GPU. It should work, and if you get errors or something we can try to help, but if you don't have a GPU I think it makes more sense to try tweaking model parameters or changing the attributes tok2vec uses or something.

[egumasa]

I have been working on to make transformer work using the hardware I have. I was able to train the model. Due to timeline of the project, I was not able to identify the source of the issue, but I tried the default config as you suggested and updated the spacy version to spacy 3.2.1. It seems that these combination did the trick.
Using M1 mac mini (16GB of RAM) with miniforge (natively running on apple silicon and Thinc-Apple-Opt), the model was trained on CPU in about half a day (approx. 10 hours).

Thank you for your help again!!