Train only NER component

[aakashb95]

Hi,
I have my .spacy files for NER training and I used the base config from the spacy training documentation.
How do I carry forward other components of en_core_web_sm/en_core_web_trf like pos_tagger, senter, etc, while only training the NER component?

Also, what other models apart from roberta-base can be used here?

[paths]
train = "train.spacy"
dev = "valid.spacy"

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.ner]
factory = "ner"
incorrect_spans_key = null
moves = null
scorer = {"@scorers":"spacy.ner_scorer.v1"}
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "roberta-base"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0
ents_per_type = null

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[ljvmiranda921]

How do I carry forward other components of en_core_web_sm/en_core_web_trf like pos_tagger, senter, etc, while only training the NER component?

You can source components from different models through the config.

Also, what other models apart from roberta-base can be used here?

For components.transformer.model, you can virtually try anything from the Huggingface hub. You can check more in the TransformerModel docs.

[aakashb95]

Thanks, I tried source_components but am facing the following issue:

custom_ner_nlp = spacy.load("model-best/")
custom_ner_nlp.pipe_names
>>> ['transformer', 'ner']

nlp = spacy.load("en_core_web_trf", exclude=['ner'])
nlp.pipe_names
>>> 
['transformer', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer']

Now I try to add the "ner" component from my custom pipeline to the "en_core_web_trf" and it completely mis-tags the entities:

NER tagging from my custom model:

NER tagging after adding "ner" to "en_core_web_trf":

What am I doing wrong?

[ljvmiranda921]

Hi @aakashb95 , you also need to include the transformer component. You can refer to the shared embedding layers doc for more information 😄

[aakashb95]

Thanks! That made a lot of sense and I included transformer and ner components from my custom model into the en_core_web_trf.

While this fixed the above problem, POS tagger misbehaved:

If my understanding of shared embedding layers is correct, pos-tag misbehaves probably because there is a mismatch between my custom transformer component and en_core_web_trf's transformer component due to training.

How do I fix this? I do not have training data for pos tagger.
Is it something to do with config file while training? if yes, could you please point me to a sample config file wherein only transformers and ner are trained and tagger, parser, sentencizer are all included BUT frozen?

[ljvmiranda921]

Hi @aakashb95 , thanks for waiting !

It's still possible to use the POS Tagger by including an independent "listener." You can achieve this by calling the replace_listeners function and providing it with the component to replace listeners for (in your case, the tagger). You can check how it's done in practice in the NER double project, when they tried to combine two components that were trained separately.