Include a Tagger/Parser in a NER transformer-based pipeline with no training data

[oliviercwa]

I am training a NER using a transformer architecture.

In the pipeline, I would like to have a Tagger and DependencyParser to sentencize the text, however I don't want to train these two components since I do not have any training date for the tagger and Parser.

• If I include the Tagger and Parser in the training pipeline and "freeze" them, I am getting a warning that their performance might be degraded, which makes sense.
• If I don't freeze them, I need training data.
• If I include trained component from another model (e.g. en_core_web_md), I am getting a warning, which again makes sense since the vectors are different: "Sourced component 'tagger' may not work as expected: source vectors are not identical to current pipeline vectors."

Is there a way to add a Tagger and Parser in a Transformer-based pipeline without having to train them ?

[thomashacker]

Hey oliviercwa ✨,
your first approach of freezing the components is the right choice here. However, it would be helpful if you could also send us your whole config file.

The performance decreases if your NER uses the same embedding layer as the other components. You can add a new layer just for the NER (with slower runtime as a trade-off), to maintain the performance of the frozen components.

[oliviercwa]

Thanks. Here is the full config.
However it seems that the NER is not pulling from the transformer vectors and instead is using the vectors from en_core_web_md

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "xx"
pipeline = ["tagger", "parser", "transformer", "ner"]
batch_size = 128
disabled = ["tensor2attr", "post-zendoc-ner-util"]
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.tagger]
source = "en_core_web_md"

[components.parser]
source = "en_core_web_md"

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v1"
name = "xlm-roberta-base"

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 510
stride = 100

[components.transformer.model.tokenizer_config]
use_fast = true

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = ["tagger", "parser"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

[thomashacker]

Thanks!

I see you're sourcing the parser and tagger from en_core_web_md, but you also need to source the tok2vec component. Your pipeline should look like this pipeline = ["tok2vec","tagger", "parser", "transformer", "ner"]

You can source the tok2vec the same way as you did with the other components

[components.tok2vec]
component = "tok2vec"
source = "en_core_web_md"

And don't forget to also include it in the frozen_components.

frozen_components = ["tok2vec","tagger", "parser"]

I'm a bit confused about your entries in disabled = ["tensor2attr", "post-zendoc-ner-util"]. In general, disabled should contain components from the pipeline