Combining pretrained NER and trained NER in single pipeline

[akki248]

Hi,

I'm trying to combine a pretrained NER model (en_ner_bc5cdr_md) with a custom trained NER model in a single pipeline. The query is regarding the process of combining the NER models.

Before doing this, I trained the custom NER model by explicitly providing the path to the vectors of the pretrained model in the config.cfg file.

This is shown in the snapshot below:

Also, before training the custom NER model, I see that the static word vectors of en_ner_bc5cdr_md are loaded successfully.

Now, in order to combine the NER models in a single pipeline, I do the following:

Load the pretrained scispacy model to predict the disease and chemical labels

ensemble_nlp_1 = spacy.load("en_ner_bc5cdr_md", exclude=['tagger','parser','lemmatizer','attribute_ruler'])

Load the trained custom NER model that predicts geography, product name and therapeutic area labels

trained_nlp = spacy.load("Trained_model_en_core_sci_md_vectors/")

Add the tok2vec layer of the trained custom NER after the tok2vec layer of the pretrained model

ensemble_nlp_1.add_pipe("tok2vec", source=trained_nlp, name="trained_prod_ta_geo_tok2vec", after="tok2vec")

Add the trained custom NER model after the pretrained NER model

ensemble_nlp_1.add_pipe("ner", source=trained_nlp, name="trained_ner_prod_ta_geo", after="ner")
print(ensemble_nlp_1.pipe_names)

After executing the above commands, I get no error regarding the tok2vec layers.
Then, I save the pipeline to disk, using:
ensemble_nlp_1.to_disk("Ensemble_1_NER")

And, this creates a folder with the pipeline components. The config file for this ensemble NER is shown below:

[paths]
vectors = "output/en_core_sci_md_vectors"
init_tok2vec = null
parser_tagger_path = "output/en_core_sci_md_parser_tagger/model-best"
dev_path = "assets/BC5CDR-IOB/devel.tsv"
train_path = "assets/BC5CDR-IOB/train.tsv"
vocab_path = "project_data/vocab_md.jsonl"
train = null
dev = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","trained_prod_ta_geo_tok2vec","ner","trained_ner_prod_ta_geo"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2Vec.v1"

[components.ner.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.ner.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v1"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.trained_ner_prod_ta_geo]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.trained_ner_prod_ta_geo.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.trained_ner_prod_ta_geo.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = 96
upstream = "*"

[components.trained_prod_ta_geo_tok2vec]
factory = "tok2vec"

[components.trained_prod_ta_geo_tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.trained_prod_ta_geo_tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 96
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.trained_prod_ta_geo_tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "specialized_ner_reader"
file_path = ${paths.dev_path}

[corpora.train]
@readers = "specialized_ner_reader"
file_path = ${paths.train_path}

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.2
accumulate_gradient = 1
patience = 0
max_epochs = 7
max_steps = 0
eval_frequency = 500
frozen_components = ["tok2vec","parser","tagger","attribute_ruler","lemmatizer"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_sequence.v1"
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 1
stop = 32
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = ${paths.vocab_path}
lookups = null
before_init = {"@callbacks":"replace_tokenizer"}
after_init = null

[initialize.components]

[initialize.tokenizer]

Does this process of combining two NER models and config.cfg file of the ensemble NER look appropriate? Do I have to make any changes here? Any suggestions/guidance is welcome.

The inference/prediction of all the labels - disease, chemical, product name, geography and therapeutic area, surprisingly works and the end goal is to deploy the ensemble NER using AWS SageMaker.

[polm]

Just a note since you have done this in multiple threads: please do not share screenshots of code, copy and paste the code as text instead. Screenshots of code are hard to read and don't allow us to copy/paste things to check in our own environments.

If you need to call out a particular part of the code, like with your arrows, you can add a comment that says XXX or something.

[Pandalei97]

As far as I can see, you need to pay attention to the tok2vec layers that your NER layers listen to.

Your trained_ner_prod_ta_geo component is listening to the en_ner_bc5cdr_md's tok2vec but not trained_prod_ta_geo_tok2vec's, since it will just search for the layer named tok2vecin your new pipeline.

You need to either replace the listenier when souring it or change the tok2vec that trained_ner_prod_ta_geo listens to by your own.
You can find more details in @adrianeboyd's answer.

[akki248]

Thanks.

Sorry, I actually get really confused in the config setup. Here, is what I have done after referring to the thread and your guidance:

# Load the pretrained scispacy model to predict the disease and chemical labels
ensemble_nlp_1 = spacy.load("en_ner_bc5cdr_md", exclude=['tagger','parser','lemmatizer','attribute_ruler'])

# Load the trained custom NER model that predicts geography, product name and therapeutic area labels
trained_nlp = spacy.load("Trained_model_en_core_sci_md_vectors_recall/model-best")
trained_nlp.replace_listeners("tok2vec", "ner", ["model.tok2vec"])

ensemble_nlp_1.add_pipe("tok2vec", source=trained_nlp, name="trained_prod_ta_geo_tok2vec", after="tok2vec")
ensemble_nlp_1.add_pipe("ner", source=trained_nlp, name="trained_ner_prod_ta_geo", after="ner")

print(ensemble_nlp_1.pipe_names)

The updated config file of the ensemble NER is:

[paths]
vectors = "output/en_core_sci_md_vectors"
init_tok2vec = null
parser_tagger_path = "output/en_core_sci_md_parser_tagger/model-best"
dev_path = "assets/BC5CDR-IOB/devel.tsv"
train_path = "assets/BC5CDR-IOB/train.tsv"
vocab_path = "project_data/vocab_md.jsonl"
train = null
dev = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["tok2vec","trained_prod_ta_geo_tok2vec","ner","trained_ner_prod_ta_geo"]
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
batch_size = 1000

[components]

[components.ner]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v1"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2Vec.v1"

[components.ner.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.ner.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v1"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v1"
width = 96
attrs = ["NORM","PREFIX","SUFFIX","SHAPE"]
rows = [5000,2500,2500,2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v1"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.trained_ner_prod_ta_geo]
factory = "ner"
moves = null
update_with_oracle_cut_size = 100

[components.trained_ner_prod_ta_geo.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.trained_ner_prod_ta_geo.model.tok2vec]
@architectures = "spacy.Tok2Vec.v2"

[components.trained_ner_prod_ta_geo.model.tok2vec.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 96
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.trained_ner_prod_ta_geo.model.tok2vec.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.trained_prod_ta_geo_tok2vec]
factory = "tok2vec"

[components.trained_prod_ta_geo_tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.trained_prod_ta_geo_tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = 96
attrs = ["ORTH","SHAPE"]
rows = [5000,2500]
include_static_vectors = true

[components.trained_prod_ta_geo_tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[corpora]

[corpora.dev]
@readers = "specialized_ner_reader"
file_path = ${paths.dev_path}

[corpora.train]
@readers = "specialized_ner_reader"
file_path = ${paths.train_path}

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.2
accumulate_gradient = 1
patience = 0
max_epochs = 7
max_steps = 0
eval_frequency = 500
frozen_components = ["tok2vec","parser","tagger","attribute_ruler","lemmatizer"]
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_sequence.v1"
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 1
stop = 32
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = true

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
ents_per_type = null
ents_f = 1.0
ents_p = 0.0
ents_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = ${paths.vocab_path}
lookups = null
before_init = {"@callbacks":"replace_tokenizer"}
after_init = null

[initialize.components]

[initialize.tokenizer]

Could you please have a look now if it's appropriate? Please note that I did not create a blank model.

I saw in the thread that you tried to do the same thing but after creating a new blank model initialised with the vectors. Is it possible for you to share the process or code as to how you did that? Any reason why you did that way?

[Pandalei97]

Hello, you can check your config file by using spacy debug config config.cfg
More details in: https://spacy.io/api/cli#debug

Whether you start from a pretrained model or a blank model, that's the same. It depends on your usage. In our project we have our own word vectors so I have to do it that way. I'm not able to share our code. The code that I posted in the threads are modified specifically for my questions.

In spacy 3, many components listen to a shared tok2vec layer by default. It depends on your own pipeline design choice to use shared or independent tok2vec. Just make sure when sourcing the components from the other pipeline, you need to pay attention to the tok2vec layer that it listens.

One point important, if you use 'replace_listeners', you won't need to add the tok2vec again in the pipeline, since it will be integrated directly in your ner model.

For me, it's enough to do:

# Load the pretrained scispacy model to predict the disease and chemical labels
ensemble_nlp_1 = spacy.load("en_ner_bc5cdr_md", exclude=['tagger','parser','lemmatizer','attribute_ruler'])

# Load the trained custom NER model that predicts geography, product name and therapeutic area labels
trained_nlp = spacy.load("Trained_model_en_core_sci_md_vectors_recall/model-best")
trained_nlp.replace_listeners("tok2vec", "ner", ["model.tok2vec"])

ensemble_nlp_1.add_pipe("ner", source=trained_nlp, name="trained_ner_prod_ta_geo", after="ner")

print(ensemble_nlp_1.pipe_names)

[akki248]

Hi!

Thanks a lot for the help and sorry for the delayed response. So, if I follow the code snippet provided by you, I get 3 components (as shown below):

# Load the pretrained scispacy model to predict the disease and chemical labels
ensemble_nlp_1 = spacy.load("en_ner_bc5cdr_md", exclude=['tagger','parser','lemmatizer','attribute_ruler'])

# Load the trained custom NER model that predicts geography, product name and therapeutic area labels
trained_nlp = spacy.load("Trained_model_en_core_sci_md_vectors_recall/model-best")
trained_nlp.replace_listeners("tok2vec", "ner", ["model.tok2vec"])

ensemble_nlp_1.add_pipe("ner", source=trained_nlp, name="trained_ner_prod_ta_geo", after="ner")

print(ensemble_nlp_1.pipe_names)

Now, if I create an ensemble the other way around i.e. load the trained model first and then add the pretrained model in the pipeline and replace the listeners, I get 4 components:

# Load the trained custom NER model that predicts geography, product name and therapeutic area labels
ensemble_nlp2_v1 = spacy.load("Trained_model_en_core_sci_md_vectors_recall/model-best")
ensemble_nlp2_v1.replace_listeners("tok2vec", "ner", ["model.tok2vec"])

# Load the pretrained scispacy model to predict the disease and chemical labels
pretrained_nlp = spacy.load("en_ner_bc5cdr_md", exclude=['tagger','parser','lemmatizer','attribute_ruler'])

ensemble_nlp2_v1.add_pipe("tok2vec", source=pretrained_nlp, name="pretrained_disease_chemical_tok2vec", before="tok2vec")
ensemble_nlp2_v1.add_pipe("ner", source=pretrained_nlp, name="pretrained_ner_disease_chemical", after="ner")

print(ensemble_nlp2_v1.pipe_names)

Is this alright? Shouldn't here also there be three components with the 'tok2vec' layer integrated into the 'ner' component of the trained custom NER model?

Thanks.

[polm]

It's not quite done but you might want to take a look at the new double NER example project in explosion/projects#59, which was partly inspired by this thread. It has examples of combining two NER models using the config or code.

[polm]

This has been released.

https://github.com/explosion/projects/tree/v3/tutorials/ner_double

[akki248]

OK. Thank you so much! :)