Merging of two different pipelines that use transformers

[danielvasic]

Hello,

I have some troubles that didn't occur in spaCy 2.3.0. I have tried to train two different pipelines one for tagging and parsing and other for tagging and NER. When I try to add parsing component from the first pipeline to the other the precision of the component drops down drastically I suppose because the transformer model is fine-tuned on different sets of weights than for the parsing component. I have tried to add transformer component from the second model to the first but that just drops the accuracy of the NER component. Is there any way to merge these components and still preserve the original accuracy.

Info about spaCy

• spaCy version: 3.0.0rc2
• Platform: Darwin-19.6.0-x86_64-i386-64bit
• Python version: 3.7.9
• Pipelines: hr_pipeline (0.0.1)

[svlandeg]

What exactly are the configurations of your pipelines, are you using Tok2Vec or Transformer listeners?
And how exactly are you combining the two pipelines?

In spaCy 2.3, each component would have its own Tok2Vec layer - there would be no multi-task learning and no interference. In spaCy 3, you have the possibility to define a Tok2Vec layer or transformer only once in the pipeline, and then use a listener to fetch the outputs in different components. This is documented here.

My guess is that somehow, after adding a component from another pipeline, that this component ends up "listening" to the wrong transformer/tok2vec component. This is why Tok2VecListener optionally takes an upstream_name to define the upstream Tok2Vec component to fetch the results from. It looks like TransformerListener does not expose this option to define an upstream name, but it probably should and it's an easy fix. I'll look into it. You should then be able to distinguish between the different Tok2Vec/Transformer layers in your pipeline by naming them differently (ofcourse you'll have to make sure to add these components to your final pipeline as well).

[badri-rutgers]

Is there a good write up on transformer listener vs tok2vc listener? When do u use which. Is there an example of a pipeline that uses transformer listener and how, if we need to update a component, use replace_listener(transformer). For example, a nlp pipeline that uses webtrf model uses tok2vec as I have seen.

[polm]

Hey, I think I addressed your questions about this here, but if not feel free to follow up there.

Note this Discussion dates to when v3 was still pre-release, and a lot of the details mentioned here are out of date because of that.

[danielvasic]

Hello Sofie, I'm using TransformerListener, I'm combining the pipelines by using add_pipe and with source from other pipeline. II suppose that I have to change the name of the transformer component from the parser to match upstream_name. Also I suppose that I need to add this parameter to components.parser.model.tok2vec section of the config.cfg file and I do not need to retrain the model again. Another possible solution could be if I understood correctly not to use TransformerListener just to train independent componen? I have started the training with only parser component without the listener by using independent Tok2VecTransformer.

[svlandeg]

Yes - I think you should be able to make this work with a little hacking to avoid retraining. First you add the transformer from the other pipeline and give it a new name with nlp.add_pipe(transformer, name="other_transformer", source=...). Then you fetch the component from the other pipeline that is trained on other_transformer, let's say it's the parser component.

I think some hack like this should work:

parser.model.get_ref("tok2vec").layers[0].upstream_name = "other_transformer"

Because layers[0] should be the TransformerListener if I'm not mistaken.

You might also have to reset the listeners of the corresponding components and call nlp._link_components() again to ensure the listeners are linked up correctly. (again, all of this is a bit of a dirty hack)

Ideally, yes, this would have been set correctly in the config files before training, but you'd need this PR: explosion/spacy-transformers#230

And yes - an entirely different solution is to have a separate Tok2VecTransformer for each component. This makes your components more independent, but doesn't allow to share transformer weights between components, and training will be slower. It depends on your use-case what will work best for you.

[danielvasic]

Thanks Sofie for the help, I have added the PR but have some error, I guess the config checker needs to be updated too

Config validation error
parser.model.tok2vec -> upstream   extra fields not permitted

Removing the upstream attribute from TransformerListener now produces:

Can't construct config: calling registry function (transformer_listener_tok2vec_v1) failed
spacy-transformers.TransformerListener.v1   name 'upstream' is not defined

Also after calling parser.model.get_ref("tok2vec").layers[0].upstream_name = "other_transformer" the error Model has no attribute upstream_name. Not tested jet but I will add parser component with its own Tok2Vec component.

[svlandeg]

Hm, I added a unit test to the PR and it seems to work just fine, cf this edit which shows how to change the config. The config checker uses the function declarations, so in principle I think the PR should be fine (but I could be wrong as I don't have enough information to reproduce your specific use-case).

Your first error extra fields not permitted seems to indicate that you're calling a function that doesn't expect upstream as argument. Can you paste the full parser.model.tok2vec block? Does it refer to the updated spacy-transformers.TransformerListener.v1 from the PR?

What's weird is that your second error name 'upstream' is not defined points towards the exact opposite problem: spacy-transformers.TransformerListener.v1 (from the PR) does expect an upstream argument and hasn't gotten one...

With respect to your final error "no attribute upstream_name" - can you double check the type of parser.model.get_ref("tok2vec").layers[0]? This should be a TransformerListener, but it doesn't seem to be in your case.

Anyway. Like I mentioned before, all of this is a bit hacky and it's not ideal to be changing the functions and configs between training and predictions. I was hoping to help you prevent retraining, but it looks like things have gotten more complex and retraining (either with the fix from the PR, or with entirely independent Tok2Vec/Transformer components) will be the best option by far. Then you should be able to combine the components as you originally described.

[danielvasic]

Dear Sofie,

Thank you very much I have changed the configuration, I was adding it to wrong place. I have loaded the configuration added the components and the accuracy is unchanged :-)

All the best,
Daniel

[KennethEnevoldsen]

So just in case other might be having this problem as well:

# get two models using a transformer component:
pip install https://huggingface.co/chcaa/da_dacy_small_ner_fine_grained/resolve/main/da_dacy_small_ner_fine_grained-any-py3-none-any.whl
spacy download da_core_news_trf

# combine then into one pipeline:
import spacy

nlp = spacy.load("da_core_news_trf", exclude="ner")
nlp_ner = spacy.load("da_dacy_small_ner_fine_grained")

nlp.add_pipe(factory_name="transformer", name="ner-transformer", source=nlp_ner)
comp = nlp.add_pipe(factory_name="ner", source=nlp_ner)
# make sure that is listens to the correct component
comp.tok2vec.layers[0].layers[0].upstream_name = "ner-transformer"
nlp._link_components() # unsure if this is needed?
doc = nlp("Ord som Aarhus og kl. 07:30 bliver i denne tekst annoteret")

# check that everything works as intended:
for ent in doc.ents:
    print(ent)
    print(ent.label_)