Loading off-the-shelf morphologizer (fr_core_news_md) from config file in custom pipeline does not work

[AlexandreRozier]

How to reproduce the behaviour

I'm trying to build a custom French nlp pipeline, based on the morphologizer component of the fr_core_news_md model.
When building the nlp object programmatically, it seems to work fine, we can obtain valid pos_ tags.
However, when loading our custom object from a config file, the pos_ tags returned are invalid. My guess would be that we incorrectly configured the tok2vec listener of the morphologizer. Are we doing something unusual?

repro.py

import spacy
from spacy.lang.fr import French
from thinc.config import Config

# Works
fr_core_news_md = spacy.load('fr_core_news_md')
valid_nlp = French()
valid_nlp.add_pipe('tok2vec', source=fr_core_news_md)
valid_nlp.add_pipe('morphologizer', source=fr_core_news_md)
doc = valid_nlp('Bonjour, comment-allez vous ?')
print([t.pos_ for t in doc])  # ['NOUN', 'PUNCT', 'ADV', 'AUX', 'NOUN', 'VERB', 'PUNCT']


# Does not work
config = Config().from_disk("foo.config")
nlp_invalid = French.from_config(config)
nlp_invalid.initialize()
doc_invalid = nlp_invalid('Bonjour, comment-allez vous ?')
print([t.pos_ for t in doc_invalid])  # ['PROPN', 'PROPN', 'PROPN', 'PROPN', 'PROPN', 'PROPN', 'PROPN']

foo.config

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "fr"
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

pipeline = ["tok2vec","morphologizer"]

[components]

[components.tok2vec]
source = "fr_core_news_md"

[components.morphologizer]
source = "fr_core_news_md"

[initialize]
vectors = "fr_core_news_md"

Your Environment

## Info about spaCy

- **spaCy version:** 3.2.2
- **Platform:** Linux-5.11.0-46-generic-x86_64-with-glibc2.17
- **Python version:** 3.8.10

[ljvmiranda921]

Hi @AlexandreRozier , the reason why the code above doesn't work is because nlp.initialize() clears all the weights. In relation to this, you can check out the spacy assemble command and pass your config file. Under the hood, assemble does not initialize the components you've sourced.

You can also do this programmatically by disabling all the sourced components before calling the nlp.initialize() method:

with nlp.select_pipes(disable=[*sourced]):
    nlp.initialize()

Again, it may be easier to use the spacy assemble command, as it does this step for you. You can also check out its implementation to see how it creates the pipeline.

[AlexandreRozier]

Thanks a lot for your answer, this makes sense ! However, I have a follow-up question.
We are trying to use a custom transformer in our pipeline, a CamemBERT finetuned on our data. When loading our pipeline using French.from_config(config), running nlp('Bonjour') gives us the following error:

~/.miniconda3/envs/origami/lib/python3.7/site-packages/spacy_transformers/layers/transformer_model.py in forward(model, docs, is_train)
175 if "logger" in model.attrs:
176 log_gpu_memory(model.attrs["logger"], "begin forward")
--> 177 batch_encoding = huggingface_tokenize(tokenizer, [span.text for span in flat_spans])
178 wordpieces = WordpieceBatch.from_batch_encoding(batch_encoding)
179 if "logger" in model.attrs:

~/.miniconda3/envs/origami/lib/python3.7/site-packages/spacy_transformers/layers/transformer_model.py in huggingface_tokenize(tokenizer, texts)
276 return_tensors="np",
277 return_token_type_ids=None, # Sets to model default
--> 278 padding="longest",
279 )
280 token_data["input_texts"] = []

TypeError: 'NoneType' object is not callable

config:

[components.transformer]
factory = transformer

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "path-to-our-custom-camembert-folder"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]
output_attentions = true


[components.relation_extractor]
factory = "relation_extractor"

[components.relation_extractor.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.relation_extractor.tok2vec.pooling]
@layers = "reduce_mean.v1"

It seems that the tokenizer is incorreclty loaded in our model. That's why we were calling nlp.initialize, but as you said, this zeroes our transformer weights. Any ideas ? Maybe the config is wrong. Thank you for your help so far :)

[adrianeboyd]

It's only the components loaded with source that you don't want to initialize. A new transformer like this does need to be initialized to load in the transformer tokenizer+model and to get configured correctly for the components that are listening to it. nlp.initialize() doesn't reset the underlying transformer model weights.

[AlexandreRozier]

Hmm ok, I understand. Thanks a lot for your time, we'll try that & get back to you. Have a nice evening !

[AlexandreRozier]

I can confirm this works, thanks again !