Include Lemmatizer into a (trained) Transformer model

[maxtrem]

Hi,

I'm struggling to include a lemmatizer in into my Swedish transformer model (new spacy 3.0 nightly). I'm training the model via python -m spacy train config.cfg. The config file I'm using is basically the default transformer config created via:

python -m spacy init config -p tagger,morphologizer,parser -l sv config.cfg

I removed the line lookups = null in initialize and added a initialize.lookups. So the last part of the config basically looks like this (rest is default):

[initialize]
vectors = null
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null

[initialize.lookups]
@misc = "spacy.LookupsDataLoader.v1"
lang = "sv"
tables = ["lemma_rules"]

During the training I get the message: Added vocab lookups: lemma_rules

But the final model does not contain a lemmatizer and the lemma attribute still remains empty.

Are there any examples on how to include a lemmatizer into a transformer model? I looked through all documentation segements I found about this. But it still remains unclear to me how include the lemmatizer into the model/package.

A practical example how to include the lemmatizer would be great!

Thanks in advance!

[adrianeboyd]

If you're starting with a new config, just add lemmatizer to -p at some point after the morphologizer. Then you can adjust the mode and lookups in the config as necessary before training. (The lemmatizer itself is skipped during training because there's no model to train.)

If you have a trained model, you can install spacy-lookups-data, load the model, and run:

nlp.add_pipe("lemmatizer").initialize()
nlp.to_disk("/path/to/model")

If you don't want the default lemmatizer config (for Swedish is may be the (untested?) rule-based lemmatizer, and maybe you want the lookup lemmatizer instead, for example), you can provide options to add_pipe as shown in the sidebar examples here:

https://nightly.spacy.io/api/lemmatizer

Edited to add initialize() to load tables.

[maxtrem]

That worked out of the box, thanks a lot!

If you don't want the default lemmatizer config (for Swedish is may be the (untested?) rule-based lemmatizer, and maybe you want the lookup lemmatizer instead, for example), you can provide options to add_pipe as shown in the sidebar examples here:
https://nightly.spacy.io/api/lemmatizer

Is there some way to include my own rules or lookup mappings, other than those already in spacy-lookups-data, right from the beginning?
For example by somehow specifying the location of an json file in the config?

[adrianeboyd]

You can use any function that returns the data in the right format (as a Lookups object) and use it in place of spacy.LookupsDataLoader.v1. You provide the code for the registered function using the --code argument.

spaCy/spacy/language.py

Lines 92 to 96 in dc816bb

	@registry.misc("spacy.LookupsDataLoader.v1")
	def load_lookups_data(lang, tables):
	util.logger.debug(f"Loading lookups from spacy-lookups-data: {tables}")
	lookups = load_lookups(lang=lang, tables=tables)
	return lookups

[maxtrem]

@adrianeboyd
That looks really interesting! The config file makes a lot more sense to me now. :)

Thank you for your help!

[maxtrem]

I tried to write a my own registered function together with --code. Unfortunately it doesn't work as expected.
My config.cfg looks like this (relevant expert):

[components.lemmatizer]
factory = "lemmatizer"
mode = "lookup"
model = null

[initialize.lookups]
@misc = "test"
lang = "sv"
tables = ["lemma_lookup"]

I'm training with the command:

python -m spacy train config.cfg --code code.py --paths.train corpus/sv_talbanken-ud-train.spacy --paths.dev corpus/sv_talbanken-ud-dev.spacy --gpu-id 0 -o output

My code.py looks like this:

from spacy.util import registry
from spacy.lookups import Lookups
import json 

d = {
  'lemma_rules': "sv_lemma_rules.json",
  'lemma_lookup': "sv_lemma_lookup.json",
}

@registry.misc('test')
def load_lookups_data(lang, tables):
    lookups = Lookups()
    for table in tables:
        with open(d[table]) as f:
          language_data = json.load(f)
        lookups.add_table(table, language_data)
    return lookups

Where I used the json files from spacy-lookups-data for testing. This itself doesn't look too bad, the lemmatizer is activated and runs during training:

Added vocab lookups: lemma_lookup
...
E    #       LOSS TRANS...  LOSS TAGGER  LOSS MORPH...  LOSS PARSER  TAG_ACC  POS_ACC  MORPH_ACC  LEMMA_ACC  DEP_UAS  DEP_LAS  SENTS_F  SCORE 
---  ------  -------------  -----------  -------------  -----------  -------  -------  ---------  ---------  -------  -------  -------  ------
  0       0        2630.44      1292.14        1295.62      1893.97     5.65     0.00      35.63      78.29    19.15     0.73     0.00    0.28

And I'm getting an accuracy of 78.29 for the lemmatizer. But when I now change lemma_lookup to the sv_lemma_rules.json:

d = {
  'lemma_rules': "sv_lemma_rules.json",
  'lemma_lookup': "sv_lemma_rules.json",
}

Where I would expect an error I'm getting exactly the same accuracy of 78.29. I also tried another json with different lookups, but still I'm getting an accuracy of 78.29. So my custom load_lookups_data does not really have an effect.

When replacing the json-file string by an empty string:

d = {
  'lemma_rules': "sv_lemma_rules.json",
  'lemma_lookup': "",
}

Then I'm getting an error message:

✘ Can't construct config: calling registry function (load_lookups_data)
failed
test   [Errno 2] No such file or directory: ''

So I suppose the lookup function is beeing called with tables = ["lemma_lookup"], but for some reason it will still use the original data from spacy-lookups-data.

What is wrong with my custom load_lookups_data registration? How can I get the training script to use my own data?

Thanks!

[adrianeboyd]

If you're loading the lemmatizer with add_pipe into the already trained model, I missed a step in my description above. The tables are loaded by the one-time initialize method, so you would need:

nlp.add_pipe("lemmatizer").initialize()

If the lemmatizer is in your config, the initialize step is run when the model is initialized before training. You need to change the mode to rule to have it use the rules instead of the lookup table, is that the problem? (The v2 lemmatizer would switch by default if rules for available, but for v3 this is now explicit and you have to set the mode that you want.)

[maxtrem]

@adrianeboyd Thanks for your answer!

You need to change the mode to rule to have it use the rules instead of the lookup table, is that the problem?

No, that is not the problem. This example is just to show that it doesn't make any difference what kind of json I load in my custom loader function.
When mode is set to lookup I will always get an accuracy of 78.29 and when mode is set to rule I will always get an accuracy of 62.41 no matter what kind of json file I use.

So my assumption is that when mode is set to lookup it will only consider the default json:
https://raw.githubusercontent.com/explosion/spacy-lookups-data/master/spacy_lookups_data/data/sv_lemma_lookup.json

While when mode is set to rule it only considers:
https://raw.githubusercontent.com/explosion/spacy-lookups-data/master/spacy_lookups_data/data/sv_lemma_rules.json

I just tested with an empty json file and I'm still getting 78.29 (with model=lookup):

from spacy.util import registry
from spacy.lookups import Lookups
import json 

d = {
  'lemma_rules': "sv_lemma_rules.json",
  'lemma_lookup': "some_empty_file.json", # empty json file without data
}

@registry.misc('test')
def load_lookups_data(lang, tables):
    lookups = Lookups()
    for table in tables:
        with open(d[table]) as f:
          language_data = json.load(f)
        lookups.add_table(table, language_data)
    return lookups

So the problem is, that not matter what the return value of my custom loader function is, it will only consider some default data, probably provided by spacy-lookups-data.

[adrianeboyd]

Ah, I see what might be happening. It's not [initialize.lookups], which is for the lookups tables in the vocab. You need a section for [initialize.components.lemmatizer]. Otherwise you are ending up with the default lemmatizer as defined in the Swedish settings.

[maxtrem]

I tried with:

[initialize.components.lemmatizer]
@misc = "test"
lang = "sv"
tables = ["lemma_lookup"]

But this resulted in an error:

✘ Config validation error
initialize -> components -> lemmatizer   value is not a valid dict

{'vectors': None, 'init_tok2vec': None, 'vocab_data': None, 'lookups': None, 'components': {'lemmatizer': {'@misc': 'test', 'lang': 'sv', 'tables': ['lemma_lookup']}}, 'tokenizer': {}}

I tried as well with:

[initialize.lemmatizer]
@misc = "test"
lang = "sv"
tables = ["lemma_lookup"]

This went through without error but yielded the same result as before. Is there something more I can try?

[adrianeboyd]

You want something like this. It's just the lookups that are loaded by the registered function:

[initialize.components]

[initialize.components.lemmatizer]

[initialize.components.lemmatizer.lookups]
@misc = "spacy.LookupsDataLoader.v1"
lang = "sv"
tables = ["lemma_lookup"]

Make sure this is working from spacy-lookups-data (check the tables in nlp.get_pipe("lemmatizer").lookups) and then try switching to your custom loader.

[maxtrem]

@adrianeboyd Yes, now it works!
Once again thanks for your help!