What is "trainable" about the lemmatizer?

[BramVanroy]

I can see that either a rule-based or lookup lemmatizer can be implemented and that many models use a trained, rule-based lemmatizer. I cannot find what kind of rule-deduction is being done, however. I've been looking for the lemmatizer model here but could not find it.

Tl;dr: what kind of lemmatization strategy is used when using rule? Tree edit operations, string edit operations, etc.

[adrianeboyd]

Hmm, are you referring to having TrainablePipe as the base class in the docs? The "Trainable" flag next to that is correct: there is not currently a trainable lemmatizer.

In terms of the docs, the provided base class is a mistake as compared to the source (the base class is currently Pipe), but the Lemmatizer class has been designed so that it can be extended in the future without changing the API, so it has an unused Model parameter.

The current rule-based lemmatizers are all hand-written.

[BramVanroy]

That's what I thought. I remember that pronouns used to be lemmatised as -PRON-, although that does not seem to be rule based following the rules that I found in spacy-lookups-data.

So concretely, if I want to update the lemma rules and lookups according to my own dataset, how can I tell the config to use my custom files? From what I read in the documentation it seems that this has to be part of the lookups data package:

Additional lexeme and vocab data from spacy-lookups-data

Can I use a path? My guess is that I can build my custom Lookups and add the required tables with .add_table and then save the whole Lookup .to_disk. That file path could then be used in in initialize.lookups. Is that correct? The one doubt that I have is that I am not sure whether there are naming conventions for the tables. I.e., how does the trainer (lemmatizer) know which tables it must use in my custom Lookup?

I am sorry about all these questions but I really have tried looking for all this information. The training-config system is very customizable and that is a great thing, but precisely because of its abilities I find it hard to wrap my head around it when I want to do something with custom data/annotations/architectures. I hope you understand.

[adrianeboyd]

v3 gets rid of -PRON- as a lemma.

The lookups function in initialize should return a Lookups object, and internally you can load those tables however you want. Simple example:

spaCy/spacy/tests/pipeline/test_lemmatizer.py

Lines 11 to 25 in 8233359

	def nlp():
	@registry.misc("cope_lookups")
	def cope_lookups():
	lookups = Lookups()
	lookups.add_table("lemma_lookup", {"cope": "cope", "coped": "cope"})
	lookups.add_table("lemma_index", {"verb": ("cope", "cop")})
	lookups.add_table("lemma_exc", {"verb": {"coping": ("cope",)}})
	lookups.add_table("lemma_rules", {"verb": [["ing", ""]]})
	return lookups
	nlp = English()
	nlp.config["initialize"]["components"]["lemmatizer"] = {
	"lookups": {"@misc": "cope_lookups"}
	}
	return nlp

The lemmatizer has default table lists for each mode:

spaCy/spacy/pipeline/lemmatizer.py

Lines 40 to 53 in 8233359

	@classmethod
	def get_lookups_config(cls, mode: str) -> Tuple[List[str], List[str]]:
	"""Returns the lookups configuration settings for a given mode for use
	in Lemmatizer.load_lookups.
	mode (str): The lemmatizer mode.
	RETURNS (Tuple[List[str], List[str]]): The required and optional
	lookup tables for this mode.
	"""
	if mode == "lookup":
	return (["lemma_lookup"], [])
	elif mode == "rule":
	return (["lemma_rules"], ["lemma_exc", "lemma_index"])
	return ([], [])

If you write a custom lemmatizer, you can override it as needed (French rule uses lookup as a backoff):

spaCy/spacy/lang/fr/lemmatizer.py

Lines 17 to 23 in 8233359

	@classmethod
	def get_lookups_config(cls, mode: str) -> Tuple[List[str], List[str]]:
	if mode == "rule":
	required = ["lemma_lookup", "lemma_rules", "lemma_exc", "lemma_index"]
	return (required, [])
	else:
	return super().get_lookups_config(mode)

The Lemmatizer API is a bit clunky, but it was designed like this so you can add a new lemmatizer mode without implementing __init__, initialize, __call__, or serialization (well, unless you need data in a format other than tables), and it's intentionally all pure python. You should just need to add [mode]_lemmatize and get_lookups_config (only if necessary).

I'd recommend looking at the existing custom lemmatizers for more examples: spacy/lang/*/lemmatizer.py

[BramVanroy]

If I understand you correctly, the best way to go forward to modifying the lemmatizer would be to define a custom lemmatizer in Python (a separate python file). This file should then be referenced with --code during training and packaging so that nlp.after_pipeline_creation can replace the original lemmatizer with the custom lemmatizer? Or only changing the Lookups like so in a registered function.

     nlp.config["initialize"]["components"]["lemmatizer"] = { 
         "lookups": {"@misc": "cope_lookups"} 
     } 

The first one seems slows as I think it first loads the default English() lemmatizer (and additional tables) and then would overwrite them with my custom ones. The second one might be too restrictive if I want to have a more expansive lemmatizer, like the French one that you posted. Is there a better way to use custom Classes for components in the config?

[adrianeboyd]

You can implement a custom language with its own lemmatizer factory, or you can use a custom lemmatizer component with its own factory name. In both cases you'd typically use --code to provide the code while training. Alternatively, you can put your custom registered functions in a python package and install that in your venv. The functions would be available through the entry points.

You would not use after_creation for either of these methods. Just define the lemmatizer as a pipeline component and provide the right [components] and [initialize] blocks. If you're only customizing the tables but not the algorithm, you just need the custom loader in the [initialize] block, which doesn't need to be packaged with the model later. Code for custom components or languages would need to be packaged with the model, or as a model dependency in a separate package.

[BramVanroy]

I think I got it. I will try some things out. and get back to you if I encounter related questions to lemmatisation. Thanks.