Customizing default construction of lemmas and norms

[kinghuang]

How are token lemmas and norms generated? I've changed norms on specific tokens in a pipeline component before, but I'd like to change them as a whole in my spaCy model in the following ways.

• Treat tokens that are all caps as lowercase for lemmatization (e.g., lemmatize CARS to car).
• Default norms to the lowercase form of the lemma.

I'm not sure how to globally influence the lemmas and norms.

[adrianeboyd]

Norms and lemmas are completely separate from each other. What do you want to do with norms and lemmas?

Norms are primarily used as input features for the models (all currency symbols are replaced with the norm $, in English British and American spellings for common words are mapped to the same form, etc.), while lemmas are typically just output and not used as model features (it's technically possible with some extra settings, but no components use LEMMA as a feature by default).

The norms need to be something that you can generate without any previous statistical components in the pipeline, so you could create norms from lookup lemmas, but not from rule-based lemmas, since you need the norms to predict the POS for use with the rule-based lemmatizer. But you could use a lookup lemmatizer table to set custom norms for all the lexemes in the table in your vocab before processing, or you could set token norms however you want (from a table, with a method) with a pipeline component than runs before the statistical components. The lexeme norms are stored in a table in the vocab, so if you want generated norms, you should set token norms in a pipeline component.

If you take a trained pipeline like en_core_web_sm and change the norms, the statistical components like the tagger and parser will not perform as well because this isn't the kind of input they were trained on.

Set a lexeme norm:

nlp.vocab["WORD"].norm_ = "word"

Set a token norm:

doc[0].norm_ = "word"

[kinghuang]

Thanks for the info. I might be approaching this the wrong way. What I'm struggling with is writing matchers for text with variable capitalization schemes.

For example, here's some text along with the lemma and norm forms of the tokens.

ORTH	NORM	LEMMA
RETAINER, VALVE SPRING	retainer, valve spring	RETAINER, VALVE spring
valves spring retainer	valves spring retainer	valve spring retainer
SL CNRL PRESSURE	sl cnrl pressure	sl CNRL pressure
WL CNRL WATER TRI	wl cnrl water tri	WL CNRL WATER TRI
WL CNRL watered tri	wl cnrl watered tri	WL CNRL water tri

To match on "valve", ideally I'd like to use a token pattern like {"LEMMA": "valve"}, which would cover the plural "tokens". But, the lemma form of VALVE is VALVE not valve. Because of the difference in capitalization, I've been writing patterns as {"NORM": {"IN": ["valve", "valves"]}}. Same for "water" in the above examples.

It's not obvious to me why the lemmatized form of VALVE is VALVE, yet SPRING is lemmatized to spring. Similarly, what determines that the lemmatized form of SL is sl, but WL is WL?

[adrianeboyd]

The capitalization in lemmas for the default English rule-based lemmatizer depends on the POS, and for texts like this the distinction between NOUN and PROPN by the tagger is not going to be good. A few options:

• if you don't care about the NOUN/PROPN distinction (it's fine if "Some Name Retainers" -> "retainer" for matching purposes),
  then you could add an attribute ruler rule that changes all PROPN to NOUN before running the lemmatizer and you'll get singular+lowercased
• add a small custom component that lowercases all lemmas (but things tagged as PROPN may still be plural, so you'd get "retainers" instead of "retainer")

[kinghuang]

Ah, I didn't know that NOUN/PROPN was the distinguishing factor in the lemma output I'm getting. I can definitely selectively adjust that in my docs. Thanks!