Help needed for NER Training of de_core_news_lg

[NoraMathelemuse]

Hi,
I have been trying to use the trainable lemmentizer component for the de_core_news_lg spacy model but the config file is always failing with an error message reads:
raise ValueError(f"Can't read file: {location}")
ValueError: Can't read file: de_core_news_lg

I have located the error back to the functions.py file which is simultaneously being called when the pipeline is initialised. The code breaks on line 28 :
nlp.get_pipe("custom_lemmatizer").add_label("GENDER=Feminine")

Eventhough I have written a class which calls on the function add_label(), the add_label function is not being called in the line above. Receiving this error message:
Traceback (most recent call last):
File "", line 1, in
File "spacy/pipeline/trainable_pipe.pyx", line 205, in spacy.pipeline.trainable_pipe.TrainablePipe.add_label
NotImplementedError: [E931] Encountered Pipe subclass without Pipe.add_label method in component 'custom_lemmatizer'. If you want to use this method, make sure it's overwritten on the subclass.

Here is the whole functions.py file that might give a better idea of what I am doing wrong. I am simply stuck at this point and do not know how to progress to initialize the pipeline in order to get the labels added that i need to conitune with my work. If you know how I can overcome this I would really appreciate this.

import spacy
from spacy.language import Language
from spacy.pipeline import Lemmatizer


@Language.factory("custom_lemmatizer")
class CustomLemmatizer(Lemmatizer):
    def __init__(self, nlp, name, vocab, model=None):
        super().__init__(vocab, model)
        self.vocab = vocab

    def add_label(self, label):
        self.vocab.add(label)
        self.lookups.add_table(label, {})
config = spacy.util.load_config("./config.cfg")
nlp = spacy.blank("de", config=config)
nlp = spacy.load("de_core_news_lg")
is_gender = nlp.vocab.add_flag(lambda s: s.lower() in ["maskulinum", "feminine", "neutral"])
nlp.config["components"]["lemmatizer"] = {"factory":"trainable_lemmatizer", "vocab": nlp.vocab}
nlp.initialize()


nlp.add_pipe("trainable_lemmatizer", name="custom_lemmatizer")
nlp.get_pipe("custom_lemmatizer").add_label("GENDER=Feminine")
nlp.get_pipe("custom_lemmatizer").add_label("GENDER=Masculine")
nlp.get_pipe("custom_lemmatizer").add_label("GENDER=Neutral")   
# Add more labels as needed
examples = [
    ("Frau", {"GENDER": "Feminine"}),
    ("Mann", {"GENDER": "Masculine"}),
     ("Person", {"GENDER": "Neutral"})
]

[adrianeboyd]

This question includes many different components and features (NER, the lemmatizer, the trainable lemmatizer, a custom lemmatizer component, morphological features, vocab lookups, vocab flags), so it's a bit hard to tell what's going on.

Can you back up a step and explain what you're trying to accomplish overall?

(And just to double-check my instincts: I'd be curious about whether this code is AI-generated to some degree?)

[NoraMathelemuse]

yes, it is a lot. I am trying to create a ner which recognises the different genders in german words. To overcome a binary understanding of different relations between words but a diversified understanding. What I mean by this is that if a 'Mann' and 'Frau' are on two different ends of one vector then 'Person' needs to be somehow also in relation to those two words. I followed a tutorial for training an NER, which then led me to the trainable lemmatizer, I made use of both the Spacy documentation, writing my own code and then asked for some help from copilot because I got stuck.

If needed I could also attach my config file.

[adrianeboyd]

Okay, a lemmatizer isn't the right direction for this.

What kind of data and annotations are you working with?

• What is the kind of label or value that you want to assign to a word like "Person"?

• Do you have a fixed set of possible labels/values that could get assigned to each word (like each word gets one of the labels "A", "B", "C", or "unlabeled")? Or do you have numeric values?

• Do you have any existing word lists, where you could simply look up "Person" in order to give it a label?

• Do you have full texts where each individual word has a label, and where the label for "Person" might depend on the context in the sentence/text?

[NoraMathelemuse]

So, I want to assign the Gender of the different words as labels. I do have a non extensive word list of words which are all labelled according to Gender so Masculine, Feminine or Neutral.

-For Person it would be the label or value Neutral.

• Yes, correct it has a fixed set of possible labels, which should be either feminine, masculine, neutral or unlabelled for the words that will be trained. It can also be numeric with 1,2,3,4 but for clarity of code i chose to go with the actual words.
• I created different lists and df one list which has adjectives of the same type, like teacher but how it would be for each gender. Then I have a masculine df, feminine df and neutral df ( I created it for the purpose of the function which finds and replaces words according to gender assigned)
• I did scrape some articles with full sentences where these words are being used
  -> My question is if I have this label how do I get the NER to then accept this label and apply it on the whole dataset. This is essentially what I was trying to achieve in the line add_label. What would be another approach that gets me the same result to get unseen text labelled? Should I see it more as a classification issue instead of a named entity recognition problem?

[adrianeboyd]

Just as a note, in the context of NLP this terminology is pretty confusing vs. grammatical gender, which is even already annotated by de_core_news_lg as Gender under token.morph as a morphological feature. I might suggest trying to find a different name to use to refer to this. (I only understood what was going on originally because I googled your project.)

So to get back to the task, this isn't technically NER, but it's similar enough that it's probably easiest to get started with the NER-related tools in spacy. There are several built-in components in spacy plus tutorials and examples to help you get started. Plus there are a lot of manual annotation tools that support this kind of annotation.

If you have word lists, you can use a entity_ruler to create phrase patterns to label these words in new texts: https://spacy.io/usage/rule-based-matching#entityruler

If you want to create a training dataset to use to train a statistical ner model, you could start by automatically annotating with your patterns from the entity_ruler and then manually correct/extend the annotation with an external annotation tool. Explosion's annotation tool Prodigy is one example, but there are many options for NER annotation.

---

And just for completeness for someone coming across this in the future:

If I were going to do this as a token-level task, I would use an attribute_ruler to do the rule-based labeling to bootstrap a dataset and then train a morphologizer with a new feature (with a name other than Gender, e.g. SocietalGender=Neutral; I don't really know what to call this), and set the morphologizer to extend: True. Then the new morphologizer component could be added to de_core_news_lg and the annotation would show up as an additional feature in token.morph, so it wouldn't conflict with the existing ner annotation from the de_core_news_lg pipeline. But there are not as many external annotation tools that support morphological annotation in this format, so I'm not recommending it here.

[NoraMathelemuse]

Thank you that is really helpful actually.