Reverse lemmatization

[joakimwar]

For any given word, I would like to find all the inflected forms of that word. So I basically want to run the lemmatizing in reverse. E.g. if I input "drive" I would like a list of "driving, drove, driven, drives." Can I use spaCy to do this? Has anyone done something similar before?

edit: Languages I'm interested in are Norwegian and Swedish. Mostly Norwegian right now.

[adrianeboyd]

The focus for spacy is on analysis, not on generation, so the core spacy library doesn't offer this, but there are some third-party libraries that support this through custom extensions. I haven't used it myself, but one that looks promising for English in particular is lemminflect: https://spacy.io/universe/project/lemminflect

[joakimwar]

Thanks!

[DuyguA]

Which language is it? If you provide the language I think more specific clues will come 😉 Different morphologies are treate differently, depending on the language you might need an FST or just get away with a dictionary based solution.

[joakimwar]

Norwegian for now. Swedish in the future maybe. What is an FST?

edit: Finite-State Transducers? Never heard of or used that before, I will try and check it out. Thanks :)

[DuyguA]

OK, afaik one has to inflect nouns for gender and definiteness; adjectives also for gender, comparative/superlative and more features and verbs are not inflected much. I think each root can generate 4-6 word forms, hence number of linflected words is finite. A dictionary lookup would work in this case.

Here's the Norwegian lemma data
https://raw.githubusercontent.com/explosion/spacy-lookups-data/master/spacy_lookups_data/data/nb_lemma_exc.json . Each entry is a word/lemma pair:

"aldersbestemt": ["aldersbestemme"],
"aldersbestemte": ["aldersbestemme"],
"aldersbestemmende": ["aldersbestemme"]

What you can do is go over the dictionary and process a bit. For each value, accumulate the keys in a list, so the above pairs become:

alderbestemme: [alderbestemmende, alderbestemte, alderbestemt]

If some words are not in this lookup list, you can add some endings by predefined rules as well.

You don't need FST for Norwegian, dictionary lookup would work 😉

[joakimwar]

Thank you very much! I've done the accumulation of keys from nb_lemma_exc.json and it's working pretty well. I'm not sure how to add endings using rules though. Take the rules for lemmatizing nouns in Norwegian as an example

    "noun": [
        ["en", "e"],
        ["a", "e"],
        ["et", ""],
        ["er", "e"],
        ["ene", "e"]
    ]

I'm thinking that if I try to take every word that ends in 'e' and replace 'e' with 'en', 'a', 'er', and 'ene', I will get too many nonsense words. And I can' just add 'et' to the end of every word, which would be the reverse of ["et", ""]. I guess I can try and see, but I suspect a lot of nonsense from that 😄

I guess I could check every potential inflected form against a dictionary to see if it's a real word 🤔

It would be interesting to know how often the rules are actually used during lemmatization vs how often the word is found in the lemma_exc table. It would give an idea of how much I am missing by not having rules to fall back on.

[DuyguA]

Endings are for the words that doesn't exist in the lookup table. It's just a suggestion though, if you have a way of predicting the gender for an example, then you can add endings and generate new words. For an example , let's say Google doesn't exist in your lookup table so you have to generate some word forms yourself. Then you'd generate 5 word forms:

Googlen
Googla
Googlet
Googler
Googlene

It's just an idea though, if the dictionary lookup functions well you can just skip it. I hope word forms above of Google is good looking enough 😁