Support new languages

[Irio]

Do the project maintainers have interest in supporting new languages?

The README suggests me that's the case, with:

MorphMan supports the following languages:

• languages with spaces: English, Russian, Spanish, Korean, Hindi, etc.

While a number of issues/pull requests without acceptance or much activity might suggest the opposite 1 2 3 4 5. In #43 (comment), @shanrauf briefly mentions the lack of a nice test suite – which I can't fully understand without the complete context.

Since I would be personally interested in bringing a better support for the German language, I'd like to ask a couple of questions to ensure I don't take more time and energy from the maintainers:

1. Is there an interest for supporting more languages?
2. Would you accept any new language, or would you rather prefer to focus on an specific group?
3. What are the requirements for accepting a new language?

Depending on the answers, I could start a separate issue for discussing the support for German and tracking its progress.

[rteabeault]

I am also interested in support for German and would be willing to help add it.

[rteabeault]

I have started working on spaCy support for morphman. My primary motivation is to get german support but I believe with spaCy we could get support for many other languages. spaCy currently has ML models trained for zh, da, nl, en, fr, de, el, it, ja, lt, mb, pl, pt, ro, and es. And it may be possible to add support for other languages that do not yet have models. Admittedly I am new to both Morphman and the NLP concepts that are needed. Currently I have a new Morphman preferences pane that allows for installing spaCy and am now adding support for installing different models. In general I think it may work to have a single spaCy Morphemizer. However, because I am not positive I will try to make it easy to add language specialized spaCy Morphemizers. I will update my progress in this issue.

[ianki]

PR #193 is another approach to using spaCy, using cross-process piping with the system installed Python.

Did you succeed in installing spaCy directly inside Anki's Python modules? That could be more efficient. If you're still working on it, I can review the patches.