What is needed to add additional languages?

[cccyberwolke]

This project is very interesting, i am wondering what i need to do to add an additional language to it, in my case i want to use it for Finnish.

Maybe, if we have a list of task needed for a new language, it might be possible to add more dynamically or in bulk.

[simjanos-dev]

I will update the project with further languages. I'll add Dutch, Italian and French tomorrow.

Sorry, I didn't expect so much interest, and I cannot add everything so quickly that people want. Also I don't have any documentation yet. It might take a while.

I think you need to modify these things to add a new language:

• Need to add the language to language selection dialog.
• Need to add the language to dictionary import UI as an option.
• Need to modify the database seeder to insert a DeepL dictionary (I've just realized I'll need to separate this file).
• Need to modify /tools/django_server/tokenizer/views.py. This handles the tokenization for every the text.
• Need to modify /docker/PythonDockerfile to install Spacy models for the new language
• Need to modify /config/langapp.php file and add language codes.
• Add a flag image file.

I don't know if there's more. It seems like a lot, but it is done pretty fast, if the language has spaces, and it is written from left to right and has latin alphabet.

I'll try my best to add more languages.

[systemcrash]

@cccyberwolke Perhaps take a look at 55dd704 - and see whether adding a language in your local copy works.

[systemcrash]

@simjanos-dev thanks for a very interesting tool.

If adding a language requires changes in several places, it probably requires those several places coalescing to one, so that language addition is less error prone.

PS please put a link to this repo from the github web page so it's easier to find 👓

[simjanos-dev]

@systemcrash

I think there are some things that could be better, like /config/langapp.php and language selection dialog. But there are parts that are completely disconnected, and cannot be pulled together. Like adding a flag image, adding the model for Spacy in the dockerfile and modifying the tokenizer python service.

I'll try to make commits in a way today that shows the process step by step.

I've added the github link yesterday. Someone posted only the overview page on HN without the github link. I've only posted it on reddit where I also added the github link, and didn't know it was shared at other places as well.

[simjanos-dev]

I've added support for Chinese, Dutch, Finnish, French, Italian and Korean.

Korean lemmas are separated with + symbols and I think Chinese will need a custom font type.

There are 7 numbered commits named "Adding language support 1". They document the steps on how to add additional languages.

You can add any language this way that Spacy supports and written left-to-right and top-to-bottom. I will add more in the near future.

[simjanos-dev]

So, adding languages is a bit more difficult than I thought so. There are multiple different language codes which has to be added for different parts to work: deepl, dict cc dictionary, jellyfin, an internat database naming short form code.

Some of these codes are scattered around the files, so I'll rewrite the code a bit and add them to the config/langapp.php file, which will be renamed to config/linguacafe.php. Every php and vue supposed to use these codes, and there shouldn't be any language lists/codes elsewhere, except the python and dockerfiles, which would be very difficult to unify.

I will add a bunch of languages that Spacy do not support using the Spacy multi language model, so they won't increase RAM usage and docker image size. I haven't tested it yet, but I think it should work for languages with spaces.

These languages that use the multi language model will not support lemmatisation and tagging genders for words. I will add more Spacy supported languages and may look for further tokeniser libraries in the future, but before that I want to find a way to enable/disable language support for each language, so the image size and RAM usage won't increase too much on slower PC-s.

First two languages will be Welsh and Czech.

[arvigeus]

I would really try that if it had Vietnamese. No dice, Spacy doesn’t have it.

[simjanos-dev]

I found this. Even if Spacy does not have it, we can add a separate library for it.

One of the next thing we are working on with @sergiolaverde0 is making languages installable, so we can add as many libraries as we like, and it won't increase the docker image size indefinitely for everyone. We were talking about it in #104.

I'll try to add Vitnamese, but it might take quite a while to get there. Eventually every possible language will be added.

[simjanos-dev]

Some update on adding more languages.

I've added English, Latin and Greek as supported languages. They will be out in the next update.

I can add more languages now that use the multi language model, or have no additional dependencies. We still have a problem with adding new languages that have large dependencies.

I will keep adding languages that I can.

[jacovanc]

I would also love to add Turkish. I can see Spacy doesn't support it out of the box unfortunately. Let me know if there's anything I can do to help.

[simjanos-dev]

@jacovanc

I found some, but they are large. We can add it after we have the language installation feature.

[gelbziesel]

Is there any update on this? Trying to install Croatian.

[sergiolaverde0]

Working on it but is quite big feature. We have settled on a plan and are currently executing it, I'm literally testing the changes on the Python side right now, but I can't promise a release date for it.

[simjanos-dev]

We are working on it currently. Even if we will have issues installing languages, Croatian is a small language model, and nothing blocks adding it. I just got sidetracked with other features in the latest update.

I'm planning on adding Catalan, Croatian, Danish, Lithuanian, Macedonian, Polish, Portuguese, Romanian, Slovenian next, if I can find a dictionary for them, or DeepL supports them. These are the missing small spacy language models that are missing and can be easily added.

If we succeed with the language install feature, I also want to add Thai, Vietnamese and Tagalog, because they also were requested.

But also the same, no promise on release dates.