Adding Support for Hebrew

[eliorc]

Hi, I've already read the Adding models for new languages master thread and I couldn't find a straightforward guide for adding a new language.

To be honest, adding Hebrew to spaCy is an initiative I've tried to take before, but that time (few years ago) the language resources were very poor.
Lately there have been advancement (models for PoS, language models etc.) which are created by the community and also some initiatives that are supported (also econimically) by the goverment which makes adding Hebrew to spaCy much more feasible.

My question is, where can I find a documented step by step guide for adding a new language from scratch?

(I am aware of the he section in the languages)

[polm]

Sorry, we don't have a better guide than that thread.

What data do you have?

Since the basic tokenizer support for Hebrew is already in spaCy, if you have CoNLL format data with tagger and parser annotations, you should be able to use spacy convert and spacy train to train a model. That would be a good first step.

[polm]

The CoNLL dataset you linked to has a BY-NC-SA license, which means that it cannot be used in "commercial" contexts. spaCy itself is under the MIT license and can be used for any purpose, and many of our users use it in professional contexts. For that reason we prefer datasets without a "non-commercial use" restriction.

[eliorc]

Just updating, I'm doing my best efforts to get my hands on a commercial CoNLL dataset with as much features as possible as well as NER data.

[polm]

I think we are on the same page, but to be perfectly clear, we do not require a "commercial dataset", just a dataset that is OK for commercial use.

I would use "commercial dataset" to describe a dataset that is only available for a fee. A dataset does not have to be commercial in that sense to be OK for commercial use.

[eliorc]

Just to make sure, data under the Creative Commons Attribution 4.0 International Public License is fine for spaCy purposes?

[polm]

Yes, CC BY should be fine.

[thewh1teagle]

@eliorc
Have you made any progress with it?

[eliorc]

Nope, I don't have enough time to put into this, so I've given up

[thewh1teagle]

I just started to learn AI with python (I already have a lot of experience in programming)
And I'm very interested in that project and any project related to NLP with hebrew.
I understand that if I want to make a project for that I must create word embedding for hebrew, And I think that's what you wanted to do?
Do you mind help me with that?

[eliorc]

Sorry, as I said I don't have the availability for that