Support for Hebrew

[Hilla-Merhav]

Hello,
I am interested in adding full support for Hebrew in spaCy. After searching through the spaCy Discussions forum, I found an older thread discussing this same issue.
Recently a new UD corpus for Hebrew has been published and is licensed under CC BY-SA 4.0. However, I'm uncertain if this corpus can be used with spaCy due to license incompatibility. Can someone provide clarification on this matter?

[justus-saul]

Most of the models are trained on UD corpora and most of them have CC BY-SA 4.0. So it should be no issue.
Afaik license problems like "should not be used commercially" would be more a problem for the person wanting to use that model and not for Spacy.

[adrianeboyd]

Here are the slightly out-of-date basics for adding a new language to the trained pipelines: #3056

We look for corpora with licenses that allow commercial use that include annotations for: POS tags, lemmas, dependency parses, and named entities. Most languages use a UD corpus for POS+lemmas+dependencies and a separate NER corpus, but sometimes there's one corpus with all annotation layers. Typically NER corpora are more difficult to find with appropriate licenses, and Hebrew would probably need a third-party lemmatizer.

Hebrew in particular may be difficult because of spacy's limited ability to handle multi-word tokens from UD corpora and because spacy's built-in components do not perform well on Hebrew, see e.g. https://aclanthology.org/D19-3044.pdf.

[adrianeboyd]

I think the main initial concern is the segmentation. With the current built-in components, spacy can only easily support merged multi-word tokens. It's good that the tokenization doesn't insert convert morphemes (the spacy Doc object doesn't support this at all), but I still don't think the rule-based tokenizer is going to be able to handle this segmentation.

The experimental trainable tokenizers were mainly developed for the UD benchmark. They're a bit hacky and inefficient, and we're not planning to move them into the core spacy library.

For some languages spacy uses third-party segmentation libraries. Are there any existing python Hebrew segmentation libraries that would work for this corpus? Maybe https://github.com/amir-zeldes/RFTokenizer?

The trainable edit tree lemmatizer is a built-in spacy component, so if it works well enough that would be fine.

[Hilla-Merhav]

Thank you very much for your help, I am currently searching for the optimal solution for the Hebrew segmentation libraries matter. I am aware that an up-to-date version of the RFTokenizer is available, and I will continue my research on this topic.
In addition to a tokenizer, are there any other language-specific third-party components that you suggest using, instead of the built-in spacy component?
You previously mentioned a third-party lemmatizer. Are there any languages supported by spaCy that employ an external lemmatizer?

[adrianeboyd]

Russian and Ukrainian use pymorphy3 and Japanese uses sudachipy.

But the built-in edit tree lemmatizer may work fine and it would make sense to test it first before considering third-party libraries.

[Hilla-Merhav]

I think I have found a potentially suitable solution for a third-party segmentation library in Hebrew: https://github.com/amir-zeldes/HebPipe
But I would greatly appreciate your advice on it.

[adrianeboyd]

HebPipe looks fine for research purposes, but it looks way too heavy as a dependency just for segmentation and we wouldn't want to have a dependency that has such restrictive torch pins because it wouldn't work in practice for many users. Even having spacy suggest a default tokenizer that requires torch is a bit iffy from our side.

The segmentation models look like they may just be RFTokenizer models underneath?

I also couldn't easily figure out the details about training corpora and licensing for the provided models?

[Hilla-Merhav]

Hello again @adrianeboyd

I have some questions regarding annotated corpora for NER in Hebrew. It is mentioned here that the entity recognizer identifies non-overlapping labelled spans of tokens. I'm a bit unsure about what this implies exactly - does it mean that the entity recognizer doesn't identify nested entities – entity within a longer entity? Should I be concerned if the NER corpus I choose contains nesting?

If you have any additional insights or characteristics that I should consider before selecting a suitable NER corpus, I would greatly appreciate it!

On a related note, based on our previous correspondence, I learned about the importance of avoiding corpora with annotation schemes involving an addition of covert morphemes. Could you please confirm whether this concern is limited to tokenization and UD corpora, or if it also applies to NER corpora? Does it matter if my chosen NER corpus involves adding covert morphemes (as part of the NER annotation)? Thank you for your assistance in clarifying these points.

[adrianeboyd]

The main thing is that you need to be able to align the NER annotation to the tokens produced by the tokenizer, which are not going to include covert morphemes. The UD corpus annotation and the NER corpus annotation both need to be able to be aligned to the same tokenization.