Amharic/Tigrinya Tokenizer help

[yosiasz]

Greetings,

I notice the East Asian languages have their own tokenizer. I think we might need one for Amharic and Tigrinya as the following test is failing for both languages

from __future__ import unicode_literals

import pytest


def test_noun_chunks_is_parsed_ti(ti_tokenizer):
    """Test that noun_chunks raises Value Error for 'ti' language if Doc is not parsed.
    To check this test, we're constructing a Doc
    with a new Vocab here and forcing is_parsed to 'False'
    to make sure the noun chunks don't run.
    """
    doc = ti_tokenizer("መራሕቲ ልዕሊ 100 ሃገራት ዓለም ኣብ ግብረ መልሲን ሓድሽ ስትራቴጂ")
    doc.is_parsed = False
    with pytest.raises(ValueError):
        list(doc.noun_chunks)

What do I need to either get these tests to pass or create custom tokenizers for these laguages

Thanks