Tamil models on spaCy

[koaning]

I'm starting a thread here so that a discussion moves from Twitter to Github.

It seems that the goal is to train a Tamil model so that others may use it. I do not speak Tamil, but since spaCy does have a tokenizer for it (see the language guide) I'd argue that nothing is stopping anyone from making their own model. One might even go as far as pre-populating some vectors. Though I cannot gauge the quality; both fasttext and bpemb seem to support it.

The goal is to have this be a place for discussions on how to train a Tamil model. I personally think it’s better to worry about having a representative dataset first. It's very hard to make a meaningful general model so I'm inclined to first focus on a specific corpus that might yield a model that can be re-used. This is a personal gut feeling and I do not speak Tamil, though so I'll gladly hear it if other folks have suggestions.

[AswiN-7]

dataset:
https://github.com/UniversalDependencies/UD_Tamil-TTB

code done:
simple tokenizer, lemmatizer, pos tagger
https://github.com/AswiN-7/tamilNLP

question:
how to add this things spacy model

[adrianeboyd]

Hi, a good example for setting up a spacy pipeline for a UD corpus is here: https://github.com/explosion/projects/tree/v3/pipelines/tagger_parser_ud

python -m spacy project clone pipelines/tagger_parser_ud

For working with spacy projects: https://spacy.io/usage/projects

Be aware that this particular corpus is relatively small, so you may not see very good results, especially for the parser.

[AswiN-7]

hi

     python -m spacy pipelines/tagger_parser_ud

using the above command i cloned the project and for downloading assets changed values in project.yml file

     python -m spacy project assets

after running the above command assets are downloaded

along with this

this error is coming and i cant able to run

   python -m spacy run preprocess

Anybody know what to do about this??

[polm]

Please don't share screenshots of code or errors or file trees, just copy and paste text or terminal output. It is very hard to read your screenshots.

It looks like you just have a permissions error. Maybe there is a better way to solve it on Windows but I think it should go away if you run as Administrator.

Note that many of the example projects have only been tested on Linux/OSX and may have issues on Windows.

[AswiN-7]

now i'm shifted to ubuntu

       python -m spacy project assets

       python3 -m spacy project run preprocess

       ✘ Missing dependency specified by command 'preprocess': assets/UD_Tamil-TTB/ta_ttb-ud--test.conllu
       Maybe you forgot to run the 'project assets' command or a previous step?

what to do?

[polm]

You should be using the same python throughout your code, whether that's python or python3. (I'm not sure if those are the same on Ubuntu.)

Does the indicated filename exist? Do you not have a typo in the filename, like with the -- part?