Support of Uyghur language

[dream-1ab]

Hello, I'm in China and we are using Uyghur language in XinJiang, (we are living in the same country) and I have no powerful hardware and environment so I cannot train (or fine tune) your model, and I have no training/fine tunning LLM experience at all, Uyghur is a language using by 10 million people in XinJiang today, we need QWen to support our language because we live in the same country and Qwen is a powerful model, I can provide all the dataset of my language that I have and also I can help your guys to improve QWen if needed.

Calude sonnet 3.5 and ChatGpt4 good support for my language but we cannot use it because you know the reason.
I need to finetune/train QWen with my language at least if your guys not interested and can provide the instructions. (because QWen is based on llama may I find help with llama I think.)

but that is cool if QWen officially supports my language. I cannot wait to use QWen in my project.
thank you.

[robbiemu]

I am not with qwen team, but this post stood out because of some recent quantization Ive been doing for a model focused specifically on European language support.

do you have a quality dataset that can be used for this? And what size models are you talking about being able to run inference? Typically, you want roughly 500 samples per billion parameters just for the imatrix you would use to quantize the model as a rough rule of thumb. For training, its much, much larger (for example, the multilingual Salamandra models use 33TB of text data for 35 languages ..so about 1TB of raw text per language, in a 2/7/40b model).

I figure XinJiang must have some university building such a dataset, if it is not already built. But the fact that Clause and chatGPT already have support suggests some university there has already furnished this sort of data.

[dream-1ab]

I have 1GB of raw text data (news, discuss), I know this is not sufficient for training a model, Calude 3.5 sonnet and ChatGpt4o (especially Claude) have great support for my language.

I have no strong hardware and finance environment so I mean is 7B model I think.
few GB of text data is not sufficient for from scratch but I think it may sufficient for pre-trained model to learn grammar usage of a language so knowledges can be transferrable from another language.
if there was a hope we can prepare necessary dataset if needed.

[robbiemu]

For fine tuning, the dataset has to come in a specific format, like Question and Answer (there are variants that are used for different fine tuning approaches, but QA is typical). I found a very small dataset that aparente still works well enough for it to have been published on HF. But I also have done a little research and found that there are some challenges that Uyghur faces, especially related to multiple spellings of words/names etc. I'm unclear if the tiny dataset I can find, which has no write up on it, was nonetheless adjusted for this purpose. It would be best to get a dataset from a known and motivated source (like the universities I mentioned earlier), I think. It is possible that even with the small fine tuning set I can find, we can still find a good outcome with your data used in "continuous pre training" in combination with the fine tune dataset.
Most research that I am reading right now on Uyghur lang has moved on from simple inclusion in llms to SA and image to text. This must be a solved problem, I'm just not familiar with the proper dataset to use for this purpose. I'd be excited to get to participate in fine tuning and/or benchmarking the results to optimize the process, if this does start off in the right direction! I'll do a bit more research, to see what's else I can find.

[dream-1ab]

thank you for your reply
claude3.5 sonnet have great support of my language, grammar, understanding, responding whatever it is claude 3.5 sonnet is perfect.
because of this there is possibility to train fine tune dataset translated by claude3.5 sonnet from English or other languages?

[phly95]

I wonder how the llama models do with the Uyghur language? Might be worth testing out.

[phly95]

Llama 3.1 8b is what you should try.

[dark2luminosity]

llama 2.5 7b is the model I have tested, basically it can understand what I'm saying but cannot respond properly with my language.

how large Uyghur data do you have ?

[dream-1ab]

approximately 1.2GB in total.

[dark2luminosity]

Can you share this dataset?

[avary]

@dream-1ab please share the dataset