Support for Ministral / Mistral models

[IntinteDAO]

Description of the bug:

I tried to customize the model building for Ministral (and maybe other Mistral models) myself, but I can't because I get errors, which may be something to do with the tokenizer (as I understand it).

It would be great if AI experts could provide support for Ministral, which has models for Edge solutions.

Actual vs expected behavior:

No response

Any other information you'd like to share?

No response

[sourcelite]

Hi @IntinteDAO, welcome here. Great to know you working on model building.
Maybe to help you further we need to debug the specific error you mentioned -

1. could you please share the exact error log? (Especially the part mentioning the tokenizer or specific op failures).
2. Im curious to know if you're targeting Android (NPU) or iOS (Metal)?

Let's see if we can help you further. Thanks!

[IntinteDAO]

Hi @IntinteDAO, welcome here. Great to know you working on model building. Maybe to help you further we need to debug the specific error you mentioned -

1. could you please share the exact error log? (Especially the part mentioning the tokenizer or specific op failures).

2. Im curious to know if you're targeting Android (NPU) or iOS (Metal)?

Let's see if we can help you further. Thanks!

First of all, I wanted to run it locally on the CPU :)

Generally, I can send it, but I must say that I am not an expert in this field. I have worked extensively with AI (Gemini :P), but we were unable to accomplish this.

Sometimes the text appeared, but it was “garbled,” e.g., repeating one word over and over again.

I reported this issue because, in my opinion, Mistral has potential, and Bielik, a Polish model specially trained on the Polish language, is based on it.

I'll try to send it soon, because I did it on the company server (since such a conversion requires a lot of RAM) and I don't have it locally. But this should be treated as a concept, not something that can be fixed with a single variable.

[github-actions]

Marking this issue as stale since it has been open for 7 days with no activity. This issue will be closed if no further activity occurs.

[sourcelite]

Hi @IntinteDAO, sry for delay in issue discussion. Great going with Bielik, but now that you said you cannot share the logs due to your org restrictions. Let me provide my primary root cause analysis.

"garbled" text and repeating words

Issue would be - repetition curse decoding and KV cache - when a model is repeating one word forever - the the the the the the the the the, this is not a bug, but often a desync in KV-Cache, or even the sampling issue.

In local cpu, the sampling parameter - temp, top p if not set correctly then the model can get stuck in the probability loop. KV_Cache desynchronous - if litert_torch conversion isn't perfectly map then the rotary positional embeddings (ROPE), the model will forget where it is in the sentence, causing to loop the lastly known state.

For "Garbled Text" - nowadays ,ministral like. models use V3-tekken tokeniser - departure from std Llama/Mistral v1 tokenisers. So the IDs will not match if you use a std tokeniser config for decoding purpose of the Tekken-based model, so I urge you to check this as well.

Still need to check feasible way out of this issue... Thanks

[sourcelite]

Our case - Bielik and Mistral use a specific control tokens like INST and /INST -> the model building step in litert-torch will need a specific custom template to handle them.

I see this as a architect level conceptual support issue. I suggest you -> build a mistral v3 recipe in litert-torch that specifically will handle -

1. Tied embeddings ministral often ties input/output embeddings to sabe RAM.
2. Tekken tokeniser mapping: ensure that .litertlm bundle will know its looking at Tiktoken IDS.

Final guess based suggestion - keep your tie_word_embeddings == true in your config related file (mostly config.json). hope this info will help you.

[fervidoctor-netizen]

Sorry for waiting, I am on the office (where I have machines that can handle conversion), but I can't login to my account

convert_to_tflite.py
model.py
model_precise.py
pack_model.py
verify_ministral.py
vision_encoder.py