How to use the tokenizer on Android/kotlin

[lenyturmel]

Hi all
this is this first time I use the ONNX framework
I developed a python code that exported (and quantized) the model "intfloat/multilingual-e5-small" to an ONNX model
My exportation directory contains the model and all the tokenizer configuration files:
│ quantized_model.onnx │ sentencepiece.bpe.model │ special_tokens_map.json │ tokenizer.json │ tokenizer_config.json

In my Kotlin project on Android Studio, I added the following dependencies:
implementation("com.microsoft.onnxruntime:onnxruntime-android:latest.release") implementation("com.microsoft.onnxruntime:onnxruntime-extensions-android:latest.release")

In my Kotlin class, I initiliaze the ORT session as follow:
val modelBytes = context.assets.open("quantized_model.onnx").readBytes() ortSession = ortEnv.createSession(modelBytes, OrtSession.SessionOptions())

Then I am looking for tokenizing the input sentence with a similar code to:
val tokenizationResult = ortSession.tokenize(sentence) val inputIds = tokenizationResult.inputIds val attentionMask = tokenizationResult.attentionMask

I know it cannot compile because there is no method "tokenize" does not exist.
But I was not able to find the good solution to use the ONNX libs to tokenize.
Could you please help ?
Do you have any information or code sample that could help me for getting started with the tokenization using ONNX?

Thanks you so much for your help

[Craigacp]

You can export the tokenizer as an ONNX model itself and then run it. There's an example of using an ONNX based tokenizer here - https://github.com/oracle/sd4j/blob/main/src/main/java/com/oracle/labs/mlrg/sd4j/TextEmbedder.java#L186, though I don't have any public code for exporting them in Java. This Python code shows how to stitch the tokenizer to the top of the embedding model - https://github.com/microsoft/onnxruntime-extensions/blob/main/onnxruntime_extensions/tools/add_pre_post_processing_to_model.py#L328.

[lenyturmel]

Hi,
thank you very much for your help.
And sorry for the delay (I was in vacation)

Thanks to the links you provide, the steps to use the toknenizer which has been converted to an ONNX model seems to be very clear.
However I am a little stuck for getting started in converting the tokenizer to an ONNX model.
My use case is to use the model intfloat/multilingual-e5-small to create embeddings from sentences.
This model seems to use a sentence piece tokenizer. When I saved the model and the config files of the tokenizer I can see the following files:

• sentencepiece.bpe.model
• special_tokens_map.json
• tokenizer.json
• tokenizer.onnx
• tokenizer_config.json

I do not know how to create an ONNX model from those files.
Do you have an information about that ? any useful links that could help me ?

Any help would be very appreciated, the tokenisation is the last piece of my puzzle to be able to run ma FAQ application on Android.
Thanks

[Craigacp]

Depending on your exact use case it might be better to stitch the tokenizer to the model and run it as a single ONNX graph. That's what the code I linked to from ONNX Runtime extensions does.

I wrote the code in Oracle's vector DB tooling which does this export step but I don't think that's available for external use, it's wrapped up in their python library. Otherwise I've tended to write the protobuf directly from Python or Java in a REPL which is also not a helpful solution to that problem. As the model is mostly a sentencepiece protobuf you can construct the ONNX protobuf by hand in Java/Kotlin, but that involves really understanding the ONNX spec.

[lenyturmel]

ok, I'm not expert enough in embeddings/LLM and ONNX specifications to follow the path of manually building an ONNX protobuf.
So I will follow your advice and try again to stitch the tokenizer to the model to get a single ONNX graph. I have already made several attempts to do this without success. I hope the sample Java code will help me succeed.

In case I fail again, I will look for another solutions to generate embeddings on smartphone for FAQ which contains technical documentation in French (maybe some ready-to-use ONNX graph).

Thanks a lot for all your advice and your time to help me