Status of OnnxRuntimeGenAIChatCompletionService

[Aloshh]

Hi, we are working on a project where the requirement is to integrate SLM in a Maui application that will be running on both IOS and Android. We are thinking about using SK and I saw that there is a connector called onnx.onnxruntimegenaichatcompletionservice, but it is still in preview thus not production ready. Do you have any estimate on when it will be production ready? And in the meantime can you recomend another solution for running SLM in Maui applications running on mobile phones?

Thanks in advance

[stephentoub]

And in the meantime can you recomend another solution for running SLM in Maui applications running on mobile phones?

The Microsoft.ML.OnnxRuntimeGenAI library itself includes an IChatClient implementation:
https://github.com/microsoft/onnxruntime-genai/blob/00a5fb62a36f9635a476039fb60b40b2901c5e71/src/csharp/OnnxRuntimeGenAIChatClient.cs#L17
The expectation is that OnnxRuntimeGenAIChatCompletionService (which was written before that existed) will just be rewritten to be a thin wrapper for that. If you don't want to depend on OnnxRuntimeGenAIChatCompletionService, you can use the underlying component directly.

[Aloshh]

Thanks for the advice Stephen. But will this work in a Maui application running on both Android and IOS? Please correct me if im wrong, but it seems onnxruntimegenai does not fully support IOS at this moment?

[paulio]

I am running Onnx with builder.AddOnnxRuntimeGenAIChatCompletion on iOS from a MAUI app. It does work. I haven't tried Android yet. I can't comment on the nature of how officially supported it is.

[Aloshh]

Thanks for a lot for confirming.

Do you by any chance have a repo I can get inspiration from? And also did you have to build from source to make it work?

[paulio]

I don't currently have a repo for it. However, the roadblocks were mostly due to incompatibilities in the Semantic Kernel library which have now all been resolved. So it works using the current packages. I would note that the device needs a good amount of memory, e.g. 10th Gen iPad can only use a very limited number of token before it exceeded the memory limit