LocalAI Chat endpoint structure

[wizziLalev]

Hello,
I am using a local ollama model to do the translations for me, but for some time I cannot make Lingarr use the /api/chat url properly, so I've done my research and here are my findings:

lingarr/Lingarr.Server/Services/Translation/LocalAiService.cs

Line 101 in e47e573

var isChatEndpoint = _endpoint.TrimEnd('/').EndsWith("completions", StringComparison.OrdinalIgnoreCase);

Here you are looking for _endpoint to ends with completions - which may be valid for some other local LLM servers/implementations but ollama expects that URL to be /api/chat/ -> docs

Will be great to get your thoughts on that topic and maybe you can update the implementation - sorry my C# is not so strong to provide the PR with needed change :)

[rowanfuchs]

The chat endpoint is meant for a context based interaction, whereas the completions endpoint is for single interactions. The line you mention is to determine whether the user has provided the chat or completion endpoint. Here, you see the chosen method based on that condition.

lingarr/Lingarr.Server/Services/Translation/LocalAiService.cs

Lines 111 to 113 in e47e573

	return isChatEndpoint
	? await TranslateWithChatApi(text, cancellationToken)
	: await TranslateWithGenerateApi(text, cancellationToken);

Context based interaction isn't implemented in Lingarr, but do you receive an error with the chat method?

[wizziLalev]

I am still getting error 500 in ollama which cause the entire translate job to fail, and I was looking for a way to improve and/or debug the process. Also I think that there is a potential for speed optimization as well. My system prompt is quite detailed and a lot bigger from what is used by default and wanted to see how that is being used in combination with the text that we want to translate.

At the end of the day maybe it will be nice if you add an example of the API URLs that can be used with LocalAI models in the documentation with simple explanation of them. To be honest I was unaware that Ollama added support of completions API at all and most of the UIs available are not using it.

Will try to test few things here and there in the localAI implementation and will create a PR if manage to do some optimizations :) it will be nice exercise for my C#-fu

[rowanfuchs]

Feel free to create a pull request to add any (sub) documentation, it's absolutely welcome as long as it doesn't clutter the main documentation.

As for the error, I would assume you need to debug your Ollama installation. Prompting will become more significant once we implement RAG. Personally, I still believe this is the best approach, using multiple lines often introduces new issues, such as return values being full of hallucinations and other errors. But I am open to suggestions/other implementations.