QuarkusAiServiceTokenStream over AiServiceTokenStream

[tomas1885]

@geoand
I'm implementing support for allowing using Multi and while digging and testing I see there's some general issue or missing functionality when using streaming. For example audit events are not fired, at least not properly, and that's an easy fix.
When using rag Quarkus guardrails are not called etc.
What I'm wondering is what's the benefit of using the QuarkusAiServiceTokenStream implmentation over the AiServiceTokenStream?

[geoand]

See the Javadoc of QuarkusAiServiceTokenStream. In general if we could get rid of it, that would be great

[tomas1885]

@geoand
The reason I'm asking is because I started looking at how to allow returning a Multi with more information such as token usage etc.
Than I saw it's possible to return a TokenStream, and thought, it would be better as it already exposes all of the required data. But I see it's the default implementation which would obviously block the event loop. So I'm thinking about maybe allowing to return a Multi of an abstraction of events such as partial response, content retrieval, tool execution etc. But this off-course causes some issues when using output guardrails, which can be solved by jumping through some hoops, I implemented something like that, but I don't too good about it. So I'm wondering about plans to consolidate the TokenStream with the upstream implementation or at least the output guardrails (I saw another discussion about this), before I open any PR.

[geoand]

We generally do want to consolidate the code that has diverged, but we haven't done any work specifically on QuarkusAiServiceTokenStream.

You also be interested to know that @edeandrea is working on replacing the custom guardrails support here with the one from upstream LangChain4j.

[tomas1885]

I have a branch that allows producing a Multi with all of the chatevents, this is why I wanted to know which direction does the whole thing here is going

[edeandrea]

At least for guardrails the plan is to remove the custom implementation here in favor of the upstream implementation.

See

#1284

#1619

The output guardrails do support TokenStream, but TokenStream is not true streaming like a Multi is.