Provide a way to intercept and store chat history

[chirino]

It would be nice if we could annotate the @RegisterAiService class methods with something like @StoreHistory annotation.

I ended building something like that in: https://github.com/chirino/memory-service?tab=readme-ov-file#5-conversation-history-recording using the @RecordConversation, but I think you guys could do an even better job.

Now the tricky part might be designing a good pluggable SPI. A simple approach might be to just call some store interface with the user message and then again with the final agent response. But my store also wants to make in flight streaming responses resumable/cancelable so it needs to wrap the stream response too. Oh and we need a way to make sure the user identity is propagated to those store interfaces.

[geoand]

Seems reasonable to me!

WDYT @cescoffier ?

[geoand]

IIUC, the solution you came up with is https://github.com/chirino/memory-service/blob/bfd26f7d4690d180abb09706daa64ab15c2df80f/quarkus/memory-service-extension/runtime/src/main/java/io/github/chirino/memory/history/runtime/ConversationInterceptor.java, right?

Can you describe a little what the various pieces do?

Thanks

[chirino]

So it's just a basic invocation interceptor that before the invocation it calls:

store.appendUserMessage(invocation.conversationId(), invocation.userMessage())

and after the invocation (non-streaming) it calls something like:

store.appendAgentMessage(invocation.conversationId(), result);

but if you are streaming a response..

we need to pass it the stream.. Right now in that case my current class is not super clean as it is also dealing with response resumption, but I think we could make the store interface more generalized to handle the response stream too. So instead of doing that complicated ConversationStreamAdapter.wrap thing it's currently doing maybe it just does something like

store.appendAgentMessage(invocation.conversationId(), stringMulti);

Right now my implementation also is doing stuff to get the identity from the current thread, but perhaps (if the store is always called on a thread that we can resolve the SecurityIdentity of the user) that can also be pushed into the store implementation further simplifying things.

Keep in mind this only is working with simple Multi stream.. We also need to enhance this to work with Multi< ChatEvent>.

[geoand]

🙏🏽

[cescoffier]

I think it would be a great feature! (sorry, missed the ping)

[chirino]

I've been able to simplify the interceptor down to https://github.com/chirino/memory-service/blob/047834d3236ea0717f424a9e0e6a029a8c81c04b/quarkus/memory-service-extension/runtime/src/main/java/io/github/chirino/memory/history/runtime/ConversationInterceptor.java

Main thing missing now is handling a Multi< ChatEvent> stream