Executing Tools Concurrently

[Wal8800]

Do we have plans to enable AiService to execute tools concurrently?

[geoand]

@mariofusco does upstream LangChain4j do this?

[rameshrvr]

Just sharing my thoughts here. I've seen concurrent tool calls in our GPT agent, where the AiResponse contains bunch of tools call that has to be executed as part of single transaction. The results from all these tool calls are then aggregated and returned as a single tool call result object. However i am not sure whether the actual tool execution happens in parallel since the current tool execution implementation does not appear to support Uni or Multi.
My assumption is that while the LLM can request multiple tool invocations in a single response, our application likely processes them sequentially and then returns the combined results.
@Wal8800 It would be great if you could specify what exact concurrent execution you are looking for.
@geoand Any plans on making the tool executions completely reactive?

[geoand]

@geoand Any plans on making the tool executions completely reactive?

If that happens, it will require changes in LangChain4j...

[mariofusco]

As reported in langchain4j documentation, tools are always executed sequentially by default, but it is possible to configure the AiService to call them in parallel.

@geoand correct me if I'm wrong, but I believe that this possibility is not exposed by the Quarkus extension, right? If so I'm available to implement support for this feature, even though I'm probably not aware of all the possible implication.

@rameshrvr As for making the tool executions completely reactive this will indeed require some changes in LangChain4j internals and to be honest at the moment I really don't see what could be the advantages in doing this, can you make a practical example?

[geoand]

@geoand correct me if I'm wrong, but I believe that this possibility is not exposed by the Quarkus extension, right? If so I'm available to implement support for this feature, even though I'm probably not aware of all the possible implication.

Yeah. I was referring to the reactive APIs question above

[rameshrvr]

@rameshrvr As for making the tool executions completely reactive this will indeed require some changes in LangChain4j internals and to be honest at the moment I really don't see what could be the advantages in doing this, can you make a practical example?

@mariofusco @geoand Our entire agent pipeline is implemented in a fully reactive manner. Some agents require tools while others do not. The reactive LLM calls working well for the agents that doesn't require tools. However, the reactive chain breaks during runtime when the tool is invoked as part of LLM API call. To work around this we are writing it as imperative and offload the execution to the worker threads. So these agent calls are no longer truly reactive.
For now, we are not facing any scalability issues but we suspect this may result is scalability issues if the number of such LLM calls increases significantly.

[geoand]

We certainly could make the tool calls in parallel if necessary

[rameshrvr]

We certainly could make the tool calls in parallel if necessary

I would prefer making the tool calls truly reactive coz that's the only ticking time bomb in our system :). But after looking at the implementation in Langchain4j i feel the scope is very limited.

[geoand]

Langchain4j will likely need a redesign in its core to make all the necessary parts reactive

[Wal8800]

@Wal8800 It would be great if you could specify what exact concurrent execution you are looking for.

Looking for the same behaviour from langchain4j where it fires all the tool calls then wait for the results.

https://github.com/langchain4j/langchain4j/blob/d9c687fc50be4cf635ad9cf983bd0ef177f617d6/langchain4j/src/main/java/dev/langchain4j/service/tool/ToolService.java#L473-L496