How to interact with the plan generated by the function_calling_stepwise_planner before proceeding?

[ColtonBehannon]

In one of the Microsoft Build presentations, there was a demo from Fujitsu on how they were able to edit the plan generated by the planner. In trying to do this on my own, I see a couple ways this looks promising but can't figure either out. In function_calling_stepwise_planner.py I was thinking the behavior could be mimicked and the initial_plan could be edited after being generated here. However, this would require changing library code which is less than ideal. The second idea would be to use the new filters but I'm unsure how that could be paired with the planner. Any ideas or samples on this topic?

[dmytrostruk]

@ColtonBehannon Did you try using auto function calling feature? With function calling you can send a request and let the AI model decide how to achieve provided goal. Semantic Kernel will invoke all necessary functions for you. In this case, with filters you should be able to change the behavior in the middle of function calling process.

Example of function calling in Python is here:
https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/auto_function_calling/chat_gpt_api_function_calling.py

Let me know if this functionality could handle your scenario. Thanks!

[ColtonBehannon]

After some testing I see what you mean by it auto invoking more than one tool call and this does indeed seem promising but unless I'm misunderstanding it seems that this might require a lot of development effort to get to work properly and still might not accomplish exactly what I'm going for.

For example, say I have a use case with 3 plugins: the core plugin "time", a custom news search plugin, and a web scraper plugin. I provide the ask of "What news with respect to AI has occurred in the last month?". If I were to use the function_calling_stepwise_planner it could create and execute a plan like (this is based off of a real plan it generated):

1. Use the TimePlugin-now function to get the current date and time.
2. Use the TimePlugin-days_ago function with the output from step 1 to find the date 30 days ago.
3. Use the WebScraperPlugin-NewsWebSearch function with the query "AI", count set to a reasonable number (e.g., 10), and date set to the output from step 2 to get news from the last 30 days.
4. Compile the news results from the three searches into a summary.
5. Use the UserInteraction-SendFinalAnswer function to send the compiled summary of AI news from the last

The issue with this is that maybe I want to change a parameter in step 3 or add a new plugin in between step 3 and 4 to gather more information. The simplicity and transparency of the planner are extremely valuable here, but I cannot edit the plan before it executes. Additionally, if I could make edits I could do them all at once including multiple function calls as well as semantic processes like summarization.

If I were to just use the automatic function calling without the planner, I would need to filter every function call and then have to approve, deny, or edit each call which removes some of the appeal of it being automatic in the first place while also removing the transparency of its "reasoning".

[dmytrostruk]

The issue with this is that maybe I want to change a parameter in step 3 or add a new plugin in between step 3 and 4 to gather more information.

This is one of the reasons why we introduced Filters - to add flexibility and more control for function calling process.

If I were to just use the automatic function calling without the planner, I would need to filter every function call and then have to approve, deny, or edit each

If you want to see which functions LLM wants to call, but have a full control whether to call it or not and what arguments to pass, instead of automatic function calling you can use manual function calling:

semantic-kernel/python/samples/concepts/plugins/openai_function_calling_with_custom_plugin.py

Line 101 in d6b6200

print("========== Example 3: Use manual function calling with a non-streaming prompt ==========")

You will receive a response from LLM which functions to call (basically a plan), and then you can edit it based on your needs and execute it on your side.

The concepts of planner and function calling are very similar. In planner, we ask LLM to generate plan string with functions to execute. In function calling we ask LLM to decide whether to invoke any functions or not based on request. The implementation of these two concepts is different, but input and output are very similar, if not the same.

[ColtonBehannon]

I cannot seem to get Example 3 to run. It will run if I change auto_invoke to False but will not use or suggest any functions. Here is where the error is coming from: https://github.com/microsoft/semantic-kernel/blob/main/python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py#L94. The only change I've made to the file is changing use_azure_openai to True.

Also, what is the purpose of auto_invoke as I can't find any clear documentation and am having trouble just by reading through the repo since this flag is used even in instances where the intention is manual function calling?

Apologies for all of the replies...

[dmytrostruk]

@moonbox3 @eavanvalkenburg Could you please help here with example?

[dmytrostruk]

@ColtonBehannon The example was fixed in this PR: #6477.