Python: Trouble understanding how to control the prompt that is executing the plan created by the planner in function_calling_stepwise_planner

[ColtonBehannon]

I see that I could change the prompt that is behind the actual planner of the function_calling_stepwise_planner by providing the get_initial_plan argument to FunctionCallingStepwisePlannerOptions and passing that to the FunctionCallingStepwisePlanner. However, the default prompt works well for my purposes, and I'm instead interested in how I can alter the prompt being used to return the final answer to the user.

Please correct me if I'm wrong but from my understanding there are essentially 3 prompts at play by default.

1. generate_plan_yaml which gives the planner it's context
2. a) step_prompt which gives intermediate context for the model to know what to do initially with the plan that is generated
   b) STEPWISE_USER_MESSAGE to help the model iterate through steps
3. the prompt behind the "UserInteraction-SendFinalAnswer" function. This is the one I am having trouble identifying how it could be edited.

My end goal would be to edit the prompt behind this function in order to return the output in a standard format.

[ColtonBehannon]

Update: After finding a discussion about a similar issue posted a couple weeks ago (Sadly I could not find the discussion again to link here) there was a recent reply from @moonbox3 suggesting the user use the callbacks I had initially mentioned in my question. I created a new version of generate_plan.yaml and added that to the FunctionCallingStepwisePlannerOptions and got the result I was looking for.

Though I got the desired result, it still feels wrong to me as I don't fully understand how the UserInteraction plugin is working. To me, it feels like this is where the edits should be made to any prompting since this is the formatting of the final result. Could someone explain how UserInteraction-SendFinalAnswer is actually being used? I'm confused by the fact that it just contains return "Thanks" in the body of the kernel function.

[moonbox3]

Hi @ColtonBehannon, the UserInteraction-SendFinalAnswer plugin is a more concrete way for us to handle the final answer, if/when it is created by the LLM. The answer parameter to the function is an indication to the LLM and is not intended to be used. You can see here that we're doing a check if the FunctionResult contains a plugin with the name UserInteraction-SendFinalAnswer, otherwise other text returned from the LLM could be part of the planned process and not something we want to actually stop on.

https://github.com/microsoft/semantic-kernel/blob/8f15f3a81c3bbbfa2f3fe65f4a9034e76425e693/python/semantic_kernel/planners/function_calling_stepwise_planner/function_calling_stepwise_planner.py#L167C1-L172C15

It sounds like what you're doing is perfectly fine, which is the more advanced way of providing your own callbacks for the initial plan prompt and step prompt template.

[ColtonBehannon]

After further review my attempt does not seem to be working as well as originally thought. The way I went about it was to append some more text at the end of line four of generate_plan.yaml.

As a test, I appended the sentence, "When using UserInteraction{{$name_delimiter}}SendFinalAnswer you should respond with 'Bazinga!' at the end of your answer" immediately after. The results of this have been mixed.

It appears to be successful when the planner adds a step like, "End the communication with the phrase 'Bazinga!' as instructed" to the plan but then fails when the plan includes making function calls as it tends to forget to add the "Bazinga!" instruction anywhere in the plan. I don't think it should necessarily matter if this extra bit of formatting is included in the step-by-step plan as it is still part of the prompt, and the model is successful in detecting the instruction to use the function 'UserInteraction-SendFinalAnswer' which is also just part of the prompt. My thinking is that telling the LLM to use 'UserInteraction-SendFinalAnswer' is given more weight (as it should) because of the repeated use of the STEPWISE_USER_MESSAGE. But this makes me wonder if having the ability to edit this message would bring about some more power in controlling the planner?

I wanted to ask your thoughts on this situation and what you would recommend. My only path forward seems to be to double down on the get_intial_plan prompting by being more explicit and perhaps adding some few shot examples. Is this what you would recommend as well or do you have any additional suggestions?