Prompty, openai-messages-token-helper and tool calls

[gaborvar]

Hi,
Prompty apparently has some support for tool calls. It appears that prompty.prepare(prompt, data) compacts the calling ('assistant') message and the subsequent return ('tool') message into a single assistant message:

{'role': 'assistant', 'content': 'None\n\ntool:\n{"functionname": true, "result": ["...", "..."]}'}

The additional assistant message breaks the normal user/assistant cadence that promptmanager relies on:

azure-search-openai-demo/app/backend/approaches/promptmanager.py

Line 63 in d277675

for user_message, assistant_message in zip(remaining_messages[0::2], remaining_messages[1::2]):

Before considering a fix it would be useful to understand the future evolution of the message management part of the project. In addition to its token accounting feature, openai-messages-token-helper has overlapping features with the recently introduced prompty as both repackage messages (e.g. separating and adding back few-shots and latest user message etc.). I understand that tool calls are not prioritized in this project, however the growing code base translates into more maintenance to ensure that tool calls pass through transparently.

The best course of action also depends on prompty's stance towards tool_calls. Will the current support evolve ?

If tool_calls coexistence remains out of scope in this project then I welcome hints where I should look for a comparable scheme which is more friendly for this project. My current code builds tool calls dynamically to intelligently interface with the procedural world. Would structured outputs with response_format fit this project's philosophy better than tool_calls?

Thanks for the great work on the project

[pamelafox]

We definitely need to support either function calling, structured outputs, or function calling with structured outputs in this repo. I didn't realize there was currently an issue with the integration, looks like we need additional testing if I missed that.

As for prompty versus token-helper: The main goal of token-helper is to truncate conversation history to fit inside context windows. As the newer models have long context window lengths (128K), I'm beginning to think we may not need that truncation. It'd be great if we could remove the token-helper overhead and just assume the conversation fits inside the context window.
What are your thoughts on that?

As for structured outputs vs function calling: the main reason I haven't moved to structured outputs is that it's still not as well supported as function calling across the models and ecosystem. However, it is fairly well supported by Azure OpenAI models, so I'd be up for moving to that instead. Also curious what others think.

[gaborvar]

Hi,
For some background on making token-helper coexist with tool_calls you can look at these pull requests:

pamelafox/openai-messages-token-helper#23 (comment)
pamelafox/openai-messages-token-helper#21 (comment)
(this is not complete. I have made a few other updates recently - let me know if you are interested to adopt)

Goal of these PRs is not to support tool_calls with token counting but to transparently coexist with them. I did not attempt to count tokens of tool_calls and related messages but marked in the PR where that needs to be done.

But for longer term, I vote against fixing and maintaining token-helper. Although my project may upsize, currently I am far from depleting the context window. I only monitor token use and do rudimentary reaction if at all needed rather than the sophisticated ex-ante approach of token-helper.
If I have to choose, I choose tool-calls, not token counting.
Token-helper is also sensitive to typing. If I allow return values from the API to pass through to token-helper then it raises an exception. This should also be updated.
When OpenAI added the "audio" support to its API it also broke token-helper if used with tool_calls.

What do others think?

[King-Del-ErnestO]

Are you saying we can't add other function calls to this existing version?

If we can, how can I achieve that, I've tried various approaches and it didn't work, bot falls back to "search_sources" function

[gaborvar]

Are you saying we can't add other function calls to this existing version?

If we can, how can I achieve that, I've tried various approaches and it didn't work, bot falls back to "search_sources" function

Hi, I added tool calls, however it required changes of distant parts of the code base as they were not written with tool calls in mind. Namely, this imported module blocks messages related to processing of tool calls:

azure-search-openai-demo/app/backend/requirements.txt

Line 220 in 88f987e

openai-messages-token-helper==0.1.11

However, if in your case bot fell back to "search_sources" then you may have an unrelated issue.
Have you tried to change tools variable according to the call you wanted to make? I did not experience any issue with calling my tool after adding the descriptor of the tool to the tools variable as required by the OpenAI documentation. I did it by replacing this line:

azure-search-openai-demo/app/backend/approaches/chatreadretrieveread.py

Line 105 in 88f987e

tools: List[ChatCompletionToolParam] = self.query_rewrite_tools

Have you added instructions to the system message about using your new tool calls?

You may want to look at this def as it has to be adjusted to consider other tool calls than search_sources:

azure-search-openai-demo/app/backend/approaches/chatapproach.py

Line 19 in 88f987e

def get_search_query(self, chat_completion: ChatCompletion, user_query: str):

I replaced it with this:

def get_requested_function(self, chat_completion: ChatCompletion):
    response_message = chat_completion.choices[0].message

    if response_message.tool_calls:
        for tool in response_message.tool_calls:
            if tool.type != "function":
                continue
            return tool     #  can be extended to accept more than one function call. For now, only the first requested function's parent is returned
    # not a tool call or none of the tools were function calls
    return None

You have to adjust the calling code also accordingly.

[cforce]

@gaborvar Is it fixed? Have you any working example with an additional tool call?

[hugokoopmans]

I also am struggeling adding more then one tool. I obviously need the search_tool but before that i need to use additional tools. I am working on CO2 emission and cars soin some cases i need to ask for the licenceplate of the users car and then query some api's to get details on the car BEFORE i do a document search. I am trying to get this into prompty and the chat_read_retrieve_read approach... but untill now to no avail... any advise appriciated (i still have to study the above remarks by @gaborvar

[pamelafox]

I have an example branch that adds an additional tool here:
#2738

You will need to use a slightly different approach since you want to call a completely different API (that branch just modifies the search query) but hopefully that PR gives you a starting point.

[pamelafox]

I am going to close this issue since the original issue was about token counting of tool calls, and we no longer do token counting. @hugokoopmans Please open a new discussion if you need more guidance on customizing it.

[gaborvar]

@hugokoopmans @cforce
I created a fork to accommodate two additional tool calls. The APIs that I call are not related to the search API. Their purpose is to hand off the output from the chat in specific ways when the LLM chooses to hand off output. Happy to share the key parts of my code.

I experimented two implementation alternatives:

(1) Retained prompty and promptmanager.py and added the two new API descriptions to prompty's json files in app/backend/approaches/prompts directory

(2) Skipped prompty and assembled the prompt in my code.

I believe both are possible, however at that time (February 2025) promptmanager.py had a bug which prevented me from debugging approach (1). It is resolved since then and you could backport my code to (1).

Let me know if you are interested.

[hugokoopmans]

Hi @gaborvar that sounds interesting, i have discussion here in my team about the usefullness of prompty in our solution but i would like to stick close to the code and design in your repo so if we could look at option 1 that would be great... I now looked into the chatreadretrieveread approach... i think i need a tool-select-retriever-or-not-retrieve ;-) appoach...

[gaborvar]

@hugokoopmans

I invite you to my fork. You can review both options there.

I changed the flow of the sample app by allowing the LLM to reply without search (in STEP 1). The LLM has 4 options:

1. Search (as in azure-search-openai-demo)
2. Respond with textual completion meant to be displayed for the user
   3-4. Call either of the two APIs that create the output from the conversation

I considered two options to add tool call request and tool response data to the message history:

(A) If the LLM responds with a tool call request, do not add LLM's response to the message history, instead:

1. Complete the tool calls. Log them (arguments, results) in a separate list in session_state
2. Complete any additional steps (perhaps Search with STEP 2 depending on the LLM's decision) and create the textual UI-bound completion. If needed call LLM again (STEP 3)
3. Then add the textual response to the message list. Return with coroutine creation
   This maintains the strict "user input; LLM response" scheme in the message history without extra messages between them. I also had to fix the pathway of session_state to ensure it preserved up to chatreadretrieveread.py

or:
(B) Add complete history to the message history, including tool related LLM calls (role: assistant, tool_calls: ...), tool call result (role: tool, content: ..., tool_call_id: )
Call the LLM again if needed (STEP 3), add the resulting response (role: assistant), then return with coroutine creation
This requires changing the client code to skip non-UI-bound messages in the message list

I had to choose (A) because changing the client code seemed more risky.
If I started today I would consider implementing the above flow with MCP or similar agentic plumbing but the flow would probably remain the same. It is agentic logic without using specifically agentic-focused infrastructure.