How to influence the OpenAIFunctionsAgent on creating the initial search query

[guidorietbroek]

I've build an agent with a multi query retriever with sources and some other things. The thing is I use an AgentExecutor where an OpenAIFunctionsAgent is loaded. In the debugging mode I see that the user input is transformed to a search query for one of the tools. But it's doing it in a very bad manner and therefor the retrieval is not giving useful documents.

So when the query is something like "What kind of insurances does company X has?", the search query will be "X". Of course no results are giving which is answering the question, so the model responds that it doesn't now. It look likes this:

[llm/end] [1:chain:AgentExecutor > 2:llm:ChatOpenAI] [1.07s] Exiting LLM run with output:
{
  "generations": [
    [
      {
        "text": "",
        "generation_info": null,
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain",
            "schema",
            "messages",
            "AIMessageChunk"
          ],
          "kwargs": {
            "content": "",
            "additional_kwargs": {
              "function_call": {
                "name": "company-information",
                "arguments": "{\n  \"__arg1\": \"X\"\n}"
              }
            }
          }
        }
      }
    ]
  ],
  "llm_output": null,
  "run": null
}

Looking at the OpenAIFunctionsAgent code I do see these lines of code:

 # HACK HACK HACK:
        # The code that encodes tool input into Open AI uses a special variable
        # name called `__arg1` to handle old style tools that do not expose a
        # schema and expect a single string argument as an input.
        # We unpack the argument here if it exists.
        # Open AI does not support passing in a JSON array as an argument.
        if "__arg1" in _tool_input:
            tool_input = _tool_input["__arg1"]
        else:
            tool_input = _tool_input

I do have the feeling that this part is the 'problem'. Does anybody knows how to solve this issue?

[dosubot[bot]]

🤖

Hello,

From your description, it seems like the issue lies in the way the initial search query is being generated. The __arg1 variable is indeed used to handle old style tools that do not expose a schema and expect a single string argument as an input. If __arg1 is present in the tool input, it is unpacked and used as the tool input. If __arg1 is not present, the entire tool input is used.

However, the problem you're facing seems to be more related to the generation of the initial search query rather than the handling of __arg1. The OpenAIFunctionsAgent is likely generating a too simplistic search query based on the user's input, which is why you're not getting useful documents.

One way to improve this is by modifying the DEFAULT_QUERY_PROMPT in the MultiQueryRetriever class. This prompt is used to generate multiple versions of the user's question, which are then used to retrieve relevant documents. You can modify this prompt to generate more diverse queries, for example by using synonyms or related terms. Here's an example of how you can modify the prompt:

DEFAULT_QUERY_PROMPT = PromptTemplate(
    input_variables=["question"],
    template="""You are an AI language model assistant. Your task is 
    to generate 5 different versions of the given user 
    question to retrieve relevant documents from a vector  database. 
    By generating multiple perspectives on the user question, 
    your goal is to help the user overcome some of the limitations 
    of distance-based similarity search. Provide these alternative 
    questions separated by newlines. Original question: {question}""",
)

Another way to improve the search query creation is to modify the generate_queries method in the MultiQueryRetriever class. This method currently generates queries based on the user's input. You could modify this method to generate more diverse queries.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/retrievers/multi_query.py
• libs/langchain/langchain/agents/openai_functions_agent/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[guidorietbroek]

@dosu-bot

I think it has something to do with the function call in the OpenAIFunctionAgent. This argument __arg1 is just taking a part of the question as input, while you want to have the full context of the question to do a retrieval from one of the tools available. By passing the question to OpenAI with the available tools, OpenAI should be able to detect which tool is best to use. But then it needs the full context.

Two other questions:

1. Where can I find the full standard prompt from the OpenAiFunctionsAgent?
2. Where can I find the code which forms the API call to OpenAI from the OpenAIFunctionsAgent?

Please answer all three questions.

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing a detailed explanation of your issue. I understand that you're having trouble with the OpenAIFunctionsAgent in the LangChain Python framework, specifically with how the user input is transformed into a search query.

To answer your questions:

1. To modify the OpenAIFunctionsAgent to use the full context of the question for retrieval, you can adjust the max_tokens parameter in the get_sub_prompts method of the BaseOpenAI class. This parameter determines the maximum number of tokens to generate in the completion. Setting it to -1 will return as many tokens as possible given the prompt and the model's maximal context size. Here's the relevant code snippet:

def get_sub_prompts(
        self,
        params: Dict[str, Any],
        prompts: List[str],
        stop: Optional[List[str]] = None,
    ) -> List[List[str]]:
        """Get the sub prompts for llm call."""
        if stop is not None:
            if "stop" in params:
                raise ValueError("`stop` found in both the input and default params.")
            params["stop"] = stop
        if params["max_tokens"] == -1:
            if len(prompts) != 1:
                raise ValueError(
                    "max_tokens set to -1 not supported for multiple inputs."
                )
            params["max_tokens"] = self.max_tokens_for_prompt(prompts[0])
        sub_prompts = [
            prompts[i : i + self.batch_size]
            for i in range(0, len(prompts), self.batch_size)
        ]
        return sub_prompts

2. The standard prompt for the OpenAI API call is determined by the prompts parameter in the _agenerate method of the OpenAI class. This method is responsible for generating responses from the OpenAI model.

async def _agenerate(
        self,
        prompts: List[str],
        stop: Optional[List[str]] = None,
        run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,
        **kwargs: Any,
    ) -> LLMResult:
    ...

3. The actual API call to OpenAI is made in the acompletion_with_retry function, which uses the acreate method of the OpenAI client to make an asynchronous request to the OpenAI API.

async def acompletion_with_retry(
    llm: Union[BaseOpenAI, OpenAIChat],
    run_manager: Optional[AsyncCallbackManagerForLLMRun] = None,
    **kwargs: Any,
) -> Any:
    """Use tenacity to retry the async completion call."""
    retry_decorator = _create_retry_decorator(llm, run_manager=run_manager)

    @retry_decorator
    async def _completion_with_retry(**kwargs: Any) -> Any:
        # Use OpenAI's async api https://github.com/openai/openai-python#async-api
        return await llm.client.acreate(**kwargs)

    return await _completion_with_retry(**kwargs)

Regarding the __arg1 variable, it is used in the LangChain codebase to handle old style tools that do not expose a schema and expect a single string argument as an input. This is a workaround because OpenAI does not support passing in a JSON array as an argument. In the _parse_ai_message function, the __arg1 variable is checked for in the _tool_input dictionary. If __arg1 is present, its value is assigned to tool_input. If not, the entire _tool_input dictionary is assigned to tool_input.

I hope this helps! If you have any other questions or need further clarification, please don't hesitate to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/llms/openai.py
• libs/langchain/langchain/agents/openai_functions_agent/base.py
• libs/langchain/langchain/agents/openai_functions_multi_agent/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.