How can I use instructor with multiple functions - in a more agentic system where the LLM chooses the function to call?

[zby]

Let's say I have the classic pattern of a search and lookup functions - search finds the page, lookup looks up a fragment on that page containing a keyword. The LLM needs to decide if for the next step it wants to search for another page or lookup another part of the current page. How that can be achieved with instructor?

For reference here is the current schema that I pass to the 'tools' parameter:

[   {   'function': {   'description': 'Finish the task and return the answer.',
                        'name': 'finish',
                        'parameters': {   'properties': OrderedDict([   (   'answer',
                                                                            {   'description': 'The answer to the '
                                                                                               "user's question",
                                                                                'type': 'string'}),
                                                                        (   'reason',
                                                                            {   'description': 'The reasoning behind '
                                                                                               'the answer',
                                                                                'type': 'string'})]),
                                          'required': ['answer', 'reason'],
                                          'type': 'object'}},
        'type': 'function'},
    {   'function': {   'description': 'Retrieves a Wikipedia page, saves the result, and informs about the content of '
                                       'that page.',
                        'name': 'get',
                        'parameters': {   'properties': OrderedDict([   (   'title',
                                                                            {   'description': 'The page title',
                                                                                'type': 'string'}),
                                                                        (   'reason',
                                                                            {   'description': 'The reason for '
                                                                                               'retrieving the page',
                                                                                'type': 'string'})]),
                                          'required': ['title', 'reason'],
                                          'type': 'object'}},
        'type': 'function'},
    {   'function': {   'description': 'Looks up a word on the current page.',
                        'name': 'lookup',
                        'parameters': {   'properties': OrderedDict([   (   'keyword',
                                                                            {   'description': 'The keyword to search',
                                                                                'type': 'string'}),
                                                                        (   'reason',
                                                                            {   'description': 'The reason for '
                                                                                               'searching',
                                                                                'type': 'string'})]),
                                          'required': ['keyword', 'reason'],
                                          'type': 'object'}},
        'type': 'function'},
    {   'function': {   'description': 'Jumps to the next occurrence of the word searched previously.',
                        'name': 'next',
                        'parameters': {   'properties': OrderedDict([   (   'reason',
                                                                            {   'description': 'The reason for '
                                                                                               'searching',
                                                                                'type': 'string'})]),
                                          'required': ['reason'],
                                          'type': 'object'}},
        'type': 'function'},
    {   'function': {   'description': 'Searches Wikipedia, saves the first result page, and informs about the content '
                                       'of that page.',
                        'name': 'search',
                        'parameters': {   'properties': OrderedDict([   (   'query',
                                                                            {   'description': 'The query to search '
                                                                                               'for on Wikipedia',
                                                                                'type': 'string'}),
                                                                        (   'reason',
                                                                            {   'description': 'The reason for '
                                                                                               'searching',
                                                                                'type': 'string'})]),
                                          'required': ['query', 'reason'],
                                          'type': 'object'}},
        'type': 'function'}]

[jxnl]

Union[A, B, C], then you can use pattern matching.
its breaks the paradigm, but im trying to keep it 'only one day to do something'

[zby]

OK - I created this example code:

from pydantic import BaseModel, Field
from typing import Union
import instructor
from openai import OpenAI

class Superclass(BaseModel):
    """Superclass that provides a common 'reason' field for all actions."""
    reason: str = Field(description="The reason for this action")

class Search(Superclass):
    """Search action class with a 'query' field and a process method."""
    query: str = Field(description="The search query")

    def process(self):
        """Process the search action."""
        return f"Search method called for query: {self.query}"

class Lookup(Superclass):
    """Lookup action class with a 'keyword' field and a process method."""
    keyword: str = Field(description="The lookup keyword")

    def process(self):
        """Process the lookup action."""
        return f"Lookup method called for keyword: {self.keyword}"

class Finish(Superclass):
    """Finish action class with an 'answer' field and a process method."""
    answer: str = Field(description="The answer for finishing the process")

    def process(self):
        """Process the finish action."""
        return f"Finish method called with answer: {self.answer}"

# Union of Search, Lookup, and Finish
ActionUnion = Union[Search, Lookup, Finish]

# Example usage
action = Search(reason="Need to find information", query="Python")
print(action.process())

# Enables `response_model`
client = instructor.patch(OpenAI())
action = client.chat.completions.create(
    model="gpt-3.5-turbo",
    response_model=ActionUnion,
    messages=[
        {"role": "user", "content": "Please choose one action"},
    ]
)
assert isinstance(action, ActionUnion)
print(action.process())

print()

# Generate JSON Schemas
print("\nCombined JSON Schema for ActionUnion:")
print(ActionUnion.model_json_schema())

Is that what you had on mind?

Unfortunately it fails here:

/home/zby/gpt/instructor_examples/venv/bin/python /home/zby/gpt/instructor_examples/pydantic_orthodox.py 
Search method called for query: Python
Traceback (most recent call last):
  File "/home/zby/gpt/instructor_examples/pydantic_orthodox.py", line 43, in <module>
    action = client.chat.completions.create(
  File "/home/zby/gpt/instructor_examples/venv/lib/python3.10/site-packages/instructor/patch.py", line 356, in new_chatcompletion_sync
    response_model, new_kwargs = handle_response_model(
  File "/home/zby/gpt/instructor_examples/venv/lib/python3.10/site-packages/instructor/patch.py", line 70, in handle_response_model
    if not issubclass(response_model, OpenAISchema):
  File "/usr/lib/python3.10/abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

Process finished with exit code 1

When I comment out all the instructor processing and just try to print out the schema it still fails - so maybe pydantic is not yet supporting this?

/home/zby/gpt/instructor_examples/venv/bin/python /home/zby/gpt/instructor_examples/pydantic_orthodox.py 


Combined JSON Schema for ActionUnion:
Traceback (most recent call last):
  File "/home/zby/gpt/instructor_examples/pydantic_orthodox.py", line 57, in <module>
    print(ActionUnion.model_json_schema())
  File "/usr/lib/python3.10/typing.py", line 983, in __getattr__
    return getattr(self.__origin__, attr)
  File "/usr/lib/python3.10/typing.py", line 375, in __getattr__
    raise AttributeError(item)
AttributeError: model_json_schema

Process finished with exit code 1

[zby]

This was in a new venv with Python 3.10.12 and

(venv) zby@zby-Z4:~/gpt/instructor_examples$ pip freeze
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.2.0
async-timeout==4.0.3
attrs==23.1.0
certifi==2023.11.17
click==8.1.7
distro==1.8.0
docstring-parser==0.15
exceptiongroup==1.2.0
frozenlist==1.4.1
h11==0.14.0
httpcore==1.0.2
httpx==0.26.0
idna==3.6
instructor==0.4.5
markdown-it-py==3.0.0
mdurl==0.1.2
multidict==6.0.4
openai==1.6.1
pydantic==2.5.3
pydantic_core==2.14.6
Pygments==2.17.2
rich==13.7.0
sniffio==1.3.0
tqdm==4.66.1
typer==0.9.0
typing_extensions==4.9.0
yarl==1.9.4

[jxnl]

more like

class TakeAction(BaseModel):
   action: Union[Search, Lookup, Finish]

[jxnl]

# Union of Search, Lookup, and Finish
class TakeAction(BaseModel):
    action: Union[Search, Lookup, Finish]

    def process(self):
        """Process the action."""
        return self.action.process()


# Enables `response_model`
client = instructor.patch(OpenAI())

action = client.chat.completions.create(
    model="gpt-3.5-turbo",
    response_model=TakeAction,
    messages=[
        {"role": "user", "content": "I need to find information about Python"},
    ],
)

[zby]

Thanks! That worked.

One more question - is the order of the parameters preserved?

The schema I get from model_json_schema is:

Combined JSON Schema for ActionUnion:
{'$defs': {'Finish': {'description': "Finish action class with an 'answer' field and a process method.", 'properties': {'reason': {'description': 'The reason for this action', 'title': 'Reason', 'type': 'string'}, 'answer': {'description': 'The answer for finishing the process', 'title': 'Answer', 'type': 'string'}}, 'required': ['reason', 'answer'], 'title': 'Finish', 'type': 'object'}, 'Lookup': {'description': "Lookup action class with a 'keyword' field and a process method.", 'properties': {'reason': {'description': 'The reason for this action', 'title': 'Reason', 'type': 'string'}, 'keyword': {'description': 'The lookup keyword', 'title': 'Keyword', 'type': 'string'}}, 'required': ['reason', 'keyword'], 'title': 'Lookup', 'type': 'object'}, 'Search': {'description': "Search action class with a 'query' field and a process method.", 'properties': {'reason': {'description': 'The reason for this action', 'title': 'Reason', 'type': 'string'}, 'query': {'description': 'The search query', 'title': 'Query', 'type': 'string'}}, 'required': ['reason', 'query'], 'title': 'Search', 'type': 'object'}}, 'properties': {'action': {'anyOf': [{'$ref': '#/$defs/Search'}, {'$ref': '#/$defs/Lookup'}, {'$ref': '#/$defs/Finish'}], 'title': 'Action'}}, 'required': ['action'], 'title': 'TakeAction', 'type': 'object'}

So there are ordinary hashes - is that the same in instructor?

[jxnl]

i thought the parameters are ordered thanks to pydantic.
can't tell but is that not the case?

[zby]

Ah - it looks that python dictionaries are ordered since 3.7.

[zby]

Jason - have a look at https://github.com/zby/instructor/blob/main/examples/union/run.py
It is now a more complete example of what I would like to do.

Unfortunately I had to get back to checking the type of the returned object instead of using the process methods - because I need to keep state between calls and I don't know how to do that in the previous version.

I get a lot of errors of the wrong data structure. They are caught by instructor - but I think there are too many such errors - maybe the structure is too difficult now? Also the messages array at the end does not look right - it seems that instructor does not add the corrected message there - only the wrong one is saved.

Here is an example output:

/home/zby/gpt/instructor/venv/bin/python /home/zby/gpt/instructor/examples/union/run.py 
LLM call number:  0
Retrying, exception: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'query': 'capital of Burkina Faso'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Traceback (most recent call last):
  File "/home/zby/gpt/instructor/instructor/patch.py", line 277, in retry_sync
    return process_response(
  File "/home/zby/gpt/instructor/instructor/patch.py", line 164, in process_response
    model = response_model.from_response(
  File "/home/zby/gpt/instructor/instructor/function_calls.py", line 137, in from_response
    return cls.model_validate_json(
  File "/home/zby/gpt/instructor/venv/lib/python3.10/site-packages/pydantic/main.py", line 532, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
pydantic_core._pydantic_core.ValidationError: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'query': 'capital of Burkina Faso'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Searching for 'capital of Burkina Faso'

Getting page: 'Ouagadougou':

Found info: 
 Ouagadougou or Wagadugu ( WAH-gə-DOO-goo, Mossi: [ˈwɔɣədəɣʊ], French: [waɡaduɡu]) is the capital of Burkina Faso, and the administrative, communications, cultural and economic centre of the nation. It is also the country's largest city, with a population of 2,415,266 in 2019. The city's name is often shortened to Ouaga. The inhabitants are called ouagalais. The spelling of the name Ouagadougou is derived from the French orthography common in former French African colonies.
Ouagadougou's primary industries are food processing and textiles. It is served by an international airport and is linked by rail to Abidjan in the Ivory Coast and, for freight only, to Kaya. There are several highways linking the city to Niamey, Niger, south to Ghana, and southwest to Ivory Coast. Ouagadougou has one of West Africa's largest markets, which burned down in 2003 and has since reopened with better facilities and improved fire-prevention measures. Other attractions include the National Museum of Burkina Faso, the Moro-Naba Palace (site of the Moro-Naba Ceremony), the National Museum of Music, and several craft markets.
LLM call number:  1
Retrying, exception: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Traceback (most recent call last):
  File "/home/zby/gpt/instructor/instructor/patch.py", line 277, in retry_sync
    return process_response(
  File "/home/zby/gpt/instructor/instructor/patch.py", line 164, in process_response
    model = response_model.from_response(
  File "/home/zby/gpt/instructor/instructor/function_calls.py", line 137, in from_response
    return cls.model_validate_json(
  File "/home/zby/gpt/instructor/venv/lib/python3.10/site-packages/pydantic/main.py", line 532, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
pydantic_core._pydantic_core.ValidationError: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Retrying, exception: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Traceback (most recent call last):
  File "/home/zby/gpt/instructor/instructor/patch.py", line 277, in retry_sync
    return process_response(
  File "/home/zby/gpt/instructor/instructor/patch.py", line 164, in process_response
    model = response_model.from_response(
  File "/home/zby/gpt/instructor/instructor/function_calls.py", line 137, in from_response
    return cls.model_validate_json(
  File "/home/zby/gpt/instructor/venv/lib/python3.10/site-packages/pydantic/main.py", line 532, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
pydantic_core._pydantic_core.ValidationError: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Max retries reached, exception: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
Traceback (most recent call last):
  File "/home/zby/gpt/instructor/examples/union/run.py", line 80, in <module>
    action = client.chat.completions.create(
  File "/home/zby/gpt/instructor/instructor/patch.py", line 359, in new_chatcompletion_sync
    response = retry_sync(
  File "/home/zby/gpt/instructor/instructor/patch.py", line 305, in retry_sync
    raise e
  File "/home/zby/gpt/instructor/instructor/patch.py", line 277, in retry_sync
    return process_response(
  File "/home/zby/gpt/instructor/instructor/patch.py", line 164, in process_response
    model = response_model.from_response(
  File "/home/zby/gpt/instructor/instructor/function_calls.py", line 137, in from_response
    return cls.model_validate_json(
  File "/home/zby/gpt/instructor/venv/lib/python3.10/site-packages/pydantic/main.py", line 532, in model_validate_json
    return cls.__pydantic_validator__.validate_json(json_data, strict=strict, context=context)
pydantic_core._pydantic_core.ValidationError: 1 validation error for TakeAction
action
  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.5/v/missing
[{'role': 'system', 'content': 'Please answer a user question using wikipedia as a source.'}, {'role': 'user', 'content': 'What is the capitol of Burkina Faso'}, {'role': 'assistant', 'content': '{"arguments": "{\\n  \\"query\\": \\"capital of Burkina Faso\\"\\n}", "name": "TakeAction"}'}, {'role': 'user', 'content': "Recall the function correctly, fix the errorsexceptions found\n1 validation error for TakeAction\naction\n  Field required [type=missing, input_value={'query': 'capital of Burkina Faso'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.5/v/missing"}, {'role': 'function', 'name': 'search', 'content': "Ouagadougou or Wagadugu ( WAH-gə-DOO-goo, Mossi: [ˈwɔɣədəɣʊ], French: [waɡaduɡu]) is the capital of Burkina Faso, and the administrative, communications, cultural and economic centre of the nation. It is also the country's largest city, with a population of 2,415,266 in 2019. The city's name is often shortened to Ouaga. The inhabitants are called ouagalais. The spelling of the name Ouagadougou is derived from the French orthography common in former French African colonies.\nOuagadougou's primary industries are food processing and textiles. It is served by an international airport and is linked by rail to Abidjan in the Ivory Coast and, for freight only, to Kaya. There are several highways linking the city to Niamey, Niger, south to Ghana, and southwest to Ivory Coast. Ouagadougou has one of West Africa's largest markets, which burned down in 2003 and has since reopened with better facilities and improved fire-prevention measures. Other attractions include the National Museum of Burkina Faso, the Moro-Naba Palace (site of the Moro-Naba Ceremony), the National Museum of Music, and several craft markets."}, {'role': 'assistant', 'content': '{"arguments": "{\\n  \\"answer\\": \\"The capital of Burkina Faso is Ouagadougou\\"\\n}", "name": "TakeAction"}'}, {'role': 'user', 'content': "Recall the function correctly, fix the errorsexceptions found\n1 validation error for TakeAction\naction\n  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.5/v/missing"}, {'role': 'assistant', 'content': '{"arguments": "{\\n  \\"answer\\": \\"The capital of Burkina Faso is Ouagadougou\\"\\n}", "name": "TakeAction"}'}, {'role': 'user', 'content': "Recall the function correctly, fix the errorsexceptions found\n1 validation error for TakeAction\naction\n  Field required [type=missing, input_value={'answer': 'The capital o...na Faso is Ouagadougou'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.5/v/missing"}]

[jxnl]

try to prompt it more by using the docstrings

[zby]

That did not help much.

The combined schema is:

{'$defs': {'Finish': {'description': 'Finish the task and return the answer to '
                                     'the user.',
                      'properties': {'answer': {'description': 'The answer for '
                                                               'the question',
                                                'title': 'Answer',
                                                'type': 'string'}},
                      'required': ['answer'],
                      'title': 'Finish',
                      'type': 'object'},
           'Lookup': {'description': 'Lookup a keyword on the current page.',
                      'properties': {'keyword': {'description': 'The keyword '
                                                                'to lookup',
                                                 'title': 'Keyword',
                                                 'type': 'string'}},
                      'required': ['keyword'],
                      'title': 'Lookup',
                      'type': 'object'},
           'Search': {'description': 'Search wikipedia.',
                      'properties': {'query': {'description': 'The search '
                                                              'query',
                                               'title': 'Query',
                                               'type': 'string'}},
                      'required': ['query'],
                      'title': 'Search',
                      'type': 'object'}},
 'properties': {'action': {'anyOf': [{'$ref': '#/$defs/Search'},
                                     {'$ref': '#/$defs/Lookup'},
                                     {'$ref': '#/$defs/Finish'}],
                           'title': 'Action'}},
 'required': ['action'],
 'title': 'TakeAction',
 'type': 'object'}

it is just much more complex than the alternative:

[
    {
        "type": "function",
        "function": {
            "name": "search",
            "description": "Search wikipedia.",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        'type': "string",
                        'description': "The search query",
                    }
                },
                "required": ["query"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "lookup",
            "description": "Search wikipedia.",
            "parameters": {
                "type": "object",
                "properties": {
                    "keyword": {
                        'type': "string",
                        'description': "The keyword to lookup",
                    }
                },
                "required": ["keyword"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "finish",
            "description": "Finish the task and return the answer to the user.",
            "parameters": {
                "type": "object",
                "properties": {
                    "answer": {
                        'type': "string",
                        'description': "The answer to the question",
                    }
                },
                "required": ["keyword"]
            }
        }
    }
]

https://gist.github.com/zby/08f0aff0a3e688fc567907ff0f5a3818

What if response_model could be a list? Just like tools can be a list.

[johnwlockwood]

I'm using Unions to facilitate allowing the LLM to choose which type of entity to create, but then I tried to use it with Google's genai and that doesn't support Unions. I implemented a Google safe class that has a type field and a field for each possible type with a default_factor that makes it with an identifier that indicates it wasn't generated. the description of each field instructs it to only generate the one that matches the type. It is less reliable, only getting it right some of the time. I will experiment with the instructions to see if I can make it more reliable. definitely not as using Unions. I also had to modify the sub classes to not have optional fields, for str fields, I give it a default of "" and say Optional in the description.