Instructor does not support multiple tool calls

[agamm]

• This is actually a bug report.
• I am not getting good LLM Results
• I have tried asking for help in the community on discord or discussions and have not received a response.
• I have tried searching the documentation and have not found an answer.

What Model are you using?

• gpt-3.5-turbo
• gpt-4-turbo
• gpt-4
• Other (please specify): Qwen2-72B-Instruct with litellm

Describe the bug
When having an unrelated response the structured output fails with:

AssertionError: Instructor does not support multiple tool calls, use List[Model] instead.

To Reproduce
BerriAI/litellm#4699

Expected behavior
Return an empty response and a warning.

Screenshots
If applicable, add screenshots to help explain your problem.

[akarshghale]

Getting the same error with together ai async client when using the new llama 3.1 7B instruct.

[akarshghale]

Any updates on this?

[ssonal]

I can see that the deepinfra issue from here and the llama 3.1B issue from here are syntax related issues because both providers are deviating from openAI compatible function calling.

Deepinfra only supports "tool_choice" = 'auto' and Together.ai has its own unique requirement for Llama 3.1 function calling.

[akarshghale]

I can see that the deepinfra issue from here and the llama 3.1B issue from here are syntax related issues because both providers are deviating from openAI compatible function calling.

Deepinfra only supports "tool_choice" = 'auto' and Together.ai has its own unique requirement for Llama 3.1 function calling.

So what do you think can be done about it?

There's no workaround other than updating the lib?

[akarshghale]

I think function calling and JSON mode are 2 different things in together ai right?

https://docs.together.ai/docs/json-mode

How can I use json mode instead of tool mode?

[akarshghale]

Yeah so using JSON_SCHEMA mode works just fine.

[agamm]

I'm not using together.ai, why do you bring it up?

[jxnl]

https://python.useinstructor.com/concepts/parallel/?h=parallel#understanding-parallel-function-calling

[agamm]

Can you briefly explain why is this related?

[AT-main]

I was getting the same error when using Intstructor with a llamafile model. I ended up with the following script that worked for me.

from instructor import Mode
import instructor
from pydantic import BaseModel
from openai import OpenAI


# Define your desired output structure
class UserInfo(BaseModel):
    name: str
    age: int


# Patch the OpenAI client
client = instructor.patch(OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed"), mode=Mode.JSON_SCHEMA)

# Extract structured data from natural language
user_info = client.chat.completions.create(
    model="google_gemma-3-4b-it-Q6_K",
    response_model=UserInfo,
    messages=[{"role": "user", "content": "John Doe is 30 years old."}],
)

print(user_info.name)
#> John Doe
print(user_info.age)
#> 30