Structured Output with Groq: Error code: 400 - {'error': {'message': 'Failed to call a function. Please adjust your prompt. ...'

[ptrem]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

class Code(BaseModel):
    """Code output"""

    prefix: str = Field(description="Description of the code snip. Describes the detailed function usage, the arguments, outputs and gives an example if needed.")
    imports: str = Field(description="Code block import statements")
    function_name: str = Field(description="Name of the function")
    code: str = Field(description="Executable code block using pep 8 code style. Not including import statements and not including applying function")

structured_llm = llm.with_structured_output(Code, include_raw=True)

Description

I'm trying to implement a coding assistant using an LLM with structured outuput (llm.with_structured_output ) similar to the LangChain tutorial. Using the OpenAI API everything works fine, but using the Groq API the model struggles with generating a structured output. Especially with longer context examples.
For me the model output looks quite okay and i don't know why the tool function caller struggles with the output.
I use llama3-70b-8192, using a model with larger context length results in the same error.
Any idea what the problem is or how to implement a fallback for more robust pipeline.
Thanks for your help

The error message:

Error code: 400 -
    { 
        'error': 
            { 
                'message': 
                    "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 
                'type': 'invalid_request_error', 
                'code': 'tool_use_failed', 
                'failed_generation': 
                    '<tool-use>\n 
                        {\n 
                            "tool_calls": 
                                [\n 
                                    {\n 
                                        "id": "pending",\n 
                                        "type": "function",\n 
                                        "function": 
                                            {\n
                                                "name": "Code"\n
                                            },\n
                                        "parameters": 
                                            {\n 
                                                "code": 
                                                 """\n \
                                                    def my_function(args):\n
                                                        \'\'\'\n 
                                                        Function description
                                      
                                                        Args:\n
                                                            arg1 (str): argument 1\n 
                                                            arg2 (dict): argument 2.\n 
                                                        Returns:\n 
                                                            None\n 
                                                        '\'\'\n\n
                                                        # Code comes here\n ...
                                                        
                                                    """,\n \
                                                    "function_name": "my_function",\n
                                                    "prefix": "Description of the function.",\n
                                                    "imports": "import numpy"\n
                                            }\n
                                    }\n
                                ]\n
                        }\n
                    </tool-use>' 
            } 
    }

System Info

Platform: Linux
Python: 3.11.9

langchain 0.2.6
langchain-community 0.2.6
langchain-core 0.2.11
langchain-experimental 0.0.62
langchain-groq 0.1.6
langchain-nomic 0.1.2
langchain-openai 0.1.14
langchain-text-splitters 0.2.2
langgraph 0.1.5
langsmith 0.1.83

[dosubot[bot]]

To resolve the issue with the Groq API struggling to generate structured output for longer context examples in your coding assistant implementation, you can use the .with_structured_output() method to coerce ChatGroq into returning a structured output. Here is an example:

from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_groq import ChatGroq

class Code(BaseModel):
    """Code output"""
    prefix: str = Field(description="Description of the code snip. Describes the detailed function usage, the arguments, outputs and gives an example if needed.")
    imports: str = Field(description="Code block import statements")
    function_name: str = Field(description="Name of the function")
    code: str = Field(description="Executable code block using pep 8 code style. Not including import statements and not including applying function")

chat = ChatGroq(
    temperature=0,
    model="llama3-70b-8192",
    # api_key="" # Optional if not set as an environment variable
)

structured_llm = chat.with_structured_output(Code, include_raw=True)

response = structured_llm.invoke("Generate a Python function with detailed description, imports, and code.")
print(response)

This method leverages the tool-calling functionality to ensure the output matches the specified schema, which can help manage longer context examples more effectively. Additionally, Groq supports JSON mode, which can be specified if needed:

structured_llm = chat.with_structured_output(Code, method="json_mode", include_raw=True)

response = structured_llm.invoke(
    "Generate a Python function with detailed description, imports, and code. Respond in JSON with `prefix`, `imports`, `function_name`, and `code` keys."
)
print(response)

However, be aware that there are known issues with the Groq API's structured output functionality, as indicated by the xfail markers in the integration tests. These tests show that the structured output and tool message histories may fail with errors such as "Failed to call a function. Please adjust your prompt." Therefore, it might be necessary to adjust your prompts or wait for updates from the Groq team to fully resolve these issues ^[1][2][3][4].

To continue talking to Dosu, mention @dosu.

[momegas]

Try using llama-3.3-70b-versatile. According to their docs: https://console.groq.com/docs/tool-use

[VishnuRathore98]

Try using llama-3.3-70b-versatile. According to their docs: https://console.groq.com/docs/tool-use

Yes, it helped. Thanks.

[Rahulsharma4298]

not working for me

[TechEnthusiastArmaan]

it's working now, Thanks!

[falmanna]

Any solution here?

[ptrem]

It seemed to be a problem of the Pydantic parser. Even if the LLM produces an output that looks structured to me. With the required keys and format, the parser isn't possible to extract the information. It seems not to be optimized for the Groq API or Llama3 model as it works much better with the OpenAI API. Especially in long context examples the parser fails. I therefor use now a custom parser. With the exact output structure given in the prompt Llama3 and Gemini-1.5 pro (where the Pydantic structured output doesn't work either) it is possible to structure generate a structured output and it works quite stable.

[sopfi]

I face the same issue. Could you provide your fixed code with your custom parser here? Thanks!

[vjrngn]

Could you please provide the code that fixed this issue? Looking forward. Thanks!

[ptrem]

Yes sure. The custom parser is really depending on your problem and also on the way you prompt the LLM. I advice you to clearly prompt the LLM to generate an output with the structure you need/want. And then write a parser. My way to go was to adjust the prompt to get a reproducible output structure. And then I used chatGPT to write a parser for my outputs given some examples. Works fine for me so far, i hope this helps you.

The following example shows the instruction prompt, query and parser for my problem.

Instruction Prompt

    prompt_instruct_gen = '''You are a coding assistant with expertise to write specific python functions.
        Here is a python code example with a specific task \n ------- \n  {code_example_context} \n ------- \n
        Write the python code for a new set of parameters given in the user question.
        Ensure the following JSON structure
            \{{
                "imports": "...", # of external tools
                "code": """...""", # code block
            \}}
        \n Customize the code answer the following setup and parameters:'''

Query

    query = f'''Write the python code to <<YOUR TASK>>
    Given is the following set of parameters: \n ------- \n  {params_input_file} \n ------- \n
    Insert the generated function code in the <<<FUNCTION CONTENT>>> section. 
    Add a list of the variable parameter names in <<<VARIABLE PARAMETERS>>>.
    Return the entire code block, including the function definition and the content, in executable Python format following PEP 8 style.
    List the function arguments and the required dictionary keys in the function documentation header. 
    IMPORTANT: make sure you use the correct name for the custom import file. Depending on the project name defined in the parameters file
    
    code:"""
    imports ... # make sure of the correct import names depending on the project_name

    def generate_geometry(argument1):
        """
        Description of function.
        Args:
            argument1: Interactive instance ....
                keys: <<<VARIABLE PARAMETERS>>>
        Returns:
            none
        """

        <<<FUNCTION CONTENT>>>
    return
    """
    Respond in JSON with string keys: "imports": "...", "code": """...""".
    '''

Parser

def custom_json_code_ouput_parser(input_string):
    """
    Parser for the code output into JSON format with the keys `imports` and `code`.
    The function searches for the defined keys and the defined JSON file pattern in the input string
    and parses it into a JSON file.

    Arguments:
        input_string (str): code string generated by the LLM
    
    Return:
        parsed_dict (dict): parsed dictionary
    """

    # Define a regex pattern to extract the components
    pattern = re.compile(
        r'"imports":\s*"([^"]+)",\s*'
        r'"code":\s*"""(.*?)"""\s*}', re.DOTALL
    )
    
    try:
        match = pattern.search(input_string)
        if not match:
            raise ValueError(f"Input string does not match the expected format! {input_string}")
    except:
        pass

        pattern = re.compile(
        r'"imports":\s*"([^"]+)",\s*'
        r'"code":\s*"(.*?)"\s*}', re.DOTALL
    )
    
    try:
        match = pattern.search(input_string)
        if not match:
            raise ValueError(f"Input string does not match the expected format! {input_string}")
    except:
        pass

    imports_raw, code_raw = match.groups()
    imports = imports_raw.replace('\\n', '\n')
    code = (code_raw.replace('\\n', '\n')
            .replace('\\"', '\"'))
    
    # Create the dictionary
    parsed_dict = ParsedOutput({
        "imports": imports,
        "code": code
    })
    
    return parsed_dict

[Ansumanbhujabal]

I am facing the same problem but with JsonOutputToolsParser, PydanticToolsParser , Using
llm = ChatGroq(model="llama3-8b-8192")
Is there any other solution available other than writing custom parser ?

[ayush-0411]

I am facing same problem but it shows API error, inspite I have set api in environment as a variable.

[Anderson-hrz]

I have a scenario similar to the question. In my case, I also have a desired output of a specific JSON expected by the LLM. I am using Groq for this. That being said, what I’ve observed is that sometimes, depending on the number of output tokens from Groq, I get the error InstructorRetryException: Error code: 400 - {'error': {'message': "Failed to generate JSON. Please adjust your prompt. See failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'json_validate_failed', 'failed_generation' ....

When the number of tokens is less than 2000, my responses are as expected. These input and output token values can be found on page "console groq". Here are the images of those token outputs from the mentioned page.

I hope this helps someone.

[VishnuRathore98]

For me the issue was model-specific, first I was using " Gemma2-9b-It " then switched to "llama-3.3-70b-versatile" as suggested by @momegas, I think it was likely due to how "Gemma2-9b-It" handled tool calls or formatted its output.
Switching to "llama-3.3-70b-versatile" solved it.
Llama 3.3-70b handled tool calls correctly, ensuring compatibility with PydanticToolsParser.

[aryanranderiya]

I am using 70b too and sometimes this issue still occurs

[piyushhhxyzz]

Same, facing this issue a lot with "llama-3.3-70b-versatile"

[MagMueller]

Facing the same with llama-4-maverick-17b-128e-instruct this happens with ChatOpenAI and ChatGroq when using function_calling.

[tradanghi1999]

structured_llm = chat.with_structured_output(YourModel, method="json_mode")
response = structured_llm.invoke(messages)

I tried, and this code works.

[x4080]

I was using vercel ai chat ui sdk and llama-3.3-70b-versatile works for streaming tool calls but llama-4-scout and maverick show the error above

So it seems tool streaming is not supported for llama-4-scout and maverick

[Extarlife]

I am also having this issue. It appears to be the ChatGroq API from langchain. If I call the models above manually they have no problem formatting the tool call correctly. However using the API I get the same issue.

[mdrxy]

Could you provide a code sample?

[Extarlife]

Sure here is an example of my test code it runs fine with ChatOpenAi but does not with ChatGroq:

import os
import sys
from langchain_core.messages import HumanMessage

sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))

from agent.agent import compiled_graph


def test_agent():
    """Test the agent with a simple booking request."""

    # Create a test message that should trigger the booking subgraph
    test_message = HumanMessage(
        content="I need to book a repair appointment for a leaking pipe in the kitchen. "
        "My postcode is SW1A 2AA. I'm available from 2025-07-01T09:00:00 "
        "to 2025-07-31T17:00:00, between 09:00 and 17:00."
    )

    # Create initial state
    initial_state = {
        "messages": [test_message],
        "customer_id": "1",
        "user_info": [{"name": "Test User", "postcode": "SW1A 2AA"}],
    }

    try:
        # Run the agent
        result = compiled_graph.invoke(initial_state)
        print("Agent completed successfully!")
        print("--- Final State of the Graph ---")
        if hasattr(result, "model_dump"):
            print(result.model_dump_json(indent=2))
        else:
            print(result)

    except Exception as e:
        print(f"Error occurred: {e}")
        import traceback

        traceback.print_exc()


if __name__ == "__main__":
    test_agent()

This is the tool binding. As far as I'm aware if tools are bound this way then the tool call structure should be the same:

llm = ChatGroq(
    model="llama-3.3-70b-versatile",
    api_key=token,
    temperature=0.2,
)

builder = StateGraph(BaseState)
assistant_runnable = primary_assistant_prompt | llm.bind_tools(primary_assistant_tools)

compiled_graph = builder.compile()

Here is a sample output from each API:

Successful tool call llm = ChatOpenAI :

AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_wu0DpgDhQAaEs9Sb_5hAdg', 'function': {'arguments': '{"request":{"name":"Test User","postcode":"SW1A 2AA","call_info1":"leaking pipe in the kitchen","country":"GB","date_from":"2025-07-01","date_to":"2025-07-31","time_from":"09:00","time_to":"17:00","duration":60}}', 'name': 'search_appointments'}, 'type': 'function', 'index': 0}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 86, 'prompt_tokens': 2284, 'total_tokens': 2370, 'completion_tokens_details': None, 'prompt_tokens_details': None}, 'model_name': 'deepseek/deepseek-chat-v3-0324:free', 'system_fingerprint': None, 'id': 'gen-1752249612-cb83D1br4pbpZOuQLwGe', 'service_tier': None, 'finish_reason': 'tool_calls', 'logprobs': None}

Unsuccessful llm = ChatGroq :

Error occurred: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': '<function=ToBookingRepairAssistant>{"postcode": "SW1A 2AA", "date_from": "2025-07-01", "date_to": "2025-07-31", "time_from": "09:00", "time_to": "17:00", "call_info1": "leaking pipe in the kitchen"}'}}

Please let me know if I'm doing something wrong as I am still very new to langgraph. Or if there is any specific code you need.

[mdrxy]

It might seem trivial, but have you adjusted your prompt/tried different variations? I've found that it is the model itself that can be finicky.

[Extarlife]

I have tried a lot of different prompts the closest I can get to match the required output is:

You are an expert router responsible for analyzing user queries and directing them to the correct tool. Your primary function is to either call a tool with specific, required parameters or to ask the user for the information needed to call a tool.

IMPORTANT: When calling a tool, your entire output MUST be a single JSON object containing a tool_calls list. You MUST NOT include any conversational text,
explanations, or other formats before or after the JSON. The JSON object should be the ONLY content in your response.

INCORRECT FORMAT (DO NOT USE):
<function=ToAssistant>{{...}}</function>

CORRECT FORMAT (USE THIS):

{{
  "tool_calls=" [
    {{
      "name": "ToAssistant",
      "args": {{
        "postcode": "SW1A 2AA",
        "date_from": "2025-07-01T09:00:00",
        "date_to": "2025-07-31T17:00:00",
        "time_from": "09:00",
        "time_to": "17:00",
        "call_info1": "leaking pipe in the kitchen"
      }}
    }}
  ]
}}

Key instructions:

1. The output must be a single JSON object.
2. The JSON object must have a key named tool_calls.
3. The value of tool_calls must be a list of tool call objects.
4. Each tool call object must have a name (string) and args (dictionary).
5. For ToAssistant, the date_from and date_to arguments MUST be in YYYY-MM-DDTHH:MM:SS format.

As I mentioned in an earlier post, if you call Groq API directly it has no issues using my tools correctly.