Using CheckPointer makes the tool call break.

[ozankasikci]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.

Example Code

agent_executor = chat_agent_executor.create_tool_calling_executor(
    chat_model,
    tools,
    checkpointer=SqliteSaver.from_conn_string(path_from_root("ignore/checkpoint-data/checkpoints.sqlite")),
)

Error Message and Stack Trace (if applicable)

[2024-05-24 12:54:57,627] ERROR in app: Exception on /chat [POST]
Traceback (most recent call last):
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/PycharmProjects/assistant-ai/app/routes/chat_routes.py", line 13, in chat
    response = agent_executor.invoke(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1245, in invoke
    for chunk in self.stream(
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 834, in stream
    _panic_or_proceed(done, inflight, step)
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1334, in _panic_or_proceed
    raise exc
  File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 66, in run_with_retry
    task.proc.invoke(task.input, task.config)
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 2368, in invoke
    input = step.invoke(
            ^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3832, in invoke
    return self._call_with_config(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1492, in _call_with_config
    context.run(
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/config.py", line 347, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3706, in _invoke
    output = call_func_with_variable_args(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/config.py", line 347, in call_func_with_variable_args
    return func(input, **kwargs)  # type: ignore[call-arg]
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langgraph/prebuilt/chat_agent_executor.py", line 400, in call_model
    response = model_runnable.invoke(messages)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 4396, in invoke
    return self.bound.invoke(
           ^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 170, in invoke
    self.generate_prompt(
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 599, in generate_prompt
    return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 456, in generate
    raise e
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 446, in generate
    self._generate_with_cache(
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 671, in _generate_with_cache
    result = self._generate(
             ^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/langchain_openai/chat_models/base.py", line 522, in _generate
    response = self.client.create(messages=message_dicts, **params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/openai/_utils/_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/openai/resources/chat/completions.py", line 590, in create
    return self._post(
           ^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/openai/_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/openai/_base_client.py", line 921, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "/Users/ozan/Library/Caches/pypoetry/virtualenvs/assistant-ai-atkT-oAf-py3.12/lib/python3.12/site-packages/openai/_base_client.py", line 1020, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_v5dzbL3NBrFZ7Z4xZExU9AUv", 'type': 'invalid_request_error', 'param': 'messages.[8].role', 'code': None}}
127.0.0.1 - - [24/May/2024 12:54:57] "POST /chat HTTP/1.1" 500 -

Description

I'm trying to use langchain to build an assistant AI agent.

If I don't use checkpointer, tool calls work just fine.

If I set a SqliteSaver as the checkpointer, the tool calls fail.

System Info

langchain==0.2.0
langchain-community==0.2.0
langchain-core==0.2.0
langchain-openai==0.1.7
langchain-text-splitters==0.2.0

platform=mac

python 3.12

[hinthornw]

Thanks for raising! Tool handling is a tricky aspect of how updates and resumptions interact with the LLMs' tool calling APIs that I should probably explicitly address in the docs - once an ai message with tool calls is put in the state, the next time the LLM is called, a tool message with the corresponding id must be present in the chat history . If the tool is actually called, then that handles it. If we skip (i.e., with interrupts), then we need to add a tool message to satisfy this API requirement (can say "OK" or anything else)

Without seeing your code, if I had to guess, I would assume perhaps there is an interrupt but then when you resume you are inserting a human message or skipping the tool message entirely.

Action item for me: make How-To on managing state updates with tool calling

I think this is what is causing your issue (based on personal experience) but feel free to follow up if there's actually a break that you identify

[ozankasikci]

Hi @hinthornw !

Indeed, after reading your reply, I checked and the issue occurs if there is an interrupted tool call.

Please let me know if I can do anything to help resolve the issue.

[hinthornw]

Gonna convert this to a discussion for now since it seems to not be a bug in the API so much as a tricky implementation detail we ought to improve in our documentation/guides

[suribe06]

@ozankasikci Have you been able to make any progress in solving this? Have you tried manually inserting the AIMessage with the associated tool_call_id that failed?

[ozankasikci]

@suribe06 unfortunately no, I moved away from Langchain after seeing how unresponsive the contributors are for this kind of problem that occurs for a lot of Langchain users.

[suribe06]

@ozankasikci I'm sorry to hear that, I'm working on a possible solution for this problem, if you want when I have it ready I can share it with you.

[kishorkrishnak]

@suribe06 Can you share it with me please?

[hinthornw]

I'd put in a pre_model hook if you're using the high-level agent API! Check for a dangling ai imessage and append a 'ignored' tool message if it was dropped.

If you're writing your own graph, you can do the same thing :) Basically the challenge as I see it is you go:

Run 1: human -> ai with tool calls -> [interrupt/pause waiting for approval]; end of turn checkpoint has a conversation history of [user->ai w/ tools];
Run 2:
a) if you resume (using Command(resume=...) or a null input, depending on what you're blocking for), then it will resume at the location you interrupted at and then continue, adding a tool message to complete the pair.

b) If you aren't resuming (just send a regular human message), you aretelling the graph to start at the entrypoint (clear the interrupt). This pushes the 'user' message on top of the previous history and makes it invalid. If you then driectly pass this to the model, API, it will likely throw an error.

If you are accidentally doing (b), you can either stop diong that (by using a resume), or handle it as an implicit 'ignore'/cancel action by pushing a tool message onto the history in the entry node (or pre model hook if you are using the agent abstraction), etc.

Basically, the orchestrator is doing exactly what your code is instructing it todo, but I could see how the outcome (of creating an invalid history) could be frustrating.

Example:

def pre_model_hook(state, filler="IGNORED: superseded by new user input"):
    msgs = state.get("messages", [])
    if len(msgs) < 2 or not isinstance( msgs[-1], HumanMessage):
        return state

    seen_ids = set()
    ai_idx = None

    for i in range(len(msgs) - 2, -1, -1):
        m = msgs[i]
        if isinstance(m, ToolMessage):
           seen_ids.add(m.tool_call_id)
        elif isintance(m, AIMessage):
            ai_idx = i
            break

    if ai_idx is None:
        return state

    ai = msgs[ai_idx]
    tool_calls =ai.tool_calls or []
    expected = [tc.get("id") for tc in tool_calls if tc.get("id")]
    missing = [tcid for tcid in expected if tcid not in seen_ids]
    if not missing:
        return state

    fillers = [{"role": "tool", "tool_call_id": tcid, "content": filler} for tcid in missing]
    return {**state, "messages": msgs[:-1] + fillers + [msgs[-1]]}
    ```