docs/tutorials/chatbot/

[giscus[bot]]

docs/tutorials/chatbot/

This guide assumes familiarity with the following concepts:

https://python.langchain.com/docs/tutorials/chatbot/

[paraglondhe098]

Very helpful explanation.

[somendrew]

Explaination made it look easy! Thanks to community.

[Nevertolate83]

Hey guys is this safe to run on my MAC Desktop ?

[somendrew]

I don't think Mac has enough RAM, Use jupyter notebook and t4 kernal , it'll be so smooth and fast

[Murdock135]

It's safe to run on any computer.

[2980897531]

You can use ChatGPT's API to run it. It will not use your Mac's GPU

[a3fckx]

Loved the easy to understand explanation.

[ruksharahmed7]

Suppose I have a FastAPI backend where this chatbot code is present, the frontend is in react, and the chatbot uses the app.stream method for response streaming. In this case, do I use web-socket to enable streaming of AI response from the FastAPI backend to the React frontend?

[Fritskee]

Followed your exact tutorial, but when making the conversion to async there is an error that comes up:

NotImplementedError: Unsupported message type: <class 'coroutine'>
For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/MESSAGE_COERCION_FAILURE 

Below is the example of the code that throws it:

from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, MessagesState, StateGraph

# Define a new graph
workflow = StateGraph(state_schema=MessagesState)


# Define the function that calls the model
async def call_model(state: MessagesState):
    response = model.ainvoke(state["messages"])
    return {"messages": response}


# Define the (single) node in the graph
workflow.add_edge(START, "model")
workflow.add_node("model", call_model)

# Add memory
memory = MemorySaver()
app = workflow.compile(checkpointer=memory)

config = {"configurable": {"thread_id": "abc123"}}

query = "Hi! I'm Bob."

input_messages = [HumanMessage(query)]
output = await app.ainvoke({"messages": input_messages}, config)
output["messages"][-1].pretty_print()  # output contains all messages in state

[Fritskee]

nevermind, fixed it:

import asyncio
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
from langgraph.checkpoint.memory import MemorySaver
from langgraph.graph import START, MessagesState, StateGraph

# Initialize the model
model = ChatOpenAI(model="gpt-3.5-turbo")

# Define a new graph
workflow = StateGraph(state_schema=MessagesState)

# Define the async function that calls the model
async def call_model(state: MessagesState):
    response = await model.ainvoke(state["messages"])
    return {"messages": response}


workflow.add_edge(START, "model")
workflow.add_node("model", call_model)


memory = MemorySaver()
app = workflow.compile(checkpointer=memory)


config = {"configurable": {"thread_id": "abc123"}}


async def chat():
    query = "Hi! I'm Bob."
    input_messages = [HumanMessage(query)]
    output = await app.ainvoke({"messages": input_messages}, config)
    print("\nBot:", output["messages"][-1].content)

    query = "What's my name?"
    input_messages = [HumanMessage(query)]
    output = await app.ainvoke({"messages": input_messages}, config)
    print("\nBot:", output["messages"][-1].content)


await chat()

[Akshat-Rastogi-6]

Awesome Tutorial!!! Loved it.
I just had this one problem in Managing Conversation History that while I was trying to add human message in the messages list :

input_message = messages + [HumanMessage(content=query)]

it was adding the message at 0 index of the message list shifting the System message to the 1st index; this constantly gave me the following error:

'''
ValueError: Unexpected message with type <class 'langchain_core.messages.system.SystemMessage'> at the position 1.
'''

to solve this, I had to change the trimmer configuration and made

trimmer = trim_messages(
    max_tokens=100,  
    strategy="last",
    token_counter=model,  
    include_system=False, #here
    allow_partial=False,
    start_on="human",
)

then it started working. I guess either the system message needs to be added separately or add the HumanMessage() to the messages, but in some other way.

Please see this and let me know.

[Marsax110]

Very good documentation, but I've encountered a small problem.

In the last example of Streaming, based on the code above, I cannot achieve the effect in the document, that is, streaming output.

This is because when defining the model generation above, invoke was used instead of stream.

As a result, during the subsequent generation, the output is done in whole blocks, and the "|" only appears once, at the end.

# Create a state graph for managing conversation flow
workflow = StateGraph(state_schema=State)

model_OpenAI = ChatOpenAI(model="gpt-4o", api_key=os.getenv("OPENAI_API_KEY"))


def call_model(state: State):
    """Function to generate model responses
    
    Args:
        state (State): Current state containing message history and language settings
        
    Returns:
        dict: Dictionary containing model response messages
    """
    # Use trimmer to trim message history to stay within token limits
    trimmed_messages = trimmer.invoke(state["messages"])
    
    # Generate prompt using template with trimmed messages and language setting
    prompt = prompt_template.invoke(
        {"messages": trimmed_messages, "language": state["language"]}
    )
    
    # Call language model to generate response
    response = model_OpenAI.invoke(prompt, config={"callbacks": [langfuse_handler]})
    
    # Return model response wrapped in messages list
    return {"messages": [response]}


# Add workflow edge connecting START node to model node
workflow.add_edge(START, "model")

# Add model node and bind call_model handler function
workflow.add_node("model", call_model)

# Create memory saver for storing conversation state
memory = MemorySaver()

# Compile workflow with checkpointer for state checkpoints
app = workflow.compile(checkpointer=memory)


config = {"configurable": {"thread_id": "abc789"}}
query = "What company are you a big model for?"
language = "English"

input_messages = [HumanMessage(query)]
for chunk, metadata in app.stream(
    {"messages": input_messages, "language": language},
    config,
    stream_mode="messages",
):
    if isinstance(chunk, AIMessage):  # Filter to just model responses
        print(chunk.content, end="|")

> I am a language model created by OpenAI.|

But if I switch to .stream here, I get the error again

    # Call language model to generate response
    response = model_OpenAI.stream(prompt, config={"callbacks": [langfuse_handler]})

---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[119], line 6
      3 language = "English"
      5 input_messages = [HumanMessage(query)]
----> 6 for chunk, metadata in app.stream(
      7     {"messages": input_messages, "language": language},
      8     config,
      9     stream_mode="messages",
     10 ):
     11     if isinstance(chunk, AIMessage):  # Filter to just model responses
     12         print(chunk.content, end="|")

File ~/Library/Caches/pypoetry/virtualenvs/9091-ell-6QbIK8Ul-py3.10/lib/python3.10/site-packages/langgraph/pregel/__init__.py:1655, in Pregel.stream(self, input, config, stream_mode, output_keys, interrupt_before, interrupt_after, debug, subgraphs)
   1649     get_waiter = None  # type: ignore[assignment]
   1650 # Similarly to Bulk Synchronous Parallel / Pregel model
   1651 # computation proceeds in steps, while there are channel updates
   1652 # channel updates from step N are only visible in step N+1
   1653 # channels are guaranteed to be immutable for the duration of the step,
   1654 # with channel updates applied only at the transition between steps
-> 1655 while loop.tick(input_keys=self.input_channels):
   1656     for _ in runner.tick(
   1657         loop.tasks.values(),
   1658         timeout=self.step_timeout,
   (...)
...
--> 333     raise NotImplementedError(msg)
    335 return _message

NotImplementedError: Unsupported message type: <class 'generator'>
For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/MESSAGE_COERCION_FAILURE 

[HoangHPham]

Hi, I'm facing to the same situation. I think now the returned chunk is a whole sentence. Is it right? Or have you fix this?

[Marsax110]

I didn't solve this problem, and I ended up choosing not to use LangChian, but to use OpenAI's SDK directly to achieve my goal

[ddww]

This is my environment. The model is qwen2.5. It is deployed through vllm. When using the examples in the tutorial, the following error is reported. Can anyone who has encountered this problem help me solve it?

model = ChatOpenAI(model="Qwen2.5-32B-Instruct-AWQ",
openai_api_key="xxxxxx",
openai_api_base="http://localhost:9983/v1",
temperature=0.01)
trimmer = trim_messages(
max_tokens=65,
strategy="last",
token_counter=model,
include_system=True,
allow_partial=False,
start_on="human",
)
messages = [
SystemMessage(content="you're a good assistant"),
HumanMessage(content="hi! I'm bob"),
AIMessage(content="hi!"),
HumanMessage(content="I like vanilla ice cream"),
AIMessage(content="nice"),
HumanMessage(content="whats 2 + 2"),
AIMessage(content="4"),
HumanMessage(content="thanks"),
AIMessage(content="no problem!"),
HumanMessage(content="having fun?"),
AIMessage(content="yes!"),
]
result=trimmer.invoke(messages)
print(result)

The following is the error:
NotImplementedError: get_num_tokens_from_messages() is not presently implemented for model cl100k_base.

[Ghost-Unison]

I have also encountered the same situation. I am using Deepseek's model, and here is my code:：

from dotenv import load_dotenv
load_dotenv()
from langchain.chat_models import init_chat_model
model = init_chat_model("deepseek-chat", model_provider="deepseek")
...
trimmer = trim_messages(
    max_tokens=65,
    strategy="last",
    token_counter=model,
    include_system=True,
    allow_partial=False,
    start_on="human",
)

messages = [
    SystemMessage(content="you're a good assistant"),
    HumanMessage(content="hi! I'm bob"),
    AIMessage(content="hi!"),
    HumanMessage(content="I like vanilla ice cream"),
    AIMessage(content="nice"),
    HumanMessage(content="whats 2 + 2"),
    AIMessage(content="4"),
    HumanMessage(content="thanks"),
    AIMessage(content="no problem!"),
    HumanMessage(content="having fun?"),
    AIMessage(content="yes!"),
]

trimmer.invoke(messages)

This is the error message：

NotImplementedError: get_num_tokens_from_messages() is not presently implemented for model cl100k_base. See https://platform.openai.com/docs/guides/text-generation/managing-tokens for information on how messages are converted to tokens.
...
File E:\MyWorkSpace\.venv\Lib\site-packages\langchain_openai\chat_models\base.py:1316, in BaseChatOpenAI.get_num_tokens_from_messages(self, messages, tools)
   1314     tokens_per_name = 1
   1315 else:
-> 1316     raise NotImplementedError(
   1317         f"get_num_tokens_from_messages() is not presently implemented "
   1318         f"for model {model}. See "
   1319         "https://platform.openai.com/docs/guides/text-generation/managing-tokens"  # noqa: E501
   1320         " for information on how messages are converted to tokens."
   1321     )
   1322 num_tokens = 0
   1323 messages_dict = [_convert_message_to_dict(m) for m in messages]
...

Then it was discovered that the method get_num_tokens_from_messages() in .venv/Lib/site-packages/langchain_openai/chat_models/base.py had restrictions on the name of the model, allowing only GPT models to be used

def get_num_tokens_from_messages(
        self,
        messages: list[BaseMessage],
        tools: Optional[
            Sequence[Union[dict[str, Any], type, Callable, BaseTool]]
        ] = None,
    ) -> int:
...
        if model.startswith("gpt-3.5-turbo-0301"):
            # every message follows <im_start>{role/name}\n{content}<im_end>\n
            tokens_per_message = 4
            # if there's a name, the role is omitted
            tokens_per_name = -1
        elif model.startswith("gpt-3.5-turbo") or model.startswith("gpt-4"):
            tokens_per_message = 3
            tokens_per_name = 1
        else:
            raise NotImplementedError(
                f"get_num_tokens_from_messages() is not presently implemented "
                f"for model {model}. See "
                "https://platform.openai.com/docs/guides/text-generation/managing-tokens"  # noqa: E501
                " for information on how messages are converted to tokens."
            )
...

How can I implement trim_comessage if I want to use other models?

[jssfy]

use count_tokens_approximately as token_counter instead of the models other than (gpt-3.5, gpt-4)

trimmer = trim_messages(
max_tokens=65,
strategy="last",
token_counter=count_tokens_approximately,
include_system=True,
allow_partial=False,
start_on="human",
)

[Ghost-Unison]

ty!

[Puggo1145]

If you want to see the streaming effect clearly, you need to set flush=True in your print() function.

By default, Python buffers the output for efficiency. When you use end="", Python removes the automatic newline character and keeps buffering the content until it encounters a newline or the program ends. This is why you're seeing all output appear at once.

To fix this, modify your print statement like this:
print(chunk.content, end="", flush=True)