BadRequestError('Error code: 400 - {\'error\': {\'message\': "This model\'s maximum context length is 128000 tokens. However, your messages resulted in 142800 tokens (142586 in the messages, 214 in the functions). Please reduce the length of the messages or functions.", \'type\': \'invalid_request_error\', \'param\': \'messages\', \'code\': \'context_length_exceeded\'}}')

[shubhamnegikellton]

PROBLEM:
When using the langgraph-checkpointer-postgres approach in a conversation, the token limit should be managed efficiently. However, after 5-6 prompts, the conversation unexpectedly reaches the 128k token limit, which shouldn't happen.

Assumption (Actual reason unknown):
In every response, a retriever fetches x no. of documents, which are used as context for generating responses.
This means that with each prompt, a large amount of context (from x no. of documents) is added, significantly increasing token usage.
By the 5th or 6th prompt, the cumulative token count exceeds the 128000 token limit, likely due to this.

EXAMPLE CODE
from psycopg import Connection
from psycopg_pool import ConnectionPool
from psycopg.rows import dict_row
from langgraph.checkpoint.postgres import PostgresSaver

connection_kwargs = {"autocommit": True, "prepare_threshold": 0}

async with AsyncConnectionPool(conninfo=conninfo, max_size=20, kwargs=connection_kwargs) as pool:
graph = create_react_agent(
llm,
build_tools,
messages_modifier=_modify_messages,
checkpointer=AsyncPostgresSaver(pool), # type:ignore[arg-type]
)

        async for event in graph.astream_events(
            {"messages": [("human", search_params.question)]},
            config={"configurable": {"thread_id": conversation_id, "recursion_limit": 20}},
            stream_mode="values",
            version="v2",
        ):

Error Message and Stack Trace (if applicable)

BadRequestError('Error code: 400 - {\'error\': {\'message\': "This model\'s maximum context length is 128000 tokens. However, your messages resulted in 142800 tokens (142586 in the messages, 214 in the functions). Please reduce the length of the messages or functions.", \'type\': \'invalid_request_error\', \'param\': \'messages\', \'code\': \'context_length_exceeded\'}}')Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/__init__.py", line 1874, in astream
    async for _ in runner.atick(
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/runner.py", line 362, in atick
    await arun_with_retry(
  File "/usr/local/lib/python3.12/site-packages/langgraph/pregel/retry.py", line 127, in arun_with_retry
    async for _ in task.proc.astream(task.input, config):
  File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 568, in astream
    async for chunk in aiterator:
  File "/usr/local/lib/python3.12/site-packages/langchain_core/tracers/event_stream.py", line 180, in tap_output_aiter
    first = await py_anext(output, default=sentinel)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/utils/aiter.py", line 76, in anext_impl
    return await __anext__(iterator)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1453, in atransform
    async for ichunk in input:
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1453, in atransform
    async for ichunk in input:
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1453, in atransform
    async for ichunk in input:
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 1016, in astream
    yield await self.ainvoke(input, config, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langgraph/utils/runnable.py", line 224, in ainvoke
    ret = await asyncio.create_task(coro, context=context)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langgraph/prebuilt/chat_agent_executor.py", line 595, in acall_model
    response = await model_runnable.ainvoke(state, config)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 3064, in ainvoke
    input = await asyncio.create_task(part(), context=context)  # type: ignore
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/runnables/base.py", line 5364, in ainvoke
    return await self.bound.ainvoke(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 307, in ainvoke
    llm_result = await self.agenerate_prompt(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 796, in agenerate_prompt
    return await self.agenerate(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 756, in agenerate
    raise exceptions[0]
  File "/usr/local/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 912, in _agenerate_with_cache
    async for chunk in self._astream(messages, stop=stop, **kwargs):
  File "/usr/local/lib/python3.12/site-packages/langchain_openai/chat_models/base.py", line 774, in _astream
    response = await self.async_client.create(**payload)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/openai/resources/chat/completions.py", line 1720, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/openai/_base_client.py", line 1843, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/openai/_base_client.py", line 1537, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/openai/_base_client.py", line 1623, in _request
    return await self._retry_request(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/openai/_base_client.py", line 1670, in _retry_request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/openai/_base_client.py", line 1638, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 128000 tokens. However, your messages resulted in 142800 tokens (142586 in the messages, 214 in the functions). Please reduce the length of the messages or functions.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

Single document looks like this : [Document(metadata={'id': 5, 'relevance_score': 0.9997073, 'source': 'https://test.sharepoint.com/teams/org_sgafnr/Shared%20Documents/Knowledge%20Navigator%20TOR%20Document%20Center/Template%20Documents/EA/07.pdf', 'document_id': '38cc4fed-e7c0-40ca-a5ba-33455e726897', 'document_title': '07 SPRSS', 'document_category': 'AFN'}, page_content='Jiangxi Ganzhou Rural Vitalization and Comprehensive Environment Improvement (RRP PRC 53049)\n\nSUMMARY POVERTY REDUCTION AND SOCIAL STRATEGY\n\nCountry:\n\nPeople’s Republic of China\n\nProject Title:\n\nJiangxi Ganzhou Rural Vitalization and Comprehensive Environment Improvement Project\n\nLending/Financing Modality:\n\nProject loan\n\nDepartment/ Division:\n\nEast Asia Department / Environment, Natural Resources and Agriculture Division\n\nI.\n\nPOVERTY AND SOCIAL ANALYSIS AND STRATEGY'), Document(metadata={'id': 14, 'relevance_score': 0.99963796, 'source': 'https://test.sharepoint.com/teams/org_sgafnr/Shared%20Documents/Knowledge%20Navigator%20TOR%20Document%20Center/Template%20Documents/EA/10Change%20Assessment.pdf', 'document_id': '2e2b5fbd-ecdb-456e-b0d2-400b3974a25e', 'document_title': '10 Climate Change Assessment', 'document_category': 'AFNR'}, page_content='Jiangxi Ganzhou Rural Vitalization and Comprehensive Environment Improvement Project (RRP PRC 53049)\n\nCLIMATE CHANGE ASSESSMENT\n\nI.\n\nBASIC PROJECT INFORMATION\n\nProject Title:\n\nJiangxi Ganzhou Rural Vitalization and Comprehensive Environment Improvement\n\nProject Cost ($ million): 455.67 Location: Sector: Theme:\n\nGanzhou Municipality, Jiangxi Province, People’s Republic of China (PRC) Agriculture, Nature Resources and Rural Development Natural resource management, ecological and environmental protection, sustainable rural development The project will result in the following outcome: living environment of rural areas along upper reaches of the Gan River improved. The project will strengthen the environmental monitoring and enforcement, and the urban-rural integration in the Ganzhou Municipality. The project has the following outputs: Output 1: Institutional capacity and knowledge for environmental management enhanced; Output 2: Green development and financing mechanisms piloted; Output 3: Rural waste and sanitation management improved; and Output 4: Water and soil conservation practices improved.\n\nBrief Description:\n\nSource: Asian Development Bank.\n\nII.\n\nSUMMARY OF CLIMATE CHANGE FINANCE\n\nProject Financing\n\nClimate Finance\n\nSource Asian Development Bank Ordinary capital resources (regular loan) Counterpart Governments Total Source: Asian Development Bank.\n\nAmount ($ million)\n\n200.00\n\n255.67 455.67\n\nAdaptation ($ million)\n\n44.77\n\n19.15 63.92\n\nMitigation ($ million)\n\n35.17\n\n18.77 53.94\n\nIII.\n\nSUMMARY OF CLIMATE RISK SCREENING AND ASSESSMENT\n\nA. Sensitivity of Project Components to Climate or Weather Conditions and the Sea Level\n\n1. Rural environment infrastructure. Higher temperature may affect the operation of wastewater facilities; more precipitation and intense storms may increase the risk of flood damage to rural infrastructures.\n\n2. River rehabilitation. More precipitation and intense storms may increase the risk of river flooding; temperature and precipitation pattern variation may change watershed runoff in quantity and quality leading to ecological degradation.\n\n3. Agriculture. Large temperature difference may cause crop degradation; more precipitation and intense storms may cause increased flood damage to the agricultural facilities.\n\n4. Forestry. High evaporation and low precipitation for long time may cause drought, plant death and increased risk for wildfire.\n\n5. Water and soil conservation. Peak intense precipitation may intensify soil erosion and landslides.\n\nB. Climate Risk Screening\n\n2\n\nAccording to the statistical analyses of historical climate data in Ganzhou from 1960 to 2019, the annual mean temperature showed an overall increasing trend and raised from 19.14°C in 1960 to 19.96°C in 2019. The average temperatures in both summer and winter increased during 1960–2019, and summer temperature increased at a fast rate after 2000. The number of high temperature days (>35°C) in summer steadily increased in general during 1960–2019 in Ganzhou and nearly doubled in 2019 compared with the number in 1960. The number of low temperature days (<0°C) in winter steadily decreased in general during 1960–2019 in Ganzhou and nearly halved in 2019 compared with the number in 1960. It can be concluded that Ganzhou is experiencing more extreme heats in summer and a warmer winter with less extremely cold days. The annual precipitation fluctuated largely from 1960 to 2019 with a slightly increasing trend in general, while the number of rainy days in winter season steadily decreased.\n\nThe climate risk screening identified the following risks which may affect the project: (i) the temperature increase, especially the increased extreme high temperature events in the summer, may cause higher evaporation and more frequent droughts; and (ii) the increase in rainfall variability and the intensity of extreme rainfall events may potentially increase the flood and drought risk.\n\nClimate Risk Classification: medium\n\nC. Climate Risk and Adaptation Assessment\n\nThe temperature and precipitation variations during 2021–2100 in the region of Ganzhou Municipality were projected for future climate change analysis under three Representative Concentration Pathway (RCP) scenarios (RCP 4.5, RCP 6.0, and RCP 8.5). The projected climate change trend is based on the analysis for decadal climate conditions from 2021 to 2100 with reference to baseline period of 1989 to 2019.')]

**Note: 
Here, The document_retriever fetches x documents every time (looks like the above example), and this entire conversation, along with the retrieved documents, is being passed as context to the LLM. This causes the context window to exceed the maximum token limit, leading to issues when using the checkpointer to provide conversation history.**

Also, checked this code for checkpointer, and inside the value["channel_values"] which is consuming each ToolMessage:

  async def aget_tuple(self, config: RunnableConfig) -> Optional[CheckpointTuple]:
      """Get a checkpoint tuple from the database asynchronously.

      This method retrieves a checkpoint tuple from the Postgres database based on the
      provided config. If the config contains a "checkpoint_id" key, the checkpoint with
      the matching thread ID and "checkpoint_id" is retrieved. Otherwise, the latest checkpoint
      for the given thread ID is retrieved.

      Args:
          config (RunnableConfig): The config to use for retrieving the checkpoint.

      Returns:
          Optional[CheckpointTuple]: The retrieved checkpoint tuple, or None if no matching checkpoint was found.
      """
      thread_id = config["configurable"]["thread_id"]
      checkpoint_id = get_checkpoint_id(config)
      checkpoint_ns = config["configurable"].get("checkpoint_ns", "")
      if checkpoint_id:
          args: tuple[Any, ...] = (thread_id, checkpoint_ns, checkpoint_id)
          where = "WHERE thread_id = %s AND checkpoint_ns = %s AND checkpoint_id = %s"
      else:
          args = (thread_id, checkpoint_ns)
          where = "WHERE thread_id = %s AND checkpoint_ns = %s ORDER BY checkpoint_id DESC LIMIT 1"

      async with self._cursor() as cur:
          await cur.execute(
              self.SELECT_SQL + where,
              args,
              binary=True,
          )

          async for value in cur:
              return CheckpointTuple(
                  {
                      "configurable": {
                          "thread_id": thread_id,
                          "checkpoint_ns": checkpoint_ns,
                          "checkpoint_id": value["checkpoint_id"],
                      }
                  },
                  await asyncio.to_thread(
                      self._load_checkpoint,
                      value["checkpoint"],
                      value["channel_values"],
                      value["pending_sends"],
                  ),
                  self._load_metadata(value["metadata"]),
                  (
                      {
                          "configurable": {
                              "thread_id": thread_id,
                              "checkpoint_ns": checkpoint_ns,
                              "checkpoint_id": value["parent_checkpoint_id"],
                          }
                      }
                      if value["parent_checkpoint_id"]
                      else None
                  ),
                  await asyncio.to_thread(self._load_writes, value["pending_writes"]),
              )

Package Information:
langchain = "0.3.11"
langchain-community = "0.3.11"
langchain-experimental = "0.3.3"
langchain-openai = "0.2.12"
langchain-postgres = "0.0.12"
langgraph = "0.2.58"
psycopg = { extras = ["binary"], version = "3.2.3" }
psycopg-pool = "3.2.3"
sqlalchemy = { version = "2.0.36", extras = ["asyncio"] }
sqlmodel = "0.0.22"
asyncpg = "0.30.0"
langgraph-checkpoint-postgres = "2.0.15"

Help me to resolve the above issue, by including the ToolMessage for current prompt only instead of complete chat ToolMessage. if you have similiar issue please respond to this.

[cruz636]

Hi @shubhamnegikellton ! I've encountered the same problem in my app. I have an script that interacts every 15 minutes with my chat-interface for testing. Since I am using threads as short-memory storage, it is reaching the context size limit because it sends the whole history.

There are some ideas that I am testing for this:
a) summarize the history so it keeps small ( the downside is that it will start to hallucinate if the content gets too big )
b) use a new thread for the User at the begining of his interaction with the chat
c) move to a long-term memory storage ( this would fit better on your app IMHO )

Useful links:
1- short-term memory
2- long-term memory

[abhishekshingadiya]

or we can do one more thing when we reached this exception. send last call to llm to generate question(which have all information from first question to last) to create new thread. (if your app is not having high dependency on each answer)

[iamnikhildogra]

I am also facing this issue, I am using a multi-agent supervisor architecture, short term memory for storage as well, I have reviewed various methods like pre_model_hook to trim messages but its giving same error again and again.

@cruz636 I also tried the second approach as well, not working.