How to stop agent execution?

[elupanov]

What is the correct way to stop (abort, terminate) graph execution at a specific point in time? It is a common scenario in applications when a user wants to stop generation and regenerate a response later. I haven't found any solution for LangGraph in Python. The only information I found is the implementation for the JS version.

[david1542]

I'm also interested in this functionality. I've opened another discussion here with some reproducible code.

[hinthornw]

If i understand the question correction, this is solved by the "double texting" native feature of LangGraph python.

As you could imagine, we implement this using queue workers and by cancelling the tasks upon request. The way to implement yourself would be via asyncio tasks. There's an example here.

[elupanov]

This feature is available only in LangGraph Cloud, not in the open source version. However, it may be possible to implement something similar on top of LangGraph execution. The only unclear aspect is whether it is possible to stop execution in the middle of node processing.

[hinthornw]

The code that david shared is actually pretty close to correct (I'm following up with them in the linked discussion).

Assuming values are checkpointed and you cancel a task in the middle of the node, you'd end up with the last checkpoint being the state of your graph before it entered the node itself.

Are you looking to go even further and capture the intermediate state of the node when it's cancelled (so you can e.g., save a partial LLM completion)?

[elupanov]

For my case, starting over from a checkpoint before a node is acceptable. I'm just wondering if it is possible to stop inside a node, as you mentioned, for example, when the LLM is generating content. Thanks for the hint!

[BadrElfarri]

@hinthornw Could you elaborate on the case where the LLM completion is to be aborted, I want to abort the completion upon UI stop button has been clicked

[elupanov]

I've came up to the following solution pretty similar to the davids's solution. I have a FastAPI server with LangGraph agent.
In the example blow I just left code only related to stopping functionality. Functions have more logic.

The full version is here https://github.com/akveo/ai-cookbook/blob/main/agent/app/server.py

from typing import AsyncGenerator, Dict
import asyncio

# Track active connections
active_connections: Dict[str, asyncio.Event] = {}

@app.post("/agent/stop")
async def stop_agent(request: Request):
    """Endpoint for stopping the running agent."""
    
    if thread_id in active_connections:
        active_connections[thread_id].set()
        
@app.post("/agent")
async def agent(request: Request):

   stop_event = asyncio.Event()
   active_connections[thread_id] = stop_event
   
   async def generate_events() -> AsyncGenerator[dict, None]:
        try:
            async for chunk in graph.astream(
                input,
                config,
                stream_mode=["debug", "messages", "updates", "custom"],
            ):
                if stop_event.is_set():
                    break

                yield {}
                
        finally:
            if thread_id in active_connections:
                del active_connections[thread_id]
                
       return EventSourceResponse(generate_events())

[BadrElfarri]

Thanks I will test it out 😄

[XiaoLiuAI]

This solution does not work when multiple server instances are running. For example, fastapi server with worker > 1. Each worker is an independent process and thread_id could be sent to another instance.

It is very depressed that langgraph does not support such a common functionality.

[satyaprakash1729]

In a distributed setup, we definitely need a pub-sub system.
A background thread that's running in each instance will subscribe to cancel/abort event and do exactly as described above. It'll be a no-op in all other instances except one.

[yongge007]

Thank you for your idea, I have found a python-kafka-pub-sub repo, share with your guys.https://github.com/dcortesnet/Python-Kafka-pub-sub/blob/main/server-pub/main.py