Massive Parallel Graph Execution with Batch APIs

[EliezerIsrael]

I've put together a LangGraph StateGraph that takes a foreign language word and its context, and iterates on some dictionary tool calls until it determines the right dictionary entry. It works well, word by word.

I want to run it against a huge corpus - many hundreds of thousands of words. I'd like to use a batch API, like the Anthropic Batch API. That can execute thousands of calls in parallel, but asynchronously.

Effectively, I'd like to run many thousands of graphs simultaneously, running the collection of current LLM calls across all of the graphs, then collecting the responses some time later, advancing each graph to the next step, until they all resolve.

I can imagine a way to handle it - run the graphs until they reach an LLM call, batch the calls, resume execution when the batch completes, rinse and repeat.

I imagine this is a repeating pattern for large scale async loads. Is there any kind of scaffolding for this already built? I'm going to solve this one way or the other - if I'm barking up the right tree here, I'm happy to take some architectural notes to make it more broadly usable.

[nfcampos]

Hi, this is a question we think a bunch about.
Are you looking to run the graph thousands of times in parallel, or a single run of the graph with thousands of nodes in parallel?
For the first, I'd suggest looking at LangGraph Platform, we've spent a bunch of time figuring out how to run LangGraph at scale, with horizontal scaling etc. https://langchain-ai.github.io/langgraph/concepts/scalability_and_resilience/
For the second I'd suggest looking at our experimental Kafka adapter, designed to run graphs in a distributed fashion https://github.com/langchain-ai/langgraph/tree/main/libs/scheduler-kafka

[EliezerIsrael]

Thank you @nfcampos. Following your note, I just looked into both options.
I'm looking to run it thousands of times in parallel, and I'm particularly interested in being able to use the Batch API. My jobs aren't time sensitive, but they rack of the cost, and I very quickly run afoul of rate limiting. It's using the Batch API in concert with LangGraph that I'm interested in. I haven't seen that discussed on LangGraph Platform. Am I missing something?