Cannot create supervisor using models with fallbacks

[lucazappaterra]

Hello,
we are facing a limitation within the following environment configuration

name         : langgraph-supervisor                                                       
 version      : 0.0.27                                                                     
 description  : An implementation of a supervisor multi-agent architecture using LangGraph 

dependencies
 - langchain-core >=0.3.40
 - langgraph >=0.3.5
 - langgraph-prebuilt >=0.1.7
---
name         : langgraph-prebuilt                                                                  
 version      : 0.2.2                                                                               
 description  : Library with high-level APIs for creating and executing LangGraph agents and tools. 

dependencies
 - langchain-core >=0.3.22
 - langgraph-checkpoint >=2.0.10

Why we need it

Our team is currently dealing with an issue that emerges when interacting with Gemini APIs that returns "internal server error" (error 500, reported also in this previous issue).

To avoid it, we are trying to setup a fallback mechanism over the OpenAI client towards a GPT model, using the .with_fallbacks primitive that creates a RunnableWithFallbacks object from a BaseLanguageModel object.

Problems in current implementation

The current setup for create_supervisor is not allowing us to bind tools to a RunnableWithFallbacks object for reasonable logical arguments, as depicted in this issue.

In the following the sections, you can find a proposition for a feature addition which requires changes in both langgraph-supervisor and langgraph-prebuilt.

Our changes

- In create_supervisor

def create_supervisor(
    agents: list[Pregel],
    *,
    ...,
    fallback_models: list[LanguageModelLike] = None
) -> StateGraph:
    ...
    
    if fallback_models:
        fallback_models_with_tools = []
        for fallback_model in fallback_models:
            if _should_bind_tools(fallback_model, all_tools):
                if _supports_disable_parallel_tool_calls(fallback_model):
                    fallback_model = cast(BaseChatModel, fallback_model).bind_tools(
                        all_tools, parallel_tool_calls=parallel_tool_calls
                    )
                else:
                    fallback_model = cast(BaseChatModel, fallback_model).bind_tools(all_tools)
                fallback_models_with_tools.append(fallback_model)

        model = model.with_fallbacks(fallback_models_with_tools)

    ...

    return builder

This change allows to provide a default model and a series of fallback models, both equipped with tools at runtime when creating the supervisor, without running into any issue.

- In create_react_agent

At the same time, note that this alone is not compatible with (at least) the response_format parameter in create_supervisor.
This traces back to create_react_agent, where the output_schema is handled and bound to the model in a separate node.
To fix this, it is sufficient to change the _get_model() function in langgraph-prebuilt/chat_agent_executor.py in the following way, possibly adding a more robust type checking to ensure the model is indeed a BaseChatModel:

from langchain_core.runnables.fallbacks import RunnableWithFallbacks 

def _get_model(model: LanguageModelLike) -> BaseChatModel:
    """Get the underlying model from a RunnableBinding or return the model itself."""
    ...

    if isinstance(model, RunnableWithFallbacks):
        model = model.runnable

    ...
    return model

Conceptual MWE

Here a conceptual approach to the creation of a supervisor with fallbacks included (no agents are provided here, but we tested it work as well with them).

Please notice that this is not including the response_format parameter that requires the create_react_agent change that was previously mentioned

from langchain.chat_models import init_chat_model
from langgraph_supervisor import create_supervisor
from langchain_core.tools import tool

base_model="openai:gemini-2.5-flash"
base_api_key_name="GOOGLE_API_KEY"
base_url="https://generativelanguage.googleapis.com/v1beta/openai/"
fallback_model="openai:gpt-4.1-mini"
fallback_api_key_name="OPENAI_API_KEY"
fallback_url=None

@tool
def sum(a, b):
    """ A simple tool that sums two numbers.    
    """
    return a + b

fallback_llm = init_chat_model(
    api_key=os.getenv("OPENAI_API_KEY"),
    base_url=fallback_url,
    model=fallback_model,
    temperature=0
)

gemini = init_chat_model(
    api_key=os.getenv("GOOGLE_API_KEY"),
    base_url=base_url,
    model=base_model,
    temperature=0
)

supervisor = create_supervisor(
    model=gemini,
    agents=[],
    tools=[sum],
    fallback_models=[fallback_llm], # <--- this is the new parameter
).compile(name="supervisor")