How to use/configure LLM w/ configurable alternative + .with_structured_output

[hschmied]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_openai import AzureChatOpenAI, ChatOpenAI
from langchain.schema.runnable import ConfigurableField
from langchain_core.pydantic_v1 import BaseModel, Field

class JustForTesting(BaseModel):
    test: str = Field("test")

llm = ChatOpenAI().configurable_alternatives(
    ConfigurableField(id="llm"),
    default_key="oai",
    azure=AzureChatOpenAI(
        openai_api_version="...",
        azure_endpoint="..."
    )
)

llm.with_structured_output(JustForTesting)
## >> AttributeError: 'RunnableConfigurableAlternatives' object has no attribute 'with_structured_output'

####################################

# llm = ChatOpenAI()
# ##OR##
# llm = AzureChatOpenAI(
#     openai_api_version="...",
#     azure_endpoint="...",
# )

# llm.with_structured_output(JustForTesting)
# > works as expected for either one of the two

Description

I am using configurable_alternatives for my chains & graphs, which works quite well.

Now I'd also like to use '.with_structured_output(...)' which is a method on the llm-object, but by using configurable_alternatives, I have a 'RunnableConfigurableAlternatives'-object, which doesn't know about '.with_structured_output'

I guess I could make it work, if could extract the llm out of the runnable first and then add the '.with_structured_output', but I am stuck on how to do that. Or does anyone have a better/different idea?

Thanks!

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 22.6.0: Tue Nov 7 21:48:06 PST 2023; root:xnu-8796.141.3.702.9~2/RELEASE_X86_64
Python Version: 3.11.2 (main, Mar 27 2023, 18:39:42) [Clang 14.0.6 ]

Package Information

langchain_core: 0.1.31
langchain: 0.1.12
langchain_community: 0.0.28
langsmith: 0.1.25
langchain_cli: 0.0.20
langchain_experimental: 0.0.54
langchain_openai: 0.0.8
langchain_text_splitters: 0.0.1
langchainhub: 0.1.15
langgraph: 0.0.28
langserve: 0.0.37

[dosubot[bot]]

Based on the information provided, it seems like you're trying to use the with_structured_output(...) method on an instance of RunnableConfigurableAlternatives. However, as per the context provided, the RunnableConfigurableAlternatives class does not have a with_structured_output(...) method, which is why you're encountering an AttributeError.

The with_structured_output(...) method is typically used with instances of the StructuredPrompt class or similar classes that support structured output. This method modifies the behavior of the language model or the runnable it is being connected to, so that the output produced by the language model is structured according to a predefined schema.

In your case, it seems like you're trying to use the with_structured_output(...) method to structure the output of the RunnableConfigurableAlternatives instance. However, this class does not inherently support structured output, which is why the method call is failing.

If you want to structure the output of the RunnableConfigurableAlternatives instance, you might need to do this manually after the instance has been run, by parsing the output and structuring it according to your JustForTesting schema. Alternatively, you could consider wrapping the RunnableConfigurableAlternatives instance in a custom runnable that supports structured output, and then use the with_structured_output(...) method on this wrapper.

Here's a rough example of how you might do this:

class StructuredRunnable(Runnable):
    def __init__(self, runnable, schema):
        self.runnable = runnable
        self.schema = schema

    def run(self, *args, **kwargs):
        result = self.runnable.run(*args, **kwargs)
        return self.schema(**result)

llm = StructuredRunnable(
    ChatOpenAI().configurable_alternatives(
        ConfigurableField(id="llm"),
        default_key="oai",
        azure=AzureChatOpenAI(
            openai_api_version="...",
            azure_endpoint="..."
        )
    ),
    JustForTesting
)

In this example, StructuredRunnable is a custom runnable that wraps another runnable and structures its output according to a given schema. You can then use this StructuredRunnable to wrap your RunnableConfigurableAlternatives instance and structure its output according to your JustForTesting schema.

Please note that this is just a rough example and might need to be adjusted based on your specific use case and the exact behavior of the RunnableConfigurableAlternatives class and the ChatOpenAI class.

I hope this helps! If you have any further questions or if something is unclear, please let me know.

Sources

• libs/core/langchain_core/prompts/structured.py
• libs/core/langchain_core/runnables/configurable.py
• libs/langchain/langchain/output_parsers/structured.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[eyurtsev]

@hschmied this currently does not work. We need to fix up our handling with RunnableBinding to re-surface the methods of the underlying wrapped entity.

[codekiln]

Cross-linking related #19279

[codekiln]

Possibly also related to #23419; see https://github.com/langchain-ai/langchain/pull/23419/files#diff-fb3c0238f3cd4c88f5cdf2c46682191dbefcefba8b8bb7830aff415191e305a9

[codekiln]

• langchain[minor]: Generic configurable model by baskaryan · Pull Request #23419 · langchain-ai/langchain has been merged, and the July 25, 2004 changelog blog post is here.
• The docs now have a How to init any model in one line | 🦜️🔗 LangChain guide, which explicitly mentions with_structured_output under "Using a configurable model declaratively"

I don't yet understand how to switch between two providers that specify their models with different keys; I'm not quite sure I get it. Will post back here when I determine whether it's possible to use this function to switch at runtime between, say, AzureChatOpenAI and BedrockChat, even though the former and the latter don't use the same key for specifying the model.

[codekiln]

@dosu-bot can you give me an example of how, using the new langchain.chat_models.base.init_chat_model function, I can make a chain that has an LLM configured to use structured output and which lets me configure at runtime whether to use azure_openai and bedrock? Here's some starter code, I'd like you to fill in the spot where I mentioned you in a comment:

class AnswerWithJustification(pydantic.v1.BaseModel):
    """An answer to the user question along with justification for the answer."""
    answer: str
    justification: str

sample_prompt = "What weighs more a pound of bricks or a pound of feathers"

... 
# dosu-bot you fill in this part
... 

azure_openai_result = azure_openai_configured_structured_llm.invoke(sample_prompt)
print(f"{azure_openai_result=}")

bedrock_claude_result = bedrock_claude_configured_structured_llm.invoke(sample_prompt)
print(f"{bedrock_claude_result=})

[codekiln]

Ok, here's an example: #25283 (reply in thread) - note that the example there is for configurable fields, not configurable alternatives.