Using with_structured_output on instances of ChatOllama

[tomragonneau]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Literal, TypedDict

from langchain_community.chat_models import ChatOllama
from langchain_core.pydantic_v1 import BaseModel, Field


class DocumentGraderModel(BaseModel):
    score: Literal['relevant', 'irrelevant'] = Field(
        ...,
        description='Relevance of the document to the question. Either "relevant" or "irrelevant".',
    )


llm = ChatOllama(model='llama3')
llm_with_structured_output = llm.with_structured_output(DocumentGraderModel)

Description

The code above raises a NotImplementedError. This is expected. I wonder:

1. Is there any plan to implement it in the future in this class ?
2. It seems it is possible to use OllamaFunctions to do this. Is it safe ? Is there any plan to move this class from langchain-experimental to langchain-community?

Thanks for the help,
Tom.

System Info

System Information
------------------
> OS:  Windows
> OS Version:  10.0.22621
> Python Version:  3.11.9 | packaged by Anaconda, Inc. | (main, Apr 19 2024, 16:40:41) [MSC v.1916 64 bit (AMD64)]

Package Information
-------------------
> langchain_core: 0.2.8
> langchain: 0.2.5
> langchain_community: 0.2.5
> langsmith: 0.1.77
> langchain_experimental: 0.0.61
> langchain_text_splitters: 0.2.1
> langgraph: 0.0.66

Packages not installed (Not Necessarily a Problem)
--------------------------------------------------
The following packages were not found:

> langserve

[dosubot[bot]]

Hey @tomragonneau! I'm here to help you with any bugs, questions, or contributions while you wait for a human maintainer. Let's dive into the issue together!

1. The with_structured_output method is not currently implemented in the ChatOllama class, which is why you are encountering a NotImplementedError. However, there are ongoing or planned updates to the ChatOllama class in the langchain_community package that include the with_structured_output method. This method is already defined in the OllamaFunctions class, which inherits from ChatOllama ^[1].

2. Yes, it is possible to use OllamaFunctions to achieve structured output. The OllamaFunctions class extends ChatOllama with additional capabilities for tool integration, structured output, and function call handling. It is designed to be safe and provides a robust method for achieving structured output. There are no specific details available about moving this class from langchain-experimental to langchain-community, but using OllamaFunctions is a viable and supported approach for now ^[1][2].

To continue talking to Dosu, mention @dosu.

[Vedant-Bisen]

One option is to import OllamaFunctions from langchain_experimental itself.

Another work around which I am using is definding my own with_structured_output

class Entities(BaseModel):
    """Identifying information about entities."""
    names: List[str] = Field(
        ...,
        description="All the person, organization, or business entities that appear in the text",
)

# Create the prompt template
prompt1 = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are extracting any important entities from the text . "
            "Return only a list of string extracted entities and nothing else"
            "format to return should be something like entity1,entity2...",
            
        ),
        (
            "human",
            "extract information from the following input: {question}",
        ),
    ]
)
def extract_entities(question: str) -> Entities:
    # Format the prompt with the provided question
    formatted_prompt = prompt1.format(question=question)

    # Get the response from the LLM
    response = llm.invoke(formatted_prompt) 

    # Process the response to extract entity names (assuming response is a string)
    entity_names = process_response(response)  
    print(entity_names)

    return Entities(names=entity_names)

def process_response(response: str) -> List[str]:
    return [name.strip() for name in response.split(",") if name.strip()]

Using this allows me to get the response from my llm into a structured class Entities

Hope this helps

[julian24bas]

I figured out, that whith_structured_output function works with ChatOllama using models which support tool use like llama3.1

from typing import Optional, List
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama


class Person(BaseModel):
    """Information about a person."""

    # ^ Doc-string for the entity Person.
    # This doc-string is sent to the LLM as the description of the schema Person,
    # and it can help to improve extraction results.

    # Note that:
    # 1. Each field is an `optional` -- this allows the model to decline to extract it!
    # 2. Each field has a `description` -- this description is used by the LLM.
    # Having a good description can help improve extraction results.
    name: Optional[str] = Field(default=None, description="The name of the person")
    hair_color: Optional[str] = Field(
        default=None, description="The color of the person's hair if known"
    )
    height_in_meters: Optional[str] = Field(
        default=None, description="Height measured in meters"
    )


class Data(BaseModel):
    """Extracted data about people."""

    # Creates a model so that we can extract multiple entities.
    people: List[Person]


# Define a custom prompt to provide instructions and any additional context.
# 1) You can add examples into the prompt template to improve extraction quality
# 2) Introduce additional parameters to take context into account (e.g., include metadata
#    about the document from which the text was extracted.)
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert extraction algorithm. "
            "Only extract relevant information from the text. "
            "If you do not know the value of an attribute asked to extract, "
            "return null for the attribute's value.",
        ),
        # Please see the how-to about improving performance with
        # reference examples.
        # MessagesPlaceholder('examples'),
        ("human", "{text}"),
    ]
)

llm = ChatOllama(model="llama3.1", temperature=0)

runnable = prompt | llm.with_structured_output(schema=Person)

text = "Alan Smith is 6 feet tall and has blond hair. Alan Poe is 3 feet tall and has grey hair."
response = runnable.invoke({"text": text})
print(response)

This works for me. However when I change the schema to the Data class to get a list of all persons in the text instead of a single person, I get this error:

pydantic.v1.error_wrappers.ValidationError: 1 validation error for Data
people
  value is not a valid list (type=type_error.list)

This is whats getting passed to the pydantic model. Maybe it's because the list is passed as a string.

{'people': '[{"name": "Alan Smith", "height": "6 feet", "hair color": "blond"}, {"name": "Alan Poe", "height": "3 feet", "hair color": "grey"}]'}

[avfranco-br]

I figured out, that whith_structured_output function works with ChatOllama using models which support tool use like llama3.1

from typing import Optional, List
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.prompts import ChatPromptTemplate
from langchain_ollama import ChatOllama


class Person(BaseModel):
    """Information about a person."""

    # ^ Doc-string for the entity Person.
    # This doc-string is sent to the LLM as the description of the schema Person,
    # and it can help to improve extraction results.

    # Note that:
    # 1. Each field is an `optional` -- this allows the model to decline to extract it!
    # 2. Each field has a `description` -- this description is used by the LLM.
    # Having a good description can help improve extraction results.
    name: Optional[str] = Field(default=None, description="The name of the person")
    hair_color: Optional[str] = Field(
        default=None, description="The color of the person's hair if known"
    )
    height_in_meters: Optional[str] = Field(
        default=None, description="Height measured in meters"
    )


class Data(BaseModel):
    """Extracted data about people."""

    # Creates a model so that we can extract multiple entities.
    people: List[Person]


# Define a custom prompt to provide instructions and any additional context.
# 1) You can add examples into the prompt template to improve extraction quality
# 2) Introduce additional parameters to take context into account (e.g., include metadata
#    about the document from which the text was extracted.)
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert extraction algorithm. "
            "Only extract relevant information from the text. "
            "If you do not know the value of an attribute asked to extract, "
            "return null for the attribute's value.",
        ),
        # Please see the how-to about improving performance with
        # reference examples.
        # MessagesPlaceholder('examples'),
        ("human", "{text}"),
    ]
)

llm = ChatOllama(model="llama3.1", temperature=0)

runnable = prompt | llm.with_structured_output(schema=Person)

text = "Alan Smith is 6 feet tall and has blond hair. Alan Poe is 3 feet tall and has grey hair."
response = runnable.invoke({"text": text})
print(response)

This works for me. However when I change the schema to the Data class to get a list of all persons in the text instead of a single person, I get this error:

pydantic.v1.error_wrappers.ValidationError: 1 validation error for Data
people
  value is not a valid list (type=type_error.list)

This is whats getting passed to the pydantic model. Maybe it's because the list is passed as a string.

{'people': '[{"name": "Alan Smith", "height": "6 feet", "hair color": "blond"}, {"name": "Alan Poe", "height": "3 feet", "hair color": "grey"}]'}

Hello @julian24bas, have you had any luck yet to fix this issue with lists? I've been facing same issue but couldn't find a solution yet.

[julian24bas]

Hi @avfranco-br unfortunately not. I opened an issue here. As a workaround for now I'm using the JSON-mode which works ok for my purpose, but is most likely not optimal.

from langchain_ollama import ChatOllama
from langchain_core.prompts import ChatPromptTemplate


llm_params = {
        'base_url': 'http://localhost:11434',
        'model': 'llama3.1:70b',
        'num_ctx': 32768,
        'temperature': 0.0,
        'format': "json"
    }

llm = ChatOllama(**llm_params)

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "some background information/instruction for the system",
        ),
        # Optionally add examples to improve results.
        # Please see the how-to about improving performance with
        # reference examples.
        # MessagesPlaceholder('examples'),
        ("human", "{json_schema}\nAnswer using this JSON-Schema..\n{context}"),
    ]
)

chain = prompt | llm

answer = chain.invoke({"context": some_context, "json_schema": SomePydanticModel.model_json_schema()})

[avfranco-br]

Thanks @julian24bas for the reply. Same here, using a similar JSON approach but this one, for your reference, https://python.langchain.com/v0.2/docs/how_to/structured_output/#prompting-and-parsing-model-outputs-directly.

[GuyPaddock]

What if you try the newer version of ChatOllama? Try changing your import of:

from langchain_community.chat_models import ChatOllama

To:

from langchain_ollama import ChatOllama

[tomragonneau]

Hi @GuyPaddock,

Thanks for your message, I'll try this ASAP.

[julian24bas]

I also have this issue. Using the new langchai_ollama package I get None as a result.

For me it is not clear, if this feature is even implemented. On this overview page the feature has a green check. However on the ChatOllama page it has not.

[julian24bas]

Maybe this workaround helps for the moment. However this not optimal for sure.