How to return citations from Agent? #27208

[DEENUU1]

Hi, I am using Agent from Langchain and would like to return inline citations in a text. For example: "Messi is a.... [1]" where [1] is a citation and I can display it.

Currently, my application allows me to return output from tools so I have access to the whole fetched content from different tools but I would like to add inline citations with core information.
Is it possible to do it?

I saw the documentation but I don't understand how to add it to Agent.

def get_llm(provider: str, model: str, temperature: float) -> Any:
    if provider == "openai":
        return ChatOpenAI(model=model, temperature=temperature, openai_api_key=settings.OPENAI_API_KEY, streaming=True)
    else:
        raise ValueError(f"Unsupported provider: {provider}")


def get_prompt() -> ChatPromptTemplate:
    return ChatPromptTemplate.from_messages(
        [
            (
                "system",
                "You are a helpful assistant.",
            ),
            ("placeholder", "{chat_history}"),
            ("human", "{input}"),
            ("placeholder", "{agent_scratchpad}"),
        ]
    )

def get_tools() -> list:
    serper_tool = get_google_serper_tool()
    return [
        # serper_tool,
        GoogleDriveTool(),
        CurrentTimeTool(),
        WeatherTool(),
        AskNewsSearch(),
        WikipediaQueryRun(api_wrapper=WikipediaAPIWrapper()),
        # YahooFinanceNewsTool(),
        ScraperTool(),
        BraveSearchWrapper.from_api_key(api_key=settings.BRAVE_API_KEY, search_kwargs={"count": 5})
    ]

async def get_agent(session_id: str, llm: LLM, user_id: str) -> RunnableWithMessageHistory:
    llm = get_llm(llm.provider, model=llm.model, temperature=0)
    tools = get_tools()
    prompt = get_prompt()

    agent = create_tool_calling_agent(llm, tools, prompt)

    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

    agent_with_chat = RunnableWithMessageHistory(
        agent_executor,
        lambda session_id: get_session_history(session_id, user_id),
        input_messages_key="input",
        history_messages_key="chat_history",
    )

    return agent_with_chat


agent = await get_agent(str(session_id), llm_model, str(user_id))

async for event in agent.astream_events(
                    {"input": full_input},
                    config={"configurable": {"session_id": session_id, "user_id": str(user_id)}},
                    version="v1",
            ):
                kind = event["event"]
                if kind == "on_chat_model_stream":
                    content = event["data"]["chunk"].content
                    if content:
                        await websocket.send_json({"type": "chunk", "content": content, "message_id": ai_message.id})
                        full_response += content

                elif kind == "on_tool_start":
                    await websocket.send_json(
                        {"type": "tool_start", "tool_name": event["name"], "run_id": event["run_id"]}
                    )

                elif kind == "on_tool_end":
                    tool_output = event.get("data", {}).get("output", "")
                    processed_output = process_tool_output(tool_output)

                    tool_output_obj = create_tool_output(db, ToolOutputInputSchema(
                        content=json.dumps(processed_output),
                        tool_name=event["name"],
                        run_id=event["run_id"],
                        message_id=ai_message.id
                    ))
                    tool_output_objs.append(tool_output_obj.model_dump())

                    await websocket.send_json(
                        {"type": "tool_end", "tool_name": event["name"], "run_id": event["run_id"]}
                    )
                    await websocket.send_json(
                        {"tool_object": tool_output_obj.model_dump(), "message_id": ai_message.id})

[dosubot[bot]]

I found a similar discussion on how to return citations from an Agent in LangChain ^[1]. Here's how you can add inline citations to text using the Agent:

1. Setup: Ensure you have the necessary dependencies installed and your environment is set up.

2. Create the Agent: Use a language model like ChatOpenAI and create a citation fuzzy match chain.

3. Invoke the Agent: Run the chain with your question and context to generate an answer with citations.

4. Add Citations: The chain will return the answer with highlighted citations, showing which parts of the context are referenced.

Here's an example setup:

from langchain.chains import create_citation_fuzzy_match_chain
from langchain_openai import ChatOpenAI

question = "What did the author do during college?"
context = """
My name is Jason Liu, and I grew up in Toronto Canada but I was born in China.
I went to an arts highschool but in university I studied Computational Mathematics and physics. 
As part of coop I worked at many companies including Stitchfix, Facebook.
I also started the Data Science club at the University of Waterloo and I was the president of the club for 2 years.
"""

llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
chain = create_citation_fuzzy_match_chain(llm)
result = chain.run(question=question, context=context)

def highlight(text, span):
    return (
        "..."
        + text[span[0] - 20 : span[0]]
        + "*"
        + "\033[91m"
        + text[span[0] : span[1]]
        + "\033[0m"
        + "*"
        + text[span[1] : span[1] + 20]
        + "..."
    )

for fact in result.answer:
    print("Statement:", fact.fact)
    for span in fact.get_spans(context):
        print("Citation:", highlight(context, span))
    print()

This setup will allow you to generate text with inline citations based on the agent's observations and grounded answers ^[2][3][4][5].

To continue talking to Dosu, mention @dosu.

[dosubot[bot]]

To integrate inline citations into your current setup using create_tool_calling_agent and AgentExecutor, you can follow these steps:

1. Define a Structured Output: Use Pydantic to define a schema that includes citations.

   from pydantic import BaseModel, Field
   from typing import List
   
   class Citation(BaseModel):
       source_id: int = Field(..., description="The ID of the source.")
       quote: str = Field(..., description="The quote from the source.")
   
   class CitedAnswer(BaseModel):
       answer: str = Field(..., description="The answer to the user question.")
       citations: List[Citation] = Field(..., description="Citations from the sources.")

2. Modify the LLM: Use the .with_structured_output method to ensure the language model returns the structured output.

   llm = get_llm(llm.provider, model=llm.model, temperature=0).with_structured_output(CitedAnswer)

3. Update the Agent: When creating the agent, ensure it uses the modified LLM.

   agent = create_tool_calling_agent(llm, tools, prompt)

4. Process the Output: In your event loop, handle the structured output to extract and display citations.

   async for event in agent.astream_events(...):
       if kind == "on_chat_model_stream":
           content = event["data"]["chunk"].content
           if content:
               # Process the structured output to extract citations
               cited_answer = CitedAnswer.parse_raw(content)
               answer_text = cited_answer.answer
               citations = cited_answer.citations
               # Format and send the response with citations
               formatted_response = f"{answer_text} " + " ".join(f"[{c.source_id}]" for c in citations)
               await websocket.send_json({"type": "chunk", "content": formatted_response, "message_id": ai_message.id})

This approach will allow you to generate responses with inline citations using your existing agent setup ^[1][2].

To continue talking to Dosu, mention @dosu.

[DEENUU1]

@dosu i am getting this error "Error in session 8: This function requires a .bind_tools method be implemented on the LLM."

[dosubot[bot]]

The error you're encountering indicates that the .bind_tools method needs to be implemented on the LLM you're using. Here's how you can do it:

1. Define Your Tools: Use the @tool decorator or Pydantic classes to define your tools.

   Example with @tool decorator:

   from langchain_core.tools import tool
   from pydantic import BaseModel, Field
   
   class OperationInput(BaseModel):
       a: int = Field(description="First number")
       b: int = Field(description="Second number")
   
   @tool("add", args_schema=OperationInput, return_direct=True)
   def add(a: int, b: int) -> int:
       return a + b
   
   tools = [add]

2. Bind Tools to the LLM: Use the .bind_tools method to bind these tools to your LLM.

   llm = get_llm(llm.provider, model=llm.model, temperature=0)
   llm_with_tools = llm.bind_tools(tools)

3. Use the Modified LLM: Ensure that the agent uses the LLM with tools bound.

   agent = create_tool_calling_agent(llm_with_tools, tools, prompt)

This setup will ensure that your LLM can utilize the tools you've defined ^[1][2].

To continue talking to Dosu, mention @dosu.

[DEENUU1]

@dosu

Error in session 7: This function requires a .bind_tools method be implemented on the LLM.

from typing import List

from langchain.agents import AgentExecutor, create_tool_calling_agent
from langchain_core.runnables import RunnableWithMessageHistory
from pydantic import BaseModel, Field

from ai.history import get_session_history
from ai.llm import get_llm
from ai.prompt import get_prompt
from ai.tools.tools import get_tools
from utils.get_llm_model import LLM


class Citation(BaseModel):
    source_id: int = Field(..., description="The ID of the source.")
    quote: str = Field(..., description="The quote from the source.")

class CitedAnswer(BaseModel):
    answer: str = Field(..., description="The answer to the user question.")
    citations: List[Citation] = Field(..., description="Citations from the sources.")

async def get_agent(session_id: str, llm: LLM, user_id: str) -> RunnableWithMessageHistory:
    tools = get_tools(user_id)

    llm = get_llm(llm.provider, model=llm.model, temperature=0.7).bind_tools(tools)
    llm_with_tools = llm.with_structured_output(CitedAnswer)
    print(f"get_agent user_id: {user_id}")
    prompt = get_prompt()

    agent = create_tool_calling_agent(llm_with_tools, tools, prompt)

    agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

    agent_with_chat = RunnableWithMessageHistory(
        agent_executor,
        lambda session_id: get_session_history(session_id, user_id),
        input_messages_key="input",
        history_messages_key="chat_history",
    )

    return agent_with_chat

            async for event in agent.astream_events(
                    {"input": full_input},
                    config={"configurable": {"session_id": session_id, "user_id": str(user_id)}},
                    version="v1",
            ):
                kind = event["event"]
                if kind == "on_chat_model_stream":
                    content = event["data"]["chunk"].content
                    if content:
                        # Process the structured output to extract citations
                        cited_answer = CitedAnswer.parse_raw(content)
                        answer_text = cited_answer.answer
                        citations = cited_answer.citations
                        # Format and send the response with citations
                        formatted_response = f"{answer_text} " + " ".join(f"[{c.source_id}]" for c in citations)
                        await websocket.send_json(
                            {"type": "chunk", "content": formatted_response, "message_id": ai_message.id})

[dosubot[bot]]

To resolve the error "This function requires a .bind_tools method be implemented on the LLM," ensure that the language model (llm) you are using has a bind_tools method implemented. Here's how you can address this:

1. Check LLM Implementation: Ensure your language model class has a bind_tools method. If it doesn't, you need to implement it.

2. Use a Compatible LLM: If your current LLM doesn't support bind_tools, consider using a compatible model like ChatOpenAI that does.

Here's an example of how you might implement the bind_tools method:

# Ensure your LLM class has a bind_tools method
class MyLanguageModel(BaseLanguageModel):
    def bind_tools(self, tools):
        # Implement the method to bind tools
        pass

# Use the LLM with create_tool_calling_agent
model = MyLanguageModel()
llm_with_tools = model.bind_tools(tools)
agent = create_tool_calling_agent(llm_with_tools, tools, prompt)

Make sure the bind_tools method is correctly implemented to handle the tools you are passing ^[1][2].

To continue talking to Dosu, mention @dosu.