How to store chat history with query engine?

[d3buggerdan]

@dosu
I have the following query engine:
storage_context = StorageContext.from_defaults(vector_store=vector_store)
Settings.llm = llm
Settings.embed_model = embed_model
index = VectorStoreIndex.from_documents(
[], storage_context=storage_context
)
default_retriever = index.as_retriever(
vector_store_query_mode=VectorStoreQueryMode.HYBRID,
similarity_top_k=10,
)
response_synthesizer = get_response_synthesizer(
response_mode="tree_summarize",
streaming=True,
)
query_engine = RetrieverQueryEngine(
retriever=default_retriever,
response_synthesizer=response_synthesizer,
)
response = query_engine.query

and i want to save the conversation history similar to how the chatengine works. so for example i can ask a question, it will save it to memory and reference it for the state of the chat

[dosubot[bot]]

To store chat history with your query engine setup, you can use the ChatMemoryBuffer class to manage the chat history. Here's how you can integrate it into your existing setup:

1. Initialize the ChatMemoryBuffer: Create an instance of ChatMemoryBuffer to store the chat history.
2. Modify the RetrieverQueryEngine: Update the RetrieverQueryEngine to include methods for storing and retrieving chat history.

Here's an example of how you can achieve this:

from llama_index.legacy.core.llms.types import ChatMessage, MessageRole
from llama_index.legacy.memory import ChatMemoryBuffer
from llama_index.legacy.service_context import ServiceContext
from llama_index.legacy.llm_predictor.base import LLMPredictorType
from llama_index.legacy.core.base_query_engine import BaseQueryEngine
from llama_index.legacy.chat_engine import CondenseQuestionChatEngine

# Initialize the memory buffer
memory_buffer = ChatMemoryBuffer()

# Define your query engine, LLM predictor, and other necessary components
query_engine = BaseQueryEngine()  # Replace with your actual query engine
llm_predictor = LLMPredictorType()  # Replace with your actual LLM predictor

# Initialize the CondenseQuestionChatEngine
chat_engine = CondenseQuestionChatEngine(
    query_engine=query_engine,
    condense_question_prompt=PromptTemplate(DEFAULT_TEMPLATE),
    memory=memory_buffer,
    llm=llm_predictor,
    verbose=True
)

# Example usage
user_message = "What is the weather like today?"
response = chat_engine.chat(user_message)

# Retrieve the chat history
chat_history = chat_engine.chat_history
for message in chat_history:
    print(f"{message.role}: {message.content}")

In this example:

• The ChatMemoryBuffer instance is used to store the chat history.
• The chat method handles user messages and stores them in the memory buffer.
• The chat_history property retrieves and prints the chat history.

This setup ensures that the conversation history is saved and can be referenced for maintaining the state of the chat ^[1].

To continue talking to Dosu, mention @dosu.

[d3buggerdan]

@dosu whenever i use the chat engine compared to the query engine the results are a lot worse and most times it cannot get the relevant information from my index? what can i do?

[d3buggerdan]

@dosu sometimes when i put a question in the chat engine it does not run the calling function query_engine_tool and it gets it wrong but when it does call it it gets it right

[dosubot[bot]]

The SimpleChatEngine is designed for straightforward chat interactions and does not inherently integrate with a query engine, which can lead to less accurate results when compared to a more sophisticated setup that uses a query engine. To improve the accuracy and ensure that the query_engine_tool function is consistently called, you can integrate the EvalQueryEngineTool class into your chat engine setup.

Here’s how you can modify your setup to ensure the query_engine_tool is always called:

1. Integrate EvalQueryEngineTool: Use the EvalQueryEngineTool to handle queries and ensure it is called for each user input.
2. Modify SimpleChatEngine: Update the SimpleChatEngine to use the EvalQueryEngineTool.

Here’s an example of how to achieve this:

from llama_index.core.tools.eval_query_engine import EvalQueryEngineTool
from llama_index.core.base.base_query_engine import BaseQueryEngine
from llama_index.core.evaluation import AnswerRelevancyEvaluator
from llama_index.core.chat_engine.simple import SimpleChatEngine
from llama_index.core.llms.types import ChatMessage
from llama_index.core.memory import ChatMemoryBuffer
from llama_index.core.service_context import ServiceContext

# Initialize the service context
service_context = ServiceContext.from_defaults()

# Initialize the chat memory buffer with a token limit
chat_memory = ChatMemoryBuffer(token_limit=3000)

# Initialize the query engine (replace with your actual query engine)
query_engine = BaseQueryEngine()  # Replace with your actual query engine

# Initialize the EvalQueryEngineTool
query_engine_tool = EvalQueryEngineTool.from_defaults(
    query_engine=query_engine,
    evaluator=AnswerRelevancyEvaluator()
)

# Define a custom SimpleChatEngine that uses EvalQueryEngineTool
class CustomChatEngine(SimpleChatEngine):
    def __init__(self, query_engine_tool, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.query_engine_tool = query_engine_tool

    def get_response(self, input_query: str):
        # Ensure the query_engine_tool is called to retrieve the correct information
        tool_output = self.query_engine_tool.call(input=input_query)
        return tool_output.content

    async def get_response_async(self, input_query: str):
        # Ensure the query_engine_tool is called to retrieve the correct information
        tool_output = await self.query_engine_tool.acall(input=input_query)
        return tool_output.content

# Initialize the CustomChatEngine
chat_engine = CustomChatEngine(
    query_engine_tool=query_engine_tool,
    llm=service_context.llm,
    memory=chat_memory,
    prefix_messages=[],
    callback_manager=service_context.callback_manager
)

# Example usage
user_message = "What is the weather like today?"
response = chat_engine.get_response(user_message)
print(response)

# Retrieve the chat history
chat_history = chat_engine.chat_history
for msg in chat_history:
    print(f"{msg.role}: {msg.content}")

This setup ensures that the query_engine_tool is always called when processing user inputs, leading to more accurate and relevant responses from your index ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[d3buggerdan]

@dosu

chat_engine = index.as_chat_engine(chat_mode="best", llm=llm,query_engine=query_engine, verbose=True)
while True:
user_input = input("You: ")
response = chat_engine.chat(
user_input, tool_choice="query_engine_tool")
print(response)

is there anyway i can print out the raw query it got from the index database? or see what it searches when i make a query to debug why it is not working properly? 

[dosubot[bot]]

Yes, you can print out the raw SQL query to debug why it is not working properly. The SQLStructStoreQueryEngine and BaseSQLTableQueryEngine classes have methods that handle the SQL query execution and return the raw SQL query string.

For SQLStructStoreQueryEngine, you can modify the _query method to print the query_bundle.query_str:

def _query(self, query_bundle: QueryBundle) -> Response:
    """Answer a query."""
    # Print the raw SQL query
    print(f"Raw SQL Query: {query_bundle.query_str}")
    
    response_str, metadata = self._run_with_sql_only_check(query_bundle.query_str)
    return Response(response=response_str, metadata=metadata)

For BaseSQLTableQueryEngine, you can modify the _query method to print the sql_query_str:

def _query(self, query_bundle: QueryBundle) -> Response:
    """Answer a query."""
    retrieved_nodes, metadata = self.sql_retriever.retrieve_with_metadata(query_bundle)

    sql_query_str = metadata["sql_query"]
    # Print the raw SQL query
    print(f"Raw SQL Query: {sql_query_str}")
    
    if self._synthesize_response:
        partial_synthesis_prompt = self._response_synthesis_prompt.partial_format(
            sql_query=sql_query_str,
        )
        response_synthesizer = get_response_synthesizer(
            service_context=self._service_context,
            callback_manager=self._service_context.callback_manager,
            text_qa_template=partial_synthesis_prompt,
            verbose=self._verbose,
        )
        response = response_synthesizer.synthesize(
            query=query_bundle.query_str,
            nodes=retrieved_nodes,
        )
        cast(Dict, response.metadata).update(metadata)
        return cast(Response, response)
    else:
        response_str = "\n".join([node.node.text for node in retrieved_nodes])
        return Response(response=response_str, metadata=metadata)

Additionally, if you are using the LlamaIndex framework, you can set the logging level to DEBUG for more detailed outputs, which can help in debugging. Here is an example of how to set up logging:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

By adding these print statements and setting the logging level to DEBUG, you can see the raw SQL query that is being executed, which can help you debug any issues with the query.

For example, in a Google Drive data connector setup using LlamaIndex, you can set the logging level to DEBUG as shown below:

import logging
import sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

# Set Logging to DEBUG for more detailed outputs
logging.getLogger().setLevel(logging.DEBUG)

This will provide more detailed outputs, including the raw SQL queries, which can be very useful for debugging.

In the context of the SQLJoinQueryEngine from LlamaIndex, you can also enable verbose logging to print the SQL query and response:

def _query_sql_other(self, query_bundle: QueryBundle) -> Response:
    """Query SQL database + other query engine in sequence."""
    # first query SQL database
    sql_response = self._sql_query_tool.query_engine.query(query_bundle)
    if not self._use_sql_join_synthesis:
        return sql_response

    sql_query = (
        sql_response.metadata["sql_query"] if sql_response.metadata else None
    )
    if self._verbose:
        print_text(f"SQL query: {sql_query}\n", color="yellow")
        print_text(f"SQL response: {sql_response}\n", color="yellow")

    # given SQL db, transform query into new query
    new_query = self._sql_augment_query_transform(
        query_bundle.query_str,
        metadata={
            "sql_query": _format_sql_query(sql_query),
            "sql_query_response": str(sql_response),
        },
    )

    if self._verbose:
        print_text(
            f"Transformed query given SQL response: {new_query.query_str}\n",
            color="blue",
        )
    logger.info(f"> Transformed query given SQL response: {new_query.query_str}")
    if self._sql_augment_query_transform.check_stop(new_query):
        return sql_response

    other_response = self._other_query_tool.query_engine.query(new_query)
    if self._verbose:
        print_text(f"query engine response: {other_response}\n", color="pink")
    logger.info(f"> query engine response: {other_response}")

    response_str = self._service_context.llm.predict(
        self._sql_join_synthesis_prompt,
        query_str=query_bundle.query_str,
        sql_query_str=sql_query,
        sql_response_str=str(sql_response),
        query_engine_query_str=new_query.query_str,
        query_engine_response_str=str(other_response),
    )
    if self._verbose:
        print_text(f"Final response: {response_str}\n", color="green")
    response_metadata = {
        **(sql_response.metadata or {}),
        **(other_response.metadata or {}),
    }
    source_nodes = other_response.source_nodes
    return Response(
        response_str,
        metadata=response_metadata,
        source_nodes=source_nodes,
    )

By enabling verbose logging and adding print statements, you can see the raw SQL queries and responses, which can help you debug any issues with the queries ^{[1][2][3][4][5]}.

To continue talking to Dosu, mention @dosu.

[Restodecoca]

Hi, did you get any luck resolving this problem?