Examining the Query Generated When Using SelfQueryRetriever

[RERobbins]

How can I examine the metadata filter generated when using a SelfQueryRetriever?

Previously, when I ran a chain that included a SelfQueryRetriever the query, including the filter, was printed. But that does not seem to be happening now.

Here is a portion of the relevant code.

from langchain import PromptTemplate, LLMChain
from langchain.chains import RetrievalQA
from langchain.retrievers.self_query.base import SelfQueryRetriever
from langchain.chains.query_constructor.base import AttributeInfo

metadata_field_info = [
    AttributeInfo(
        name="organization",
        description="The company or organization that created the document.  It describes that company's policy.",
        type="string or list[string]",
    ),
    AttributeInfo(
        name="title",
        description="The title of the document",
        type="string",
    ),
    AttributeInfo(
        name="url",
        description="The url for the document",
        type="string",
    ),
]
document_content_description = "A policy"

retriever = SelfQueryRetriever.from_llm(
    LLM,
    vectordb,
    document_content_description,
    metadata_field_info,
    verbose=True,
    enable_limit=True,
)

chain = RetrievalQA.from_chain_type(
    llm = LLM,
    retriever = retriever,
    return_source_documents = True,
)

A month or so ago, when I would send a prompt to this chain, information about the filter and resulting query would be printed.

chain("How does the company name Meta protect my data?")

would result in something like the following being printed:

query='Meta data protection' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='organization', value='Meta') limit=None

That no longer seems to be the case.

I want to examine the metadata filter that is generated.

[dosubot[bot]]

🤖

Hello,

From the code you've shared, it seems like you've already set the verbose attribute to True when creating the SelfQueryRetriever instance. This should enable the logging of the generated structured query.

retriever = SelfQueryRetriever.from_llm(
    LLM,
    vectordb,
    document_content_description,
    metadata_field_info,
    verbose=True,  # Enable verbose mode
    enable_limit=True,
)

The verbose attribute is used in the _get_relevant_documents and _aget_relevant_documents methods of the SelfQueryRetriever class. When verbose is True, the generated structured query is logged to the logger:

if self.verbose:
    logger.info(f"Generated Query: {structured_query}")

However, please note that you need to have your logger configured to print or store the logs for inspection. If you're not seeing the logs, it might be due to the logger configuration. You can check your logger settings to ensure that the logs are being correctly outputted.

If you've checked your logger settings and you're still not seeing the logs, there might be an issue with the LangChain codebase. In that case, please provide more details about your environment (Python version, LangChain version, etc.) and any error messages you're seeing, so we can help troubleshoot the issue.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/retrievers/self_query/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[RERobbins]

@dosu-bot I have not done anything with logger configuration. It was my understanding that the default behavior would cause logging events to be printed on the console. That does not seem to be happening. What should I do?

…

On Wed, Oct 11, 2023 at 7:51 AM dosu-beta[bot] ***@***.***> wrote: *🤖* Hello, From the code you've shared, it seems like you've already set the verbose attribute to True when creating the SelfQueryRetriever instance. This should enable the logging of the generated structured query. retriever = SelfQueryRetriever.from_llm( LLM, vectordb, document_content_description, metadata_field_info, verbose=True, # Enable verbose mode enable_limit=True, ) The verbose attribute is used in the _get_relevant_documents and _aget_relevant_documents methods of the SelfQueryRetriever class. When verbose is True, the generated structured query is logged to the logger: if self.verbose: logger.info(f"Generated Query: {structured_query}") However, please note that you need to have your logger configured to print or store the logs for inspection. If you're not seeing the logs, it might be due to the logger configuration. You can check your logger settings to ensure that the logs are being correctly outputted. If you've checked your logger settings and you're still not seeing the logs, there might be an issue with the LangChain codebase. In that case, please provide more details about your environment (Python version, LangChain version, etc.) and any error messages you're seeing, so we can help troubleshoot the issue. I hope this helps! Let me know if you have any other questions. Sources - libs/langchain/langchain/retrievers/self_query/base.py <https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/self_query/base.py> ------------------------------ This response is meant to be useful and save you time. It is *not meant to be a precise solution*, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot <https://github.com/dosu-bot>. — Reply to this email directly, view it on GitHub <#11659 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYUR2OVYG765CUON3U7CI3X62I4PANCNFSM6AAAAAA535T2HM> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[dosubot[bot]]

🤖

Hello,

Based on the code snippet you provided, you have correctly set the verbose attribute to True when creating the SelfQueryRetriever instance. This should indeed enable the logging of the generated structured query.

However, there seems to be a known issue with the LangChain framework where the verbose output isn't printed when the chain.arun() is called. This issue was discussed in Langchain isn't verbose any more [run_manager gets dropped]. The issue seems to be related to the async code and was noticed in the LLMChain, where the run_manager is passed to other functions inside generate() but not inside agenerate().

A user suggested a patch that could be used until the LangChain team fixes the issue. You might want to check out the discussion in the issue for more details.

As for the default logging configuration in the LangChain framework, I wasn't able to find specific information within the repository. However, the Python logging module, which LangChain likely uses, has a default level of WARNING. This means that only events of this level and above will be tracked, unless the logging package is configured differently.

If you have further questions or need more assistance, feel free to ask.

Best,
Dosu

Sources

• Langchain isn't verbose any more [run_manager gets dropped]
• libs/langchain/langchain/retrievers/self_query/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[RERobbins]

@dosu-bot Thank you. When I go to earlier versions of LangChain everything works as expected. This truly does appear to be a logging bug.

…

On Wed, Oct 11, 2023 at 8:25 AM dosu-beta[bot] ***@***.***> wrote: *🤖* Hello, Based on the code snippet you provided, you have correctly set the verbose attribute to True when creating the SelfQueryRetriever instance. This should indeed enable the logging of the generated structured query. However, there seems to be a known issue with the LangChain framework where the verbose output isn't printed when the chain.arun() is called. This issue was discussed in Langchain isn't verbose any more [run_manager gets dropped] <#4329>. The issue seems to be related to the async code and was noticed in the LLMChain, where the run_manager is passed to other functions inside generate() but not inside agenerate(). A user suggested a patch that could be used until the LangChain team fixes the issue. You might want to check out the discussion in the issue for more details. As for the default logging configuration in the LangChain framework, I wasn't able to find specific information within the repository. However, the Python logging module, which LangChain likely uses, has a default level of WARNING. This means that only events of this level and above will be tracked, unless the logging package is configured differently. If you have further questions or need more assistance, feel free to ask. Best, Dosu Sources - Langchain isn't verbose any more [run_manager gets dropped] <#4329> - libs/langchain/langchain/retrievers/self_query/base.py <https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/self_query/base.py> ------------------------------ This response is meant to be useful and save you time. It is *not meant to be a precise solution*, but rather a starting point for your own research. Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant. If you want to continue the conversation, start your reply with @dosu-bot <https://github.com/dosu-bot>. — Reply to this email directly, view it on GitHub <#11659 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEYUR2MJXLT3IJUFE4JMO6LX62M5JANCNFSM6AAAAAA535T2HM> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing a detailed description of your issue. Based on the context provided, it seems like there have been similar issues in the past related to logging verbosity in LangChain.

The verbose attribute controls the verbosity of the specific object it is set on. In your case, it is set on the SelfQueryRetriever instance. This means that only the inputs and outputs to the SelfQueryRetriever instance are printed, along with any additional callbacks calls made specifically by that object. If you want to see the logging events of the entire chain when you run chain("How does the company name Meta protect my data?"), you should ensure that the global langchain.verbose value is set to True or explicitly set verbose=True on the Chain object, not just on the SelfQueryRetriever instance. You can find more information about this in the debugging guide and the debugging snippet.

However, based on the similar issues you've mentioned, it seems like there might be a bug in the current version of LangChain that affects logging. One of the issues, Langchain isn't verbose any more [run_manager gets dropped], suggests that the issue might be related to async code and the run_manager not being passed to other functions inside agenerate(). A patch was suggested by user @blob42 that could be used until the issue is fixed.

Unfortunately, I couldn't find any specific changes to the logging system in the latest version of LangChain that could potentially cause the reported issue. It would be helpful if you could provide more information about the specific version of LangChain you're using where the issue occurs, and any other relevant details about your environment.

I hope this helps, and please let me know if you have any other questions or need further clarification.

Sources

• Langchain isn't verbose any more [run_manager gets dropped]
• RetrievalQAWithSourcesChain acall does not write fromatted prompt to terminal if verbose=True
• docs/docs/guides/debugging.md
• docs/snippets/modules/chains/how_to/debugging.mdx
• libs/langchain/langchain/chains/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.