Filter Qdrant retriever by metadata

[ggnicolau]

Dear colleagues,

How can I use a filter by the metadata I've sent to Qdrant in this piece of code?

from langchain.vectorstores import Qdrant
from qdrant_client import QdrantClient

# connect to the vector database
client = QdrantClient(
    url="https://saci-nlp-qdrant.wittyplant-0ac62cca.eastus.azurecontainerapps.io:443",
)

doc_store = Qdrant(client=client, collection_name="tickers", embeddings=embeddings)

retriever = doc_store.as_retriever(
    search_type="similarity_score_threshold",
    search_kwargs={"score_threshold": 0.5, "k": 10},
)

My metadata at Qdrant currently is something like:

{
  "document_details": {
    "external_data": null,
    "stock_ticker": "ASURB.MX",
    "title": "ASUR is ASUR and nothing else is ASUR except ASUR"
  },
  "document_id": "c9d119812dc69ea5c220a68db276516c",
  "email_details": {
    "email_number": "350",
    "from_address": "Bill Gates <[email protected]>",
    "received_date": "10/23/2023, 20:11:02",
    "subject": "ASUR (Buy): In-line Q3 results; MDP review in focus",
    "to_address": "None"
  },
  "file_details": {
    "Author": "Microsoft",
    "CreationDate": "D:20240120051118-03'00'",
    "ModDate": "D:20240120051118-03'00'",
    "Producer": "mPDF 8.1.2",
    "Title": "Report #99644"
  },
  "processed_date": "2024-01-20T15:38:50.349666+00:00",
  "source": "document-99644.pdf"
}

First I want to try to filter by "received_date", in a time window (after something and before something).

[dosubot[bot]]

🤖

Hello again, @ggnicolau! I hope your coding journey has been smooth since we last interacted. Let's dive into this new challenge you've brought.

To filter by the "received_date" metadata in a specific time window using Qdrant, you can use the filter parameter in the search methods. The filter parameter accepts a MetadataFilter which is a dictionary where the keys are the metadata fields and the values are the conditions to filter by.

Here is an example of how you can filter by a specific time window:

from datetime import datetime, timedelta

# Define the time window
start_date = datetime.now() - timedelta(days=7)  # One week ago
end_date = datetime.now()  # Now

# Convert dates to timestamps
start_timestamp = int(start_date.timestamp())
end_timestamp = int(end_date.timestamp())

# Define the filter
date_filter = {
    "received_date": {
        "range": {
            "from": start_timestamp,
            "to": end_timestamp
        }
    }
}

# Use the filter in a search method
documents = qdrant.similarity_search_with_score(
    query="your query",
    k=10,
    filter=date_filter
)

In this example, qdrant is an instance of the Qdrant class. The similarity_search_with_score method is used to search for documents similar to the given query. The filter parameter is used to filter the results by the "received_date" metadata field. The range filter is used to specify a range of values. The from and to parameters define the start and end of the range, respectively. The dates are converted to timestamps because Qdrant uses timestamps for date fields. The k parameter specifies the number of documents to return.

Please note that this example assumes that the "received_date" metadata field exists and is of type date (stored as a timestamp). If the field does not exist or is of a different type, you will need to adjust the filter accordingly.

For more information, you can refer to the LangChain Qdrant vector store implementation and the Qdrant documentation.

Sources

• libs/community/langchain_community/vectorstores/qdrant.py
• libs/community/langchain_community/retrievers/qdrant_sparse_vector_retriever.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Francesco-Sch]

@dosu-bot How can I filter for a value that should be equal to the filtered value. For example if I want to filter for a specific slug that exists in the metadata, how would I do that?

[chuan298]

@dosu-bot How can I filter for a value that should be equal to the filtered value. For example if I want to filter for a specific slug that exists in the metadata, how would I do that?

Hi @Francesco-Sch, have you found th answer yet? If not, you can refer to it here https://qdrant.tech/documentation/concepts/filtering
If you use retriever with metadata, you need use Nested key. For example:

...

# Need search field="medical"
from qdrant_client.http import models
results = retriever.invoke(
    input="your_query", 
    filter=models.Filter(
                        should=[
                            models.FieldCondition(
                                key="metadata.field",
                                match=models.MatchValue(value="medical"),
                            )
                        ]
                    )
)
...

[faisal-saddique]

@dosu-bot How can I filter for a value that should be equal to the filtered value. For example if I want to filter for a specific slug that exists in the metadata, how would I do that?

Hi @Francesco-Sch, have you found th answer yet? If not, you can refer to it here https://qdrant.tech/documentation/concepts/filtering If you use retriever with metadata, you need use Nested key. For example:

...

# Need search field="medical"
from qdrant_client.http import models
results = retriever.invoke(
    input="your_query", 
    filter=models.Filter(
                        should=[
                            models.FieldCondition(
                                key="metadata.field",
                                match=models.MatchValue(value="medical"),
                            )
                        ]
                    )
)
...

Thankyou @chuan298 it works!

[Francesco-Sch]

Amazing, that seems to work. :) thanks!

[devbijay]

conversational_rag_chain = RunnableWithMessageHistory(
        rag_chain,
        get_session_history,
        input_messages_key="input",
        history_messages_key="chat_history",
        output_messages_key="answer"
    )
    if not streaming:
        resp = conversational_rag_chain.invoke(
            {"input": data.prompt},
            config={"configurable": {"session_id": chat_id, "filter": filter}}, )["answer"]
        return ChatResponse(chat_id=chat_id, response=resp)

Here how can can i use filter
I am using below Qdrant Filter:

filter = models.Filter(
        must=[
            models.FieldCondition(
                key="metadata.kb_id",
                match=models.MatchValue(value=data.kb_id),
            )
        ]
    )

I want to make sure the RAG chain should only filter document from a specific kbowledgebase id