How similarity_search_with_score() calculated while retrieving the most similar documents based on asked question.

[VikRathod]

I developed a chat app using faiss document store to find similar documents for a query. It retrieves top 5 docs, with the highest as "High_score" and the lowest as "minimum score." A condition is applied, requiring values to meet specific criteria (e.g., minimum_value_1 >= 0 and < 1.04, maximum_value_1 <= 1.45).
The application uses a condition to assess query correctness and retrieve relevant documents accordingly. Unfortunately, it classifies some relevant queries, leading to a failure in providing answers for queries that should be considered relevant.

Is there any solution that you guys have, or i am missing something in the process.

[dosubot[bot]]

Answer generated by a 🤖

Answer

Hello,

Thank you for reaching out and providing a detailed explanation of your issue. It seems like you're having trouble with the similarity_search_with_score() function in your chat app that uses the faiss document store. Here are some suggestions that might help improve the performance of your similarity search:

1. Improve the Embeddings: The quality of the embeddings plays a crucial role in the performance of the similarity search. If the embeddings are not capturing the semantic meaning of the documents well, the similarity search will not perform well. You can experiment with different language models or fine-tune your existing language model on domain-specific data.

2. Filtering and Thresholding: You can introduce a score threshold to filter out documents that have a similarity score below a certain threshold. This can help in reducing the noise in the results. You can also use the filter parameter to filter documents based on metadata.

3. Increase the number of documents fetched: The fetch_k parameter determines the number of documents to fetch before filtering. If this number is too low, it might be the case that relevant documents are not being fetched. You can experiment with increasing this number.

4. Relevance Scoring Function: The relevance scoring function can also play a crucial role in the performance of the similarity search. You can experiment with different relevance scoring functions to see which one works best for your use case.

Here is how you can implement these changes in your code:

def similarity_search_with_score(
    self,
    query: str,
    k: int = 4,
    filter: Optional[Dict[str, Any]] = None,
    fetch_k: int = 20,
    score_threshold: Optional[float] = 0.5,  # Add score_threshold parameter
    **kwargs: Any,
) -> List[Tuple[Document, float]]:
    faiss = dependable_faiss_import()
    embedding = self.embeddings.embed_query(query)
    vector = np.array([embedding], dtype=np.float32)
    if self._normalize_L2:
        faiss.normalize_L2(vector)
    scores, indices = self.index.search(vector, k if filter is None else fetch_k)
    docs = []
    for j, i in enumerate(indices[0]):
        if i == -1 or scores[0][j] < score_threshold:  # Check if score is below threshold
            continue
        _id = self.index_to_docstore_id[i]
        doc = self.docstore.search(_id)
        if not isinstance(doc, Document):
            raise ValueError(f"Could not find document for id {_id}, got {doc}")
        if filter is not None:
            filter = {
                key: [value] if not isinstance(value, list) else value
                for key, value in filter.items()
            }
            if all(doc.metadata.get(key) in value for key, value in filter.items()):
                docs.append((doc, scores[0][j]))
        else:
            docs.append((doc, scores[0][j]))
    return docs

Please note that these are general suggestions and the actual improvements might vary based on the specific use case and data. I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/vectorstores/faiss.py
• docs/extras/integrations/vectorstores/faiss.ipynb
• libs/langchain/langchain/vectorstores/meilisearch.py
• libs/langchain/langchain/vectorstores/mongodb_atlas.py
• libs/langchain/langchain/vectorstores/sklearn.py
• libs/langchain/langchain/vectorstores/scann.py
• libs/langchain/langchain/vectorstores/weaviate.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.