ConversationalRetrievalChain does'nt return score with documents when using FAISS vector store

[prasoons075]

qa = ConversationalRetrievalChain(
    retriever=vectorstore.as_retriever( search_kwargs={"k":6}),
    max_tokens_limit=4097,
    combine_docs_chain=doc_chain,
    question_generator=question_generator,
    callback_manager=manager,
    return_source_documents=True,
)

how to use this to return scores along with docs

[dosubot[bot]]

🤖

Hello,

Based on your request, it seems you want to modify the ConversationalRetrievalChain to return similarity scores along with the documents when using the FAISS vector store.

Currently, the ConversationalRetrievalChain does not support returning similarity scores directly. However, there is a similar open issue #5067 in the LangChain repository where a user suggested adding a flag like similarity = True to get similarity scores from the chain. This feature is not yet implemented.

However, you can modify the similarity_search_with_score_by_vector method in the FAISS class to return a list of tuples, where each tuple contains a Document and its corresponding similarity score. Here is the relevant part of the code:

def similarity_search_with_score_by_vector(
    self,
    embedding: List[float],
    k: int = 4,
    filter: Optional[Dict[str, Any]] = None,
    fetch_k: int = 20,
    **kwargs: Any,
) -> List[Tuple[Document, float]]:
    # ... omitted for brevity ...

    docs_and_scores = []
    for i, score in zip(selected_indices, selected_scores):
        if i == -1:
            # This happens when not enough docs are returned.
            continue
        _id = self.index_to_docstore_id[i]
        doc = self.docstore.search(_id)
        if not isinstance(doc, Document):
            raise ValueError(f"Could not find document for id {_id}, got {doc}")
        docs_and_scores.append((doc, score))
    return docs_and_scores

This method returns a list of tuples, where each tuple contains a Document and its corresponding similarity score. The scores returned are L2 distances, where a lower score represents more similarity. If you want to convert these scores to a relevance score between 0 and 1, you can use the _similarity_search_with_relevance_scores method, which applies a relevance function to the raw scores.

Please note that this modification will require a good understanding of the LangChain codebase and Python programming. If you need further assistance, feel free to ask.

Best,
Dosu

Sources

• ConversationalRetrievalChain doesn't return score with sources
• libs/langchain/langchain/vectorstores/faiss.py
• libs/langchain/langchain/chains/conversational_retrieval/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.