Some questions about a parameter"search_type="similarity_score_threshold""

[WuYanZhao107]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_community.vectorstores import FAISS
openaiembeddings = OpenAIEmbeddings(model="text-embedding-3-small")

vectorstore = FAISS.load_local('./qa_faiss_index', openaiembeddings)

retriever_1 = vectorstore.as_retriever(
    search_type="similarity_score_threshold", search_kwargs={"score_threshold": 0.4}
)

docs = retriever_from_llm.get_relevant_documents(q_name)

Description

1. I don't know what method is used to calculate similarity for this parameter: cosine similarity, L2 , MMR or so on
2. I also don't know where to check this method
3. Because I want to check the similarity score between my question and each chunk, and use it as a basis to set it upsearch_kwargs={"score_threshold": 0.4}

System Info

python 3.11.7
langchain 0.1.11
faiss-cpu 1.8.0
windows

[dosubot[bot]]

Hey @WuYanZhao107, great to see you back here! Hope you're ready to dive into another fun puzzle with LangChain. 🚀

Based on the context provided, the similarity_score_threshold parameter in LangChain is used to filter out results that have a similarity score below the specified threshold. The method used to calculate similarity is determined by the distance_strategy parameter in the TiDBVectorStore class. The default method is "cosine", but it can also be set to "l2" (Euclidean distance) or "inner_product". This information is crucial for setting the "score_threshold" appropriately, as the choice of distance strategy affects how similarity scores are calculated and interpreted.

You can find this information in the __init__ method of the TiDBVectorStore class, where distance_strategy is a parameter that specifies the strategy used for similarity search. The default value is set to "cosine".

DEFAULT_DISTANCE_STRATEGY = "cosine"  # or "l2", "inner_product"
class TiDBVectorStore(VectorStore):
    def __init__(
        self,
        ...
        distance_strategy: str = DEFAULT_DISTANCE_STRATEGY,
        ...
    ) -> None:
        ...

This setup allows users to choose the most appropriate similarity calculation method for their specific use case by setting the distance_strategy parameter when initializing a TiDBVectorStore instance.

The similarity_score_threshold parameter is used in the similarity_search_by_vector and asimilarity_search_by_vector methods of the FAISS class to filter out results that have a similarity score below the specified threshold. This parameter is set to 0.2 in the tests, indicating that only documents with a similarity score of 0.2 or higher will be returned as results.

output = docsearch.similarity_search_by_vector(query_vec, k=2, score_threshold=0.2)

output = await docsearch.asimilarity_search_by_vector(
    query_vec, k=2, score_threshold=0.2
)

In both synchronous and asynchronous versions of the method, the score_threshold=0.2 parameter is used to specify the minimum similarity score for documents to be included in the search results.

Regarding your question about where to check this method, you can find the implementation of the similarity_score_threshold parameter in the as_retriever method of the FAISS class in the LangChain repository here.

I hope this information helps! If you have any other questions, feel free to ask.

Sources

• TypeError: FAISS.similarity_search_with_score_by_vector() got an unexpected keyword argument 'score_threshold'
• libs/community/langchain_community/vectorstores/tidb_vector.py
• libs/langchain/tests/unit_tests/vectorstores/test_faiss.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[WuYanZhao107]

How can I modify the distance calculation strategy

[unical1988]

Can similarity_score_threshold be used as a parameter for similarity_search_with_score_by_vector() as well? how to get documents that are at least a score similar to a query (not at most similar as is the case of score_threshold parameter)?