Filtering documents by similarity score when using RetrievalQAWithSourcesChain

[sebastianmontero]

I'm trying to use the "similarity_score_threshold" VectorStore search type with the RetrievalQAWithSourcesChain but I get a NotImplementedError, here is the relevant code:

vector_store = Pinecone.from_existing_index(
        index_name=PINECONE_INDEX_NAME, embedding=embeddings)
    llm = ChatOpenAI(
        temperature=0, openai_api_key=OPENAI_API_KEY, model_name=model_name)
    
    retriever = vector_store.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold":0.5,  "k":10})
    # if compress_context:
    #     compressor = EmbeddingsFilter.from_llm(llm)
    #     retriever = ContextualCompressionRetriever(base_compressor=compressor, base_retriever=retriever, verbose=verbose)
    chain = RetrievalQAWithSourcesChain.from_chain_type(
        llm=llm, chain_type="stuff", retriever=retriever, verbose=verbose)

Is this possible?

[jupyterjazz]

similarity_score_threshold search type is not implemented for Pinecone. I think you can use Redis, Faiss or Weaviate

[sebastianmontero]

Thanks @jupyterjazz, do you know if there is a limitation with pinecone that prevents this?

[LukasFides]

Any news on how to use "similarity_score_threshold" with Pinecone? Is there currently a way to implement a threshold?

[mzamini92]

As far as I know you will need to use a different vector store than Pinecone. Once you have chosen a vector store that supports the "similarity_score_threshold" search type, you can use the following code to use it with the RetrievalQAWithSourcesChain:

vector_store = Redis()
llm = ChatOpenAI(
    temperature=0, openai_api_key=OPENAI_API_KEY, model_name=model_name)

retriever = vector_store.as_retriever(search_type="similarity_score_threshold", search_kwargs={"score_threshold":0.5,  "k":10})
chain = RetrievalQAWithSourcesChain.from_chain_type(
    llm=llm, chain_type="stuff", retriever=retriever, verbose=verbose)

[LukasFides]

Thank you @mzamini92. I am still a bit surprised that this is not possible. I hope there will be a way to use Pinecone with a threshold soon.

[Chameleon1977]

I was having the same issue with Pinecone. I think this is poor business practice from Pinecone to have this issue outstanding months in, after receiving a ton of funding and when it's like 20-30 minutes of dev at best. Also Chat GPT seems to suggest that this feature was removed at some point. I could still be missing something, but if not here's my workaround.

Workaround:
I hope this helps other people overcome this silly limitation. You can query the TopK 10 results, check if they are above the threshold you require. If all results are above the threshold then make another query, filtering out the ID's of the first batch to get the next TopK 10. Therefore, your not then receiving a 1000 records when your only interested in 10 and you not missing 1000 records when you only asked for 10.