OpenSearchVectorSearch Usage

[nirmaljayin]

I have a couple of doubts about this class.

• I am trying to create a vector store from a list of documents. But every time I run the code it adds the same document again So when I search for results, I am getting duplicated results.

embeddings = OpenAIEmbeddings(deployment="text-embedding-ada-002", chunk_size=1)

opensearch_vector_search = OpenSearchVectorSearch(
    opensearch_url = "http://localhost:9200",
    index_name = "test",
    embedding_function = embeddings,
)

opensearch_vector_search.add_documents(documents=documents) # `documents` is a list of Document object

• When I try to run the OpenSearchVectorSearch.similarity_search_with_relevance_scores method I am getting a NotImplementedError exception.

opensearch_vector_search.similarity_search_with_relevance_scores(query=my_query, k=20, score_threshold=0.8)

Error

    231 """Return docs and relevance scores in the range [0, 1].
    232 
    233 0 is dissimilar, 1 is most similar.
   (...)
    243     List of Tuples of (doc, similarity_score)
    244 """
    245 score_threshold = kwargs.pop("score_threshold", None)
--> 247 docs_and_similarities = self._similarity_search_with_relevance_scores(
    248     query, k=k, **kwargs
    249 )
    250 if any(
    251     similarity < 0.0 or similarity > 1.0
    252     for _, similarity in docs_and_similarities
    253 ):
    254     warnings.warn(
    255         "Relevance scores must be between"
    256         f" 0 and 1, got {docs_and_similarities}"
    257     )
...
    188     Vectorstores should define their own selection based method of relevance.
    189     """
--> 190     raise NotImplementedError

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out with your questions. Let's address them one by one.

1. Regarding the duplication of documents, it seems like the add_documents method is being called multiple times with the same list of documents. This method will add the documents to the index each time it is called. If you want to avoid duplication, you should ensure that the documents are only added once. If you need to update the documents, you should use the update_documents method instead.

2. As for the NotImplementedError you're encountering with the similarity_search_with_relevance_scores method, it appears that you're using an older version of LangChain where this method was not fully implemented. This issue was addressed in a previous issue #3286. The solution is to update your LangChain version. The placeholder NotImplementedError was removed in later versions and the method was fully implemented.

Here's how you can update LangChain:

pip install --upgrade langchain

After updating, the similarity_search_with_relevance_scores method should work as expected.

I hope this helps! If you have any other questions or run into any issues, feel free to ask.

Best,
Dosu

Sources

• Why is this implementation of vectorstore not working?

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[nirmaljayin]

@dosu-bot I have upgraded to the latest version of LangChain. But I am still getting the NotImplementedError. As for the update_documents method it is not part of the OpenSearchVectorSearch

[dosubot[bot]]

🤖

Hello,

Thank you for reaching out and providing detailed information about the issues you're facing.

For the first issue, the OpenSearchVectorSearch class in LangChain does not have an update_documents method. Instead, you can use the add_texts method to add documents to your OpenSearch index. This method allows you to provide a list of unique identifiers for each document you're adding. If you don't provide this parameter, the method will generate a unique UUID for each document. However, if you're adding the same document more than once without providing a unique ID, it will be treated as a new document each time because a new UUID will be generated. Here's an example of how you can use the ids parameter:

opensearch_vector_search.add_texts(
    texts=["text1", "text2", "text3"],
    ids=["id1", "id2", "id3"],
    bulk_size=500
)

This should prevent the duplication of documents in your OpenSearch index.

As for the second issue, the NotImplementedError you're encountering is because the similarity_search_with_relevance_scores method is not implemented in the OpenSearchVectorSearch class. The class does have a similarity_search_with_score method, which you can use as follows:

opensearch_vector_search.similarity_search_with_score(
    query="your query",
    k=4
)

This method returns a list of tuples, where each tuple contains a Document object and a relevance score.

If you're looking for a method named similarity_search_with_relevance_scores, it might not be available in the current version of LangChain you're using. I found a similar issue in the LangChain repository (Issue #3286) where the user was advised to update LangChain to the latest version. You mentioned that you have already upgraded to the latest version, but it might be worth double-checking to ensure you're using the correct version.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Best,
Dosu

Sources

• Why is this implementation of vectorstore not working?
• libs/langchain/langchain/vectorstores/opensearch_vector_search.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.