Indexing API - Supported vector databases

[srinisubramanian]

Refer this: https://python.langchain.com/docs/modules/data_connection/indexing

The doc just mentions that the requirement from the vector database that support document addition by id (add_documents method with ids argument) and delete by id (delete method with)

Is there a list of vector databases (by name) that are supported for teh Indexing?

[dosubot[bot]]

🤖

Hello,

The LangChain framework's Indexing API is designed to support a wide range of vector databases. Some of the supported databases include AlibabaCloudOpenSearch, AnalyticDB, Annoy, AtlasDB, AwaDB, AzureSearch, Bagel, Cassandra, Chroma, Clarifai, Clickhouse, DashVector, DeepLake, Dingo, DocArrayHnswSearch, DocArrayInMemorySearch, ElasticKnnSearch, ElasticVectorSearch, ElasticsearchStore, Epsilla, FAISS, Hologres, LanceDB, Marqo, MatchingEngine, Meilisearch, Milvus, MongoDBAtlasVectorSearch, MyScale, Neo4jVector, OpenSearchVectorSearch, PGEmbedding, PGVector, Pinecone, Qdrant, Redis, Rockset, SKLearnVectorStore, ScaNN, SingleStoreDB, SQLiteVSS, StarRocks, SupabaseVectorStore, Tair, Tigris, Typesense, USearch, Vectara, Weaviate, ZepVectorStore, Zilliz, and TencentVectorDB.

The Indexing API is used to index data from a loader into a vector store. It uses a manager to keep track of which documents are in the vector store, allowing it to track which documents were updated, deleted, or should be skipped. The main function in this API is index(), which takes in a data source (either a loader or an iterable of documents), a record manager, and a vector store. It also accepts optional parameters for batch size, cleanup mode, and a source id key.

Please note that the vector databases must support document addition by id (add_documents method with ids argument) and delete by id (delete method with id argument) to be compatible with the Indexing API.

For more details, you can refer to the source code in the LangChain repository:

• libs/langchain/langchain/vectorstores/opensearch_vector_search.py
• libs/langchain/langchain/vectorstores/init.py
• libs/langchain/langchain/indexes/_api.py

I hope this information is helpful. If you have any more questions, feel free to ask.

Best,
Dosu

Sources

• libs/langchain/langchain/vectorstores/opensearch_vector_search.py
• libs/langchain/langchain/vectorstores/init.py
• libs/langchain/langchain/indexes/_api.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[RyanZotti]

I had the same question. The Langchain documentation could be a lot better in this regard, but here is how I would check (adapted from how the framework does it):

import inspect
import langchain


# Modified based on code here: https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/indexes/_api.py#L227-L239
def check_compatibility(vector_store):
    methods = ["delete", "add_documents"]
    for method in methods:
        if not hasattr(vector_store, method):
            return False
    # Checking if the vectorstore has overridden the default delete method
    # implementation which just raises a NotImplementedError
    if getattr(vector_store, 'delete') == langchain.vectorstores.VectorStore.delete:
        return False
    return True

compatible = []
incompatible = []
for class_name, vector_store in inspect.getmembers(langchain.vectorstores, predicate=inspect.isclass):
    
    # The base class is listed along with the real vector stores
    if class_name == 'VectorStore':
        continue
    
    # Some vector store modules have additional, custom classes. Filter those out
    if issubclass(vector_store, langchain.vectorstores.base.VectorStore):
        is_compatible = check_compatibility(vector_store)
        if is_compatible:
            compatible.append(class_name)
        else:
            incompatible.append(class_name)

This produces the following for compatible:

• 'AnalyticDB',
• 'AwaDB',
• 'Bagel',
• 'Cassandra',
• 'Chroma',
• 'DashVector',
• 'DeepLake',
• 'Dingo',
• 'ElasticVectorSearch',
• 'ElasticsearchStore',
• 'FAISS',
• 'PGVector',
• 'Pinecone',
• 'Qdrant',
• 'Redis',
• 'ScaNN',
• 'SupabaseVectorStore',
• 'TimescaleVector',
• 'Vald',
• 'Vearch',
• 'Weaviate',
• 'ZepVectorStore'

And incompatible:

• 'AlibabaCloudOpenSearch',
• 'Annoy',
• 'AtlasDB',
• 'AzureSearch',
• 'Clarifai',
• 'Clickhouse',
• 'DocArrayHnswSearch',
• 'DocArrayInMemorySearch',
• 'ElasticKnnSearch',
• 'Epsilla',
• 'Hologres',
• 'LLMRails',
• 'LanceDB',
• 'Marqo',
• 'MatchingEngine',
• 'Meilisearch',
• 'Milvus',
• 'MongoDBAtlasVectorSearch',
• 'MyScale',
• 'Neo4jVector',
• 'OpenSearchVectorSearch',
• 'PGEmbedding',
• 'Rockset',
• 'SKLearnVectorStore',
• 'SQLiteVSS',
• 'SingleStoreDB',
• 'StarRocks',
• 'Tair',
• 'TencentVectorDB',
• 'Tigris',
• 'Typesense',
• 'USearch',
• 'Vectara',
• 'Zilliz'

At least for Langchain version 0.0.300

[srinisubramanian]

This is super!