Add BGE M3 Embeddings (Dense, Sparse, Colbert), Visualized BGE Embeddings, and Efficient HybridRetriever

[muazhari]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Please add these features:

1. BGE M3 Embeddings (Dense, Sparse, Colbert): https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/BGE_M3
2. Visualized BGE Embeddings: https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/visual
3. Efficient HybridRetriever: https://github.com/milvus-io/pymilvus/blob/master/examples/hello_hybrid_sparse_dense.py

Motivation

1. BGE M3 Embeddings (Dense, Sparse, Colbert) are currently the most performant model in its scope. I need a more performant multilingual model to minimize use case risk as much as possible.
2. Visualized BGE Embeddings is needed to include images to be queried. I need more complete information to minimize use case risk as much as possible.
3. If using existing EnsembleRetriever for HybridRetrieval with Milvus vector store, it still needed to re-operate already existing fields data like id and text by using many different collection names for each vector embeddings. The re-operation leads to data duplication, even though these vector store embeddings only need a single operation of id and text fields for all vector embeddings. Possible solutions are (1) implementing support of many vector fields in a single vector store and (2) making the vector store not too opinionated by supporting custom fields.

Proposal (If applicable)

No response

[BuxianChen]

Hi, I've been thinking about this for some time and reviewed the existing code:

• langchain_community/vectorstores/milvus.py, the implementation of Milvus VectorStore contains the create, add, delete, update and search interface. But only one vector per text.
• langchain_community/retrievers/milvus.py, it relies on Milvus VectorStore, but only contains add and search interface.

I'd like to submit a PR for Milvus to achieve these objectives, along with the challenges I have encountered in doing so. (currently, I haven't started my work on it)

• support multi-vector in one collection: Wheather should I implement a new class MultiVectorMilvus (with a lot of duplicated code from vectorstores/milvus.py) or just upgrade existing Milvus VectorStore class?
• support sparse-vector: BGE-M3 support sparse vector embedding and colbert vector embedding. Maybe colbert vector embedding is not suitable for most vectorstore, including Milvus, but Milvus 2.4 indeed support sparse vector search. Howerver, I reviewed langchain-core, the abstract of Embedding is only suitable for dense vector:

  class Embeddings(ABC):
  """Interface for embedding models."""
  
  @abstractmethod
  def embed_documents(self, texts: List[str]) -> List[List[float]]:
      """Embed search docs."""
  
  # omit others: ...

  But for sparse vector, return type (maybe) should be List[Dict[int, float]]. So I think here should be a update for langchain-core, but updating this maybe need core developer's discussion and help.
• update to MilvusClient: Unfortunately, the pymilvus==2.4 don't support MilvusClient.hybrid_search (only support Collection.hybrid_search)
• get different types of embedding simultaneously: Maybe this is a unique property of BGE-M3, the dense/sparse/colbert vector shared same Transformer Encoder layers. The lesson is the Embbedings abstraction may also need a MultiEmbeddings counterpart?