community: BM25Retriever can output retrieved documents with the similarity scores

[ITHwang]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

• Currently, BM25Retriever retrieves Documents without the similarity scores.
• It would be helpful to output the similarity scores as well by including them into the metadata attributes of Documents.

Motivation

• I wanna utilize the similarity scores calculated by retrievers for improving my ranking task.
• And the others has wanted this feature as well.
• BM25 langchain retriever should have similarity score #16445

Proposal (If applicable)

1. Add the method that returns both the retrieved documents and the similarity scores in BM25Okapi that BM25Retriever depends on.

# class naming: whatever you like
class CustomBM25(BM25Okapi):
    def get_top_n_with_score(self, query: str, documents: list[str], n=5) -> tuple[list[str], list[float]]:
        assert self.corpus_size == len(documents), "The documents given don't match the index corpus!"

        scores = self.get_scores(query)
        top_n = np.argsort(scores)[::-1][:n]

        sorted_scores = [scores[i] for i in top_n]
        sorted_documents = [documents[i] for i in top_n]

        return sorted_documents, sorted_scores

2. Change the vectorizer of BM25Retriever to CustomBM25.

class BM25Retriever(BaseRetriever):
    ...

    @classmethod
    def from_texts(
        cls,
        texts: Iterable[str],
        metadatas: Optional[Iterable[dict]] = None,
        bm25_params: Optional[Dict[str, Any]] = None,
        preprocess_func: Callable[[str], List[str]] = default_preprocessing_func,
        **kwargs: Any,
    ) -> BM25Retriever:
        """
        Create a BM25Retriever from a list of texts.
        Args:
            texts: A list of texts to vectorize.
            metadatas: A list of metadata dicts to associate with each text.
            bm25_params: Parameters to pass to the BM25 vectorizer.
            preprocess_func: A function to preprocess each text before vectorization.
            **kwargs: Any other arguments to pass to the retriever.

        Returns:
            A BM25Retriever instance.
        """
        texts_processed = [preprocess_func(t) for t in texts]
        bm25_params = bm25_params or {}
        vectorizer = CustomBM25(texts_processed, **bm25_params)
        metadatas = metadatas or ({} for _ in texts)
        docs = [Document(page_content=t, metadata=m) for t, m in zip(texts, metadatas)]
        return cls(vectorizer=vectorizer, docs=docs, preprocess_func=preprocess_func, **kwargs)

3. Re-implement the retrieval logic of BM25Retriever by inserting the similarity scores into the metadata attributes of Documents.

class BM25Retriever(BaseRetriever):
    ...

    def _get_relevant_documents(self, query: str, *, run_manager: CallbackManagerForRetrieverRun) -> list[Document]:
        processed_query = self.preprocess_func(query)
        return_docs, scores = self.vectorizer.get_top_n_with_score(processed_query, self.docs, n=self.k)

        for doc, score in zip(return_docs, scores):
            doc.metadata["score"] = score

        return return_docs

4. I think it could be one of the safest ways to implement this feature by minimizing the risks of breaking the existing functionalities.

[ITHwang]

I hope to create a PR from this discussion within a few weeks. Feel free to comment on this proposal if there's any considerations I have to take into account.