Consider BM25opt for BM25Retriever

[jankovicsandras]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Hi,

LangChain uses rank_bm25 under the hood of BM25Retriever. I made an optimized variant, that has 30-40x faster search speed and fixes some of the issues of rank_bm25, while being score compatible: https://github.com/jankovicsandras/bm25opt
Please consider supporting it, I can help adapting BM25Retriever.

Thanks for your attention!

Motivation

The motivation behind BM25opt was to speed up rank_bm25 search speed and fix common issues:

• updating the index is now possible: adding, deleting or updating documents in the index
• registering a tokenizer function in the class ensures correct tokenizing queries later
• providing reasonable standard tokenizer
• ability to override / fix Okapi idf

Proposal (If applicable)

I can write an adapted BM25Retriever and open a PR, if there's interest.