Ideas on how to create bm25 index for large corpus

[igaurab]

I have around 20 million documents in 500 jsonl files, sized at around 1.2TB

I do have memory restrictions: 200GB Ram

If you could provide me any idea around how i can approach this problem to create a bm25 index, that would be helpful.

[xhluca]

Good question! I've thought of this problem in the past but the main issue is that i need to use scipy to construct the csr internally, which is impossible to memory map (or at least hard enough that i haven't be successful). Because the calculations are over all documents (e.g. term frequency) it's hard to chunk that in a way that stays as fast as it currently is.

Out of curiosity: how many docs can you fit so far with 200gb ram? btw you can have corpus as iterable rather than list to save memory:

corpus = function_that_returns_one_doc_at_the_time()
tokenizer = Tokenizer(
      stemmer=stemmer, stopwords=stopwords, splitter=splitter
)

corpus_tokens = tokenizer.tokenize(corpus)