Get scores for specific documents in index

[kswhitecross]

Hi,

Is there a way I can get the BM25 scores for a chosen subset of documents in the index?

For example, if I have a query $q$, and I want to get the scores of 1000 arbitrary documents $d_1, ..., d_{1000}$, from an index of 10 million documents. I know I can just retrieve all 10M documents, and look at the scores of the 1000 I care about, but this would be relatively inefficient. I can also just build an index of the 1000 documents I care about, but the scores from this index will be inaccurate, as the IDF scores will not reflect all 10M documents.

Thanks!

[xhluca]

you can use get_scores method and slice the indexes of documents you are interested in.

bm25s/bm25s/__init__.py

Line 543 in bff5ad3

def get_scores(

[kswhitecross]

Thanks for getting back to me! You mean something like

relevant_doc_scores = retriever.get_scores(query_tokens)[[docid_1, docid_2, ..., docid_1000]]?

Doesn't this compute the scores of every single document in the index though? Is there a way to instead only compute the scores of the documents I care about? Something instead like:

relevant_doc_scores = retriever.get_scores(query_tokens, index_slice=[docid_1, ..., docid_1000])

[xhluca]

no it's not possible. but computing the score for all docs is very fast, in fact probably faster than trying to compute it for a select number of documents.