BM25Okapi returns different scores for same query when case changes

[eliseoe]

Hi! I noticed something unexpected while writing unit tests for BM25Okapi. When I lowercase both the corpus and the query, the scores come back as all zeros, even though the tokens match. The same code with capitalized text works fine. Let me know if I did something wrong!

from rank_bm25 import BM25Okapi
# Capitalized version
corpus = ["Hello world", "foo bar", "hello foo"]
query = "Hello"

tokenized_corpus = [doc.split() for doc in corpus]
tokenized_query = query.split(" ")
bm25 = BM25Okapi(tokenized_corpus)
print("Capitalized:", bm25.get_scores(tokenized_query))

Capitalized: [0.51082562 0. 0. ]
The exact same but lowercase only returns a zero score:

# Lowercased version
corpus = ["hello world", "foo bar", "hello foo"]
query = "hello"
tokenized_query = query.lower().split(" ")
tokenized_corpus = [doc.lower().split() for doc in corpus]
bm25 = BM25Okapi(tokenized_corpus)
print("Lowercased:", bm25.get_scores(tokenized_query))

Lowercased: [0. 0. 0.]

[AswinManohar]

Well, I tried to run this and encountered the same error. I found out that in the standard rank_bm25 implementation of BM25Okapi, it computes a negative IDF, and is often returned directly (producing negative scores like [-0.51, 0, -0.51]). However, depending on your exact version or if the scores are summed with other non-matching terms, it can result in 0.0 or be clamped to 0.0 to prevent common words from penalizing a document.

I tried the same with BM25Plus, a variant designed to fix this "common word" issue by adding a lower bound (delta) to the score computation. This solves the issue with lowercase being treated as noise.

with the "hello" query, the standard BM25 algorithm fails when a query term appears in "too many" documents (usually more than 50 percent of the corpus). It assumes such terms are noise (like "the", "and", "a") and penalizes them. The Problem: If you don't aggressively filter stop words, relevant documents can get a score of 0 (or negative) just because the query term is popular. BM25Plus adds a small constant ($\delta$, usually 1.0) to the IDF score.

Looks like BM25 calculates its own "mathematical" stop words dynamically based on your specific dataset. To BM25, a "stop word" is simply any word that appears in more than half of your documents. Because 2 out of 3 documents contain "hello", the frequency is 66%. The formula treats "hello" as noise . It assumes that if a user searches for "hello", and "hello" is in almost every document, that word is useless for ranking one document higher than another.

Here is a simple test with bm25+

from rank_bm25 import BM25Plus

  corpus = [
      "hello world",
      "foo bar",
      "hello foo"
  ]
  query = "hello"


  tokenized_corpus = [doc.split() for doc in corpus]
  tokenized_query = query.split(" ")
  bm25 = BM25Plus(tokenized_corpus)

  scores = bm25.get_scores(tokenized_query)
  print(f"BM25Plus Scores: {scores}")

I hope this works!

[samanta-sc]

This is the output of @AswinManohar's code:

BM25Plus Scores: [1.38629436 0.69314718 1.38629436]

Thank you!