Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 16 additions & 2 deletions content/develop/ai/search-and-query/administration/overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,9 +239,23 @@ These are the pre-bundled scoring functions available in Redis:
*
Identical to the default TFIDF scorer, with one important distinction:

* **BM25**
* **BM25STD (default)**

A variation on the basic TF-IDF scorer. See [this Wikipedia article for more information](https://en.wikipedia.org/wiki/Okapi_BM25).
A variation on the basic `TFIDF` scorer, see [this Wikipedia article for more info](https://en.wikipedia.org/wiki/Okapi_BM25).

The relevance score for each document is multiplied by the presumptive document score and a penalty is applied based on slop as in `TFIDF`.

{{ note }}
The `BM25` scorer was renamed `BM25STD` in Redis Open Source 8.4. `BM25` is deprecated.
{{ /note }}

* **BM25STD.NORM**

A variation of `BM25STD`, where the scores are normalized by the minimum and maximum score.

* **BM25STD.TANH**

A variation of `BM25STD.NORM`, where the scores are normalised by linear function `tanh(x)`. `BMSTDSTD.TANH` can take an optional argument, `BM25STD_TANH_FACTOR Y`, which is used to smooth the function and the score values. The default value for `Y` is 4.

* **DISMAX**

Expand Down
18 changes: 15 additions & 3 deletions content/develop/ai/search-and-query/advanced-concepts/scoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ weight: 8

When searching, documents are scored based on their relevance to the query. The score is a floating point number between 0.0 and 1.0, where 1.0 is the highest score. The score is returned as part of the search results and can be used to sort the results.

Redis Open Source comes with a few very basic scoring functions to evaluate document relevance. They are all based on document scores and term frequency. This is regardless of the ability to use [sortable fields]({{< relref "/develop/ai/search-and-query/advanced-concepts/sorting" >}}). Scoring functions are specified by adding the `SCORER {scorer_name}` argument to a search query.
Redis Open Source comes with a few scoring functions to evaluate document relevance. They are all based on document scores and term frequency. This is regardless of the ability to use [sortable fields]({{< relref "/develop/ai/search-and-query/advanced-concepts/sorting" >}}). Scoring functions are specified by adding the `SCORER {scorer_name}` argument to a search query.

If you prefer a custom scoring function, it is possible to add more functions using the [extension API]({{< relref "/develop/ai/search-and-query/administration/extensions" >}}).

Expand Down Expand Up @@ -78,16 +78,28 @@ Term frequencies are normalized by the length of the document, expressed as the
FT.SEARCH myIndex "foo" SCORER TFIDF.DOCNORM
```

## BM25 (default)
## BM25STD (default)

A variation on the basic `TFIDF` scorer, see [this Wikipedia article for more info](https://en.wikipedia.org/wiki/Okapi_BM25).

The relevance score for each document is multiplied by the presumptive document score and a penalty is applied based on slop as in `TFIDF`.

{{ note }}
The `BM25` scorer was renamed `BM25STD` in Redis Open Source 8.4. `BM25` is deprecated.
{{ /note }}

```
FT.SEARCH myIndex "foo" SCORER BM25
FT.SEARCH myIndex "foo" SCORER BM25STD
```

## BM25STD.NORM

A variation of `BM25STD`, where the scores are normalized by the minimum and maximum score.

## BM25STD.TANH

A variation of `BM25STD.NORM`, where the scores are normalised by linear function `tanh(x)`. `BMSTDSTD.TANH` can take an optional argument, `BM25STD_TANH_FACTOR Y`, which is used to smooth the function and the score values. The default value for `Y` is 4.

## DISMAX

A simple scorer that sums up the frequencies of matched terms. In the case of union clauses, it will give the maximum value of those matches. No other penalties or factors are applied.
Expand Down