You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
> Provisioned Throughput Units (PTUs) are different from standard quota in Azure OpenAI and are not available by default in Azure Government. To learn more about this offering contact your Microsoft Account Team.
Copy file name to clipboardExpand all lines: articles/search/index-similarity-and-scoring.md
+20-23Lines changed: 20 additions & 23 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,12 +9,12 @@ ms.service: azure-ai-search
9
9
ms.custom:
10
10
- ignite-2023
11
11
ms.topic: conceptual
12
-
ms.date: 05/06/2024
12
+
ms.date: 12/10/2024
13
13
---
14
14
15
15
# Relevance in keyword search (BM25 scoring)
16
16
17
-
This article explains the BM25 relevance scoring algorithm used to compute search scores for [full text search](search-lucene-query-architecture.md). BM25 relevance is exclusive to full text search. Filter queries, autocomplete and suggested queries, wildcard search or fuzzy search queries aren't scored or ranked for relevance.
17
+
This article explains the BM25 relevance scoring algorithm used to compute search scores for [full text search](search-lucene-query-architecture.md). BM25 relevance is exclusive to full text search. Filter queries, autocomplete and suggested queries, wildcard search, and fuzzy search queries aren't scored or ranked for relevance.
18
18
19
19
## Scoring algorithms used in full text search
20
20
@@ -23,39 +23,36 @@ Azure AI Search provides the following scoring algorithms for full text search:
23
23
| Algorithm | Usage | Range |
24
24
|-----------|-------------|-------|
25
25
|`BM25Similarity`| Fixed algorithm on all search services created after July 2020. You can configure this algorithm, but you can't switch to an older one (classic). | Unbounded. |
26
-
|`ClassicSimilarity`|Present on older search services. You can [opt-in for BM25](index-ranking-similarity.md) and choose an algorithm on a per-index basis. | 0 < 1.00 |
26
+
|`ClassicSimilarity`|Default on older search services that predate July 2020. On older services, you can [opt-in for BM25](index-ranking-similarity.md#enable-bm25-scoring-on-older-services) and choose a the BM25 algorithm on a per-index basis. | 0 < 1.00 |
27
27
28
-
Both BM25 and Classic are TF-IDF-like retrieval functions that use the term frequency (TF) and the inverse document frequency (IDF) as variables to calculate relevance scores for each document-query pair, which is then used for ranking results. While conceptually similar to classic, BM25 is rooted in probabilistic information retrieval that produces more intuitive matches, as measured by user research.
28
+
Both BM25 and Classic are TF-IDF-like retrieval functions that use the term frequency (TF) and the inverse document frequency (IDF) as variables to calculate relevance scores for each document-query pair, which is then used for ranking results. While conceptually similar to classic, BM25 is rooted in probabilistic information retrieval that produces more intuitive matches, as measured by user research.
29
29
30
-
BM25 offers advanced customization options, such as allowing the user to decide how the relevance score scales with the term frequency of matched terms. For more information, see [Configure the scoring algorithm](index-ranking-similarity.md).
31
-
32
-
> [!NOTE]
33
-
> If you're using a search service that was created before July 2020, the scoring algorithm is most likely the previous default, `ClassicSimilarity`, which you can upgrade on a per-index basis. See [Enable BM25 scoring on older services](index-ranking-similarity.md#enable-bm25-scoring-on-older-services) for details.
34
-
35
-
The following video segment fast-forwards to an explanation of the generally available ranking algorithms used in Azure AI Search. You can watch the full video for more background.
BM25 offers [advanced customization options](index-ranking-similarity.md), such as allowing the user to decide how the relevance score scales with the term frequency of matched terms.
38
31
39
32
## How BM25 ranking works
40
33
41
-
Relevance scoring refers to the computation of a search score (**@search.score**) that serves as an indicator of an item's relevance in the context of the current query. The range is unbounded. However, the higher the score, the more relevant the item.
34
+
Relevance scoring refers to the computation of a search score (**@search.score**) that serves as an indicator of an item's relevance in the context of the current query. The range is unbounded. However, the higher the score, the more relevant the item.
42
35
43
36
The search score is computed based on statistical properties of the string input and the query itself. Azure AI Search finds documents that match on search terms (some or all, depending on [searchMode](/rest/api/searchservice/documents/search-post#searchrequest)), favoring documents that contain many instances of the search term. The search score goes up even higher if the term is rare across the data index, but common within the document. The basis for this approach to computing relevance is known as *TF-IDF or* term frequency-inverse document frequency.
44
37
45
38
Search scores can be repeated throughout a result set. When multiple hits have the same search score, the ordering of the same scored items is undefined and not stable. Run the query again, and you might see items shift position, especially if you're using the free service or a billable service with multiple replicas. Given two items with an identical score, there's no guarantee that one appears first.
46
39
47
-
To break the tie among repeating scores, you can add an **$orderby** clause to first order by score, then order by another sortable field (for example, `$orderby=search.score() desc,Rating desc`). For more information, see [$orderby](search-query-odata-orderby.md).
40
+
To break the tie among repeating scores, you can add an [**$orderby** clause](search-query-odata-orderby.md) to first order by score, then order by another sortable field (for example, `$orderby=search.score() desc,Rating desc`).
48
41
49
42
Only fields marked as `searchable` in the index, or `searchFields` in the query, are used for scoring. Only fields marked as `retrievable`, or fields specified in `select` in the query, are returned in search results, along with their search score.
50
43
51
44
> [!NOTE]
52
45
> A `@search.score = 1` indicates an un-scored or un-ranked result set. The score is uniform across all results. Un-scored results occur when the query form is fuzzy search, wildcard or regex queries, or an empty search (`search=*`, sometimes paired with filters, where the filter is the primary means for returning a match).
53
46
47
+
The following video segment fast-forwards to an explanation of the generally available ranking algorithms used in Azure AI Search. You can watch the full video for more background.
@@ -91,7 +88,7 @@ The diagram above is only one example. Many combinations of partitions and repli
91
88
92
89
<aname="scoring-statistics"></a>
93
90
94
-
###Scoring statistics and sticky sessions
91
+
## Scoring statistics and sticky sessions
95
92
96
93
For scalability, Azure AI Search distributes each index horizontally through a sharding process, which means that [portions of an index are physically separate](#sharding-effects-on-query-results).
97
94
@@ -107,7 +104,7 @@ POST https://[service name].search.windows.net/indexes/hotels/docs/search?api-ve
107
104
}
108
105
```
109
106
110
-
Using `scoringStatistics` will ensure that all shards in the same replica provide the same results. That said, different replicas may be slightly different from one another as they're always getting updated with the latest changes to your index. In some scenarios, you may want your users to get more consistent results during a "query session". In such scenarios, you can provide a `sessionId` as part of your queries. The `sessionId` is a unique string that you create to refer to a unique user session.
107
+
Using `scoringStatistics` will ensure that all shards in the same replica provide the same results. That said, different replicas can be slightly different from one another as they're always getting updated with the latest changes to your index. In some scenarios, you might want your users to get more consistent results during a "query session". In such scenarios, you can provide a `sessionId` as part of your queries. The `sessionId` is a unique string that you create to refer to a unique user session.
111
108
112
109
```http
113
110
POST https://[service name].search.windows.net/indexes/hotels/docs/search?api-version=2024-07-01
@@ -124,12 +121,12 @@ As long as the same `sessionId` is used, a best-effort attempt is made to target
124
121
125
122
## Relevance tuning
126
123
127
-
In Azure AI Search, you can configure BM25 algorithm parameters, and tune search relevance and boost search scores through these mechanisms:
124
+
In Azure AI Search, for keyword search and the text portion of a hybrid query, you can configure BM25 algorithm parameters plus tune search relevance and boost search scores through the following mechanisms.
128
125
129
126
| Approach | Implementation | Description |
130
127
|----------|----------------|-------------|
131
-
|[Scoring algorithm configuration](index-ranking-similarity.md)| Search index ||
132
-
|[Scoring profiles](index-add-scoring-profiles.md)| Search index | Provide criteria for boosting the search score of a match based on content characteristics. For example, you might want to boost matches based on their revenue potential, promote newer items, or perhaps boost items that have been in inventory too long. A scoring profile is part of the index definition, composed of weighted fields, functions, and parameters. You can update an existing index with scoring profile changes, without incurring an index rebuild.|
128
+
|[BM25 algorithm configuration](index-ranking-similarity.md)| Search index |Configure how document length and term frequency affect the relevance score.|
129
+
|[Scoring profiles](index-add-scoring-profiles.md)| Search index | Provide criteria for boosting the search score of a match based on content characteristics. For example, you can boost matches based on their revenue potential, promote newer items, or perhaps boost items that have been in inventory too long. A scoring profile is part of the index definition, composed of weighted fields, functions, and parameters. You can update an existing index with scoring profile changes, without incurring an index rebuild.|
133
130
|[Semantic ranking](semantic-search-overview.md)| Query request | Applies machine reading comprehension to search results, promoting more semantically relevant results to the top. |
134
131
|[featuresMode parameter](#featuresmode-parameter-preview)| Query request | This parameter is mostly used for unpacking a BM25-ranked score, but it can be used for in code that provides a [custom scoring solution](https://github.com/Azure-Samples/search-ranking-tutorial). |
135
132
@@ -173,9 +170,9 @@ The featuresMode parameter isn't documented in the REST APIs, but you can use it
173
170
174
171
## Number of ranked results in a full text query response
175
172
176
-
By default, if you aren't using pagination, the search engine returns the top 50 highest ranking matches for full text search. You can use the `top` parameter to return a smaller or larger number of items (up to 1000 in a single response). Full text search is subject to a maximum limit of 1,000 matches (see [API response limits](search-limits-quotas-capacity.md#api-response-limits)). Once 1,000 matches are found, the search engine no longer looks for more.
173
+
By default, if you aren't using pagination, the search engine returns the top 50 highest ranking matches for full text search. You can use the `top` parameter to return a smaller or larger number of items (up to 1,000 in a single response). You can use `skip` and `next`to page results. Paging determines the number of results on each logical page and supports content navigation. For more information, see [Shape search results](search-pagination-page-layout.md).
177
174
178
-
To return more or less results, use the paging parameters `top`, `skip`, and `next`. Paging is how you determine the number of results on each logical page and navigate through the full payload. For more information, see [How to work with search results](search-pagination-page-layout.md).
175
+
Full text search is subject to a maximum limit of 1,000 matches (see [API response limits](search-limits-quotas-capacity.md#api-response-limits)). Once 1,000 matches are found, the search engine no longer looks for more.
0 commit comments