You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/semantic-how-to-query-request.md
+20-20Lines changed: 20 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -29,16 +29,16 @@ To use semantic ranking:
29
29
30
30
+ Semantic ranking [enabled on your search service](semantic-how-to-enable-disable.md).
31
31
32
-
+ An existing search index with rich content in a [supported query language](/rest/api/searchservice/preview-api/search-documents#queryLanguage). Semantic ranking works best on content that is informational or descriptive.
32
+
+ An existing search index with rich text content in a [supported query language](/rest/api/searchservice/preview-api/search-documents#queryLanguage). Semantic ranking applies to text (non-vector) fields and works best on content that is informational or descriptive.
33
33
34
-
+ Review the [Semantic search overview](semantic-search-overview.md) if you need an introduction to the feature.
34
+
+ Review [semantic ranking](semantic-search-overview.md) if you need an introduction to the feature.
35
35
36
36
> [!NOTE]
37
37
> Captions and answers are extracted verbatim from text in the search document. The semantic subsystem uses language understanding to determine what part of your content have the characteristics of a caption or answer, but it doesn't compose new sentences or phrases. For this reason, content that includes explanations or definitions work best for semantic ranking.
38
38
39
39
## 1 - Choose a client
40
40
41
-
You'll need a search client that supports preview APIs on the query request. Here are some options:
41
+
Choose a search client that supports preview APIs on the query request. Here are some options:
42
42
43
43
+[Search explorer](search-explorer.md) in Azure portal, recommended for initial exploration.
44
44
@@ -55,11 +55,11 @@ You'll need a search client that supports preview APIs on the query request. Her
55
55
56
56
A *semantic configuration* specifies how fields are used in semantic ranking. It gives the underlying models hints about which index fields are most important for semantic ranking, captions, highlights, and answers.
57
57
58
-
You'll add a semantic configuration to your [index definition](/rest/api/searchservice/preview-api/create-or-update-index). The tabbed sections below provide instructions for the REST APIs, Azure portal, and the .NET SDK Preview.
58
+
Add a semantic configuration to your [index definition](/rest/api/searchservice/preview-api/create-or-update-index). The following tabbed sections provide instructions for the REST APIs, Azure portal, and the .NET SDK Preview.
59
59
60
-
You can add or update a semantic configuration at any time without rebuilding your index. When you issue a query, you'll add the semantic configuration (one per query) that specifies which semantic configuration to use for the query.
60
+
You can add or update a semantic configuration at any time without rebuilding your index. When you issue a query, add the semantic configuration (one per query) that specifies which semantic configuration to use for the query.
61
61
62
-
1. Review the properties you'll need to specify. A semantic configuration has a name and at least one each of the following properties:
62
+
1. Review the properties needed in the configuration. A semantic configuration has a name and at least one each of the following properties:
63
63
64
64
+**Title field** - A title field should be a concise description of the document, ideally a string that is under 25 words. This field could be the title of the document, name of the product, or item in your search index. If you don't have a title in your search index, leave this field blank.
65
65
+**Content fields** - Content fields should contain text in natural language form. Common examples of content are the body of a document, the description of a product, or other free-form text.
@@ -71,7 +71,7 @@ You can add or update a semantic configuration at any time without rebuilding yo
71
71
72
72
A field must be searchable and retrievable.
73
73
74
-
A field must be a [supported data type](/rest/api/searchservice/supported-data-types) and it should contain strings. If you happen to include an invalid field, there's no error, but those fields won't be used in semantic ranking.
74
+
A field must be a [supported data type](/rest/api/searchservice/supported-data-types) and it should contain strings. If you happen to include an invalid field, there's no error, but those fields aren't used in semantic ranking.
75
75
76
76
| Data type | Example from hotels-sample-index |
77
77
|-----------|----------------------------------|
@@ -80,7 +80,7 @@ You can add or update a semantic configuration at any time without rebuilding yo
80
80
| Collection(Edm.String) | Tags (a comma-delimited list of strings) |
81
81
82
82
> [!NOTE]
83
-
> Subfields of Collection(Edm.ComplexType) fields aren't currently supported by semantic search and won't be used for semantic ranking, captions, or answers.
83
+
> Subfields of Collection(Edm.ComplexType) fields aren't currently supported by semantic search and aren't used for semantic ranking, captions, or answers.
This step is only for solutions using the 2020-06-30-Preview REST API or a beta SDK that doesn't support semantic configurations. Instead of setting field prioritization in the index through a semantic configuration, you'll set the priority at query time, using the "searchFields" parameter of a query.
175
+
This step is only for solutions using the 2020-06-30-Preview REST API or a beta SDK that doesn't support semantic configurations. Instead of setting field prioritization in the index through a semantic configuration, set the priority at query time, using the searchFields parameter of a query.
176
176
177
-
Using "searchFields" for field prioritization was an early implementation detail that won't be supported once semantic search exits public preview. We encourage you to use semantic configurations if your application requirements allow it.
177
+
Using searchFields for field prioritization was an early implementation detail that won't be supported once semantic search exits public preview. We encourage you to use semantic configurations if your application requirements allow it.
178
178
179
179
```http
180
180
POST https://[service name].search.windows.net/indexes/[index name]/docs/search?api-version=2020-06-30-Preview
@@ -186,19 +186,19 @@ POST https://[service name].search.windows.net/indexes/[index name]/docs/search?
186
186
}
187
187
```
188
188
189
-
Field order is critical because the semantic ranker limits the amount of content it can process while still delivering a reasonable response time. Content from fields at the start of the list are more likely to be included; content from the end could be truncated if the maximum limit is reached. For more information, see [Pre-processing during semantic ranking](semantic-search-overview.md#how-inputs-are-prepared).
189
+
Field order is critical because the semantic ranker limits the amount of content it can process while still delivering a reasonable response time. Content from fields at the start of the list are more likely to be included; content from the end could be truncated if the maximum limit is reached. For more information, see [Preprocessing during semantic ranking](semantic-search-overview.md#how-inputs-are-prepared).
190
190
191
191
+ If you're specifying just one field, choose a descriptive field where the answer to semantic queries might be found, such as the main content of a document.
192
192
193
-
+ For two or more fields in searchFields:
193
+
+ For two or more fields in `searchFields`:
194
194
195
195
+ The first field should always be concise (such as a title or name), ideally a string that is under 25 words.
196
196
197
197
+ If the index has a URL field that is human readable such as `www.domain.com/name-of-the-document-and-other-details` (rather than machine focused, such as `www.domain.com/?id=23463¶m=eis`), place it second in the list (or first if there's no concise title field).
198
198
199
199
+ Follow the above fields with other descriptive fields, where the answer to semantic queries may be found, such as the main content of a document.
200
200
201
-
When setting "searchFields", choose only fields of the following [supported data types](/rest/api/searchservice/supported-data-types):
201
+
When setting `searchFields`, choose only fields of the following [supported data types](/rest/api/searchservice/supported-data-types):
202
202
203
203
| Data type | Example from hotels-sample-index |
204
204
|-----------|----------------------------------|
@@ -210,15 +210,15 @@ If you happen to include an invalid field, there's no error, but those fields wo
210
210
211
211
## 3 - Avoid features that bypass relevance scoring
212
212
213
-
Several query capabilities in Cognitive Search bypass relevance scoring. If your query logic includes the following features, you won't get relevance scores or semantic ranking on your results:
213
+
Several query capabilities in Cognitive Search bypass relevance scoring. If your query logic includes the following features, you won't get BM25 relevance scores or semantic ranking on your results:
214
214
215
215
+ Filters, fuzzy search queries, and regular expressions iterate over untokenized text, scanning for verbatim matches in the content. Search scores for all of the above query forms are a uniform 1.0, and won't provide meaningful input for semantic ranking because there's no way to select the top 50 matches.
216
216
217
-
+ Sorting (orderBy clauses) on specific fields will also override search scores and semantic score. Given that semantic score is used to order results, including explicit sort logic will cause an HTTP 400 error to be returned if you run a semantic query over ordered results.
217
+
+ Sorting (orderBy clauses) on specific fields overrides search scores and a semantic score. Given that the semantic score is supposed to provide the ranking, adding an orderby clause results in an HTTP 400 error if you try to apply semantic ranking over ordered results.
218
218
219
219
## 4 - Set up the query
220
220
221
-
Your next step is adding parameters to the query request. To be successful, your query should be full text search (using the "search" parameter to pass in a string), and the index should contain text fields with rich semantic content and a semantic configuration.
221
+
Your next step is adding parameters to the query request. To be successful, your query should be full text search (using the `search` parameter to pass in a string), and the index should contain text fields with rich semantic content and a semantic configuration.
222
222
223
223
### [**Azure portal**](#tab/portal-query)
224
224
@@ -240,7 +240,7 @@ Your next step is adding parameters to the query request. To be successful, your
240
240
241
241
Use the [Search Documents (REST preview)](/rest/api/searchservice/preview-api/search-documents) to formulate the request.
242
242
243
-
A response includes an "@search.rerankerScore"" automatically. If you want captions, spelling correction, or answers in the response, add "captions", "speller", or "answers" to the request.
243
+
A response includes an `@search.rerankerScore` automatically. If you want captions, spelling correction, or answers in the response, add captions, speller, or answers to the request.
244
244
245
245
The following example in this section uses the [hotels-sample-index](search-get-started-portal.md) to demonstrate semantic ranking with spell check, semantic answers, and captions.
246
246
@@ -294,7 +294,7 @@ The following example in this section uses the [hotels-sample-index](search-get-
294
294
295
295
Currently, the only valid value for this parameter is "extractive". Captions can be configured to return results with or without highlights. The default is for highlights to be returned. This example returns captions without highlights: `extractive|highlight-false`.
296
296
297
-
For semantic captions, the fields referenced in the "semanticConfiguration" must have a word limit in the range of 2000-3000 words (or equivalent to 10000 tokens), otherwise, it will miss important caption results. If you anticipate that the fields used by the "semanticConfiguration" word count could be higher than the exposed limit and you need to use captions, consider [Text split cognitive skill]cognitive-search-skill-textsplit.md) as part of your [AI enrichment pipeline](cognitive-search-concept-intro.md) while indexing your data with [built-in pull indexers](search-indexer-overview.md).
297
+
For semantic captions, the fields referenced in the "semanticConfiguration" must have a word limit in the range of 2000-3000 words (or equivalent to 10,000 tokens), otherwise, it misses important caption results. If you anticipate that the fields used by the "semanticConfiguration" word count could be higher than the exposed limit and you need to use captions, consider [Text split cognitive skill]cognitive-search-skill-textsplit.md) as part of your [AI enrichment pipeline](cognitive-search-concept-intro.md) while indexing your data with [built-in pull indexers](search-indexer-overview.md).
298
298
299
299
1. Set "highlightPreTag" and "highlightPostTag" if you want to override the default highlight formatting that's applied to captions.
300
300
@@ -306,7 +306,7 @@ The following example in this section uses the [hotels-sample-index](search-get-
306
306
307
307
### [**.NET SDK**](#tab/dotnet-query)
308
308
309
-
Beta versions of the Azure SDKs include support for semantic search. Because the SDKs are beta versions, there's no documentation or samples, but you can refer to the REST API section above for insights on how the APIs should work.
309
+
Beta versions of the Azure SDKs include support for semantic search. Because the SDKs are beta versions, there's no documentation or samples, but you can refer to the REST API content in the next tab for insights on how the APIs should work.
310
310
311
311
The following beta versions support semantic configuration:
312
312
@@ -364,7 +364,7 @@ The response for the above example query returns the following match as the top
364
364
365
365
## Next steps
366
366
367
-
Recall that semantic ranking and responses are built over an initial result set. Any logic that improves the quality of the initial results will carry forward to semantic search. As a next step, review the features that contribute to initial results, including analyzers that affect how strings are tokenized, scoring profiles that can tune results, and the default relevance algorithm.
367
+
Recall that semantic ranking and responses are built over an initial result set. Any logic that improves the quality of the initial results carry forward to semantic search. As a next step, review the features that contribute to initial results, including analyzers that affect how strings are tokenized, scoring profiles that can tune results, and the default relevance algorithm.
368
368
369
369
+[Analyzers for text processing](search-analyzers.md)
Copy file name to clipboardExpand all lines: articles/search/semantic-search-overview.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,23 +33,23 @@ Semantic search is a premium feature that's billed by usage. We recommend this a
33
33
34
34
Semantic search is a collection of query-related capabilities that improve the quality of an initial BM25-ranked search result for text-based queries. When you enable it on your search service, semantic search extends the query execution pipeline in two ways:
35
35
36
-
* First, it adds secondary ranking over an initial result set that was scored using the BM25 algorithm, using language understanding models to promote the most semantically relevant results.
36
+
* First, it adds secondary ranking over an initial result set that was scored using the BM25 algorithm, using multi-lingual, deep learning models adapted from Microsoft Bing to promote the most semantically relevant results.
37
37
38
38
* Second, it extracts and returns captions and answers in the response, which you can render on a search page to improve the user's search experience.
39
39
40
40
Here are the features of semantic search.
41
41
42
42
| Feature | Description |
43
43
|---------|-------------|
44
-
| Semantic reranking| Uses the context or semantic meaning of a query to compute a new relevance score over existing BM25-ranked results. |
44
+
| Semantic ranking| Uses the context or semantic meaning of a query to compute a new relevance score over existing BM25-ranked results. |
45
45
|[Semantic captions and highlights](semantic-how-to-query-request.md)| Extracts verbatim sentences and phrases from a document that best summarize the content, with highlights over key passages for easy scanning. Captions that summarize a result are useful when individual content fields are too dense for the search results page. Highlighted text elevates the most relevant terms and phrases so that users can quickly determine why a match was considered relevant. |
46
46
|[Semantic answers](semantic-answers.md)| An optional and extra substructure returned from a semantic query. It provides a direct answer to a query that looks like a question. It requires that a document has text with the characteristics of an answer. |
47
47
48
48
## How semantic ranking works
49
49
50
50
Semantic ranking looks for context and relatedness among terms, elevating matches that make more sense for the query.
51
51
52
-
The following illustration explains the concept. Consider the term "capital". It has different meanings depending on whether the context is finance, law, geography, or grammar. Through language understanding, the semantic reranker can detect context and promote results that fit query intent.
52
+
The following illustration explains the concept. Consider the term "capital". It has different meanings depending on whether the context is finance, law, geography, or grammar. Through language understanding, the semantic ranker can detect context and promote results that fit query intent.
53
53
54
54
:::image type="content" source="media/semantic-search-overview/semantic-vector-representation.png" alt-text="Illustration of vector representation for context." border="true":::
0 commit comments