Merge pull request #1925 from HeidiSteen/heidist-freshness

[azure search] semantic ranking doc fixes

Author: v-dirichards

articles/search/hybrid-search-how-to-query.md

@@ -228,6 +228,8 @@ POST https://{{search-service-name}}.search.windows.net/indexes/{{index-name}}/d
 
 Assuming that you [enabled semantic ranker](semantic-how-to-enable-disable.md) and your index definition includes a [semantic configuration](semantic-how-to-query-request.md), you can formulate a query that includes vector search and keyword search, with semantic ranking over the merged result set. Optionally, you can add captions and answers. 
 
+Whenever you use semantic ranking with vectors, make sure `k` is set to 50. Semantic ranker uses up to 50 matches as input. Specifying less than 50 deprives the semantic ranking models of necessary inputs.
+
 ```http
 POST https://{{search-service-name}}.search.windows.net/indexes/{{index-name}}/docs/search?api-version=2024-07-01
 Content-Type: application/json
@@ -306,7 +308,7 @@ api-key: {{admin-api-key}}
 
 + Prefiltering is applied before query execution. If prefilter reduces the search area to 100 documents, the vector query executes over the "DescriptionVector" field for those 100 documents, returning the k=50 best matches. Those 50 matching documents then pass to RRF for merged results, and then to semantic ranker.
 
-+ Postfilter is applied after query execution. If k=50 returns 50 matches on the vector query side, then the post-filter is applied to the 50 matches, reducing results that meet filter criteria, leaving you with fewer than 50 documents to pass to semantic ranker.
++ Postfilter is applied after query execution. If k=50 returns 50 matches on the vector query side, followed by a post-filter applied to the 50 matches, your results are reduced by the number of documents that meet filter criteria. This leaves you with fewer than 50 documents to pass to semantic ranker. Keep this in mind if you're using semantic ranking. The semantic ranker works best if it has 50 documents as input.
 
 ## Set maxTextRecallSize and countAndFacetMode (preview)
 

articles/search/search-api-migration.md

@@ -72,7 +72,7 @@ Effective March 29, 2024 and applicable to all [supported REST APIs](/rest/api/s
 
 + For all API versions, updates on July 14, 2023 to the Microsoft-hosted semantic models made semantic ranker language-agnostic, effectively decommissioning the `queryLanguage` property. There's no "breaking change" in code, but the property is ignored.
 
-See [Migrate from preview version](semantic-how-to-configure.md#migrate-from-preview-versions) to transition your code to use `semanticConfiguration`.
+See [Migrate from preview version](semantic-code-migration.md) to transition your code to use `semanticConfiguration`.
 
 ## Upgrade to 2024-11-01-preview
 

articles/search/semantic-code-migration.md

@@ -0,0 +1,114 @@
+---
+title: Migrate semantic ranking code 
+titleSuffix: Azure AI Search
+description: Migrate semantic ranking code from preview to stable versions, and now to newer preview versions.
+manager: nitinme
+author: HeidiSteen
+ms.author: heidist
+ms.service: azure-ai-search
+ms.custom:
+  - ignite-2023
+ms.topic: how-to
+ms.date: 12/10/2024
+---
+
+# Migrate semantic ranking code from previous versions
+
+If your semantic ranking code was written against early preview APIs, this article identifies the code changes necessary for migrating to newer API versions. Breaking changes for semantic ranker are limited to query logic in recent APIs, but if your code was written against the initial preview version, you might need to change your semantic configuration as well.
+
+## Breaking changes
+
+There are two breaking changes for semantic ranker across REST API versions:
+
++ `searchFields` was replaced by `semanticConfiguration` in 2021-04-30-preview
++ `queryLanguage` was ignored starting in 2023-07-01-preview, but reinstated for query rewrite in 2024-11-01-preview
+
+Other version-specific updates pertain to new capabilities, but don't break existing code and are therefore not breaking changes.
+
+If you're using Azure SDKs, multiple APIs have been renamed over time. The SDK change logs provide the details.
+
+## API versions providing semantic ranking
+
+Check your code for the REST API version or SDK package version to confirm which one provides semantic ranking. The following API versions have some level of support for semantic ranking.
+
+| Release type | REST API version | Semantic ranker updates |
+|--|--|--|
+| initial | [2020-06-30-preview](/rest/api/searchservice/preview-api/search-documents) | Adds `queryType=semantic` to Search Documents |
+| preview | [2021-04-30-preview](/rest/api/searchservice/preview-api/search-documents)  | Adds `semanticConfiguration` to Create or Update Index |
+| preview | [2023-07-01-preview](/rest/api/searchservice/preview-api/search-documents) | Updates `semanticConfiguration`. Starting on July 14, 2023 updates to the Microsoft-hosted semantic models made semantic ranker language-agnostic, effectively decommissioning the `queryLanguage` property for semantic ranking. There's no breaking change in code, but the property is ignored. Customers were advised to remove this property from code.|
+| preview | [2023-10-01-preview](/rest/api/searchservice/operation-groups?view=rest-searchservice-2023-10-01-preview&preserve-view=true) | Adds `semanticQuery` to send a query used only for reranking purposes. |
+| stable | [2023-11-01](/rest/api/searchservice/operation-groups?view=rest-searchservice-2023-11-01&preserve-view=true) | Generally available. Introduced changes to `semanticConfiguration` that progressed to the stable version. If your code targets this version or later, it's compatible with newer API versions unless you adopt new preview features.|
+| preview | [2024-05-01-preview](/rest/api/searchservice/operation-groups?view=rest-searchservice-2024-05-01-preview&preserve-view=true) | No change |
+| stable | [2024-07-01](/rest/api/searchservice/indexes/create-or-update?view=rest-searchservice-2024-07-01&preserve-view=true) | No change |
+| preview | [2024-09-01-preview](/rest/api/searchservice/operation-groups?view=rest-searchservice-2024-09-01-preview&preserve-view=true) | No change |
+| preview | [2024-11-01-preview](/rest/api/searchservice/operation-groups?view=rest-searchservice-2024-11-01-preview&preserve-view=true) | Adds query rewrite. The `queryLanguage` property is now required if you use [query rewrite (preview)](semantic-how-to-query-rewrite.md).  |
+
+## Change logs for Azure SDKs
+
+Azure SDKs are on an independent release schedule. You should check the change logs to determine which packages provide semantic features and whether any APIs have been renamed.
+
++ [Azure SDK for .NET change log](https://github.com/Azure/azure-sdk-for-net/blob/Azure.Search.Documents_11.5.1/sdk/search/Azure.Search.Documents/CHANGELOG.md#1150-2023-11-10&preserve-view=true)
++ [Azure SDK for Python change log](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/search/azure-search-documents/CHANGELOG.md#1140-2023-10-13&preserve-view=true)
++ [Azure SDK for Java change log](https://github.com/Azure/azure-sdk-for-java/blob/azure-search-documents_11.6.1/sdk/search/azure-search-documents/CHANGELOG.md#1160-2023-11-13&preserve-view=true)
++ [Azure SDK for JavaScript change log](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/search-documents_12.0.0/sdk/search/search-documents/CHANGELOG.md#1200-2023-11-13&preserve-view=true)
+
+## 2024-11-01-preview
+
++ Adds [query rewrite](semantic-how-to-query-rewrite.md) to Search Documents.
++ Requires `queryLanguage` for query rewrite workloads. For a list of valid values, see the [REST API](/rest/api/searchservice/documents/search-post?view=rest-searchservice-2024-11-01-preview#querylanguage&preserve-view=true).
+
+## 2024-09-01-preview
+
+No changes to semantic ranking syntax from the 2024-07-01 stable version.
+
+## 2024-07-01
+
+No changes to semantic ranking syntax from the 2024-05-01-preview version.
+
+Don't use this API version. It implements a vector query syntax that's incompatible with any newer API version.
+
+## 2024-05-01-preview
+
+No changes to semantic ranking syntax from the 2024-03-01-preview version.
+
+## 2024-03-01-preview
+
+No changes to semantic ranking syntax from the 2023-10-01-preview version, but vector queries are introduced. Semantic ranking now applies to responses from hybrid and vector queries. You can apply reranking on any human-readable text fields in the response, assuming the fields are listed in `prioritizedFields`.
+
+## 2023-11-01
+
++ Excludes `SemanticDebug` and `semanticQuery`, otherwise the same as the 2023-10-01-preview version.
+
+## 2023-10-01-preview
+
++ Adds `semanticQuery`
+
+## 2023-07-01-preview
+
++ Adds `semanticErrorHandling`, `semanticMaxWaitInMilliseconds`.
++ Adds numerous semantic-related fields to the response, such as `SemanticDebug` and `SemanticErrorMode`.
++ Ignores `queryLanguage`, it's no longer used in semantic ranking.
+
+Starting on July 14, 2023, semantic ranker is language agnostic. In preview versions, semantic ranking would deprioritize results differing from the `querylanguage` specified by the field analyzer. However, the `queryLanguage` property is still applicable to [spell correction](speller-how-to-add.md) and the short list of languages supported by that feature.
+
+## 2021-04-30-preview
+
++ Semantic support is through [Search Documents](/rest/api/searchservice/preview-api/search-documents) and [Create or Update Index](/rest/api/searchservice/preview-api/create-or-update-index) preview API calls.
++ Adds `semanticConfiguration` to a search index. A semantic configuration has a name and a prioritized field list.
++ Adds ``prioritizedFields`.
+
+The `searchFields` property is no longer used to prioritize fields. In all versions moving forward, `semanticConfiguration.prioritizedFields` replaces `searchFields` as the mechanism for specifying which fields to use for L2 ranking.
+
+## 2020-06-30-preview
+
++ Semantic support is through a [Search Documents](/rest/api/searchservice/preview-api/search-documents) preview API call.
++ Adds `queryType=semantic` to the query request.
++ Adapts `searchFields` so that if the query type is semantic, the `searchFields` property determines the priority order of field inputs to the semantic ranker.
++ Adds `captions`, `answers`, and `highlights` to the query response.
+
+## Next steps
+
+Test your semantic configuration migration by running a semantic query.
+
+> [!div class="nextstepaction"]
+> [Create a semantic query](semantic-how-to-query-request.md)

articles/search/semantic-how-to-configure.md

@@ -9,14 +9,17 @@ ms.service: azure-ai-search
 ms.custom:
   - ignite-2023
 ms.topic: how-to
-ms.date: 10/20/2024
+ms.date: 12/10/2024
 ---
 
 # Configure semantic ranker and return captions in search results
 
 Semantic ranking iterates over an initial result set, applying an L2 ranking methodology that promotes the most semantically relevant results to the top of the stack. You can also get semantic captions, with highlights over the most relevant terms and phrases, and [semantic answers](semantic-answers.md).
 
-This article explains how to configure a search index for semantic reranking. 
+This article explains how to configure a search index for semantic reranking.
+
+> [!NOTE]
+> If you have existing code that calls preview or previous API versions, see [Migrate semantic ranking code](semantic-code-migration.md) for help with modifying your code.
 
 ## Prerequisites
 
@@ -28,10 +31,10 @@ This article explains how to configure a search index for semantic reranking.
 
 ## Choose a client
 
-You can use any of the following tools and software development kits (SDKs) to add a semantic configuration:
+You can specify a semantic configuration on new or existing indexes, using any of the following tools and software development kits (SDKs) to add a semantic configuration:
 
 + [Azure portal](https://portal.azure.com), using the index designer to add a semantic configuration.
-+ [Visual Studio Code](https://code.visualstudio.com/download) with the [REST client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client)
++ [Visual Studio Code](https://code.visualstudio.com/download) with the [REST client](https://marketplace.visualstudio.com/items?itemName=humao.rest-client) and a [Create or Update Index (REST) API](/rest/api/searchservice/indexes/create-or-update).
 + [Azure SDK for .NET](https://www.nuget.org/packages/Azure.Search.Documents)
 + [Azure SDK for Python](https://pypi.org/project/azure-search-documents)
 + [Azure SDK for Java](https://central.sonatype.com/artifact/com.azure/azure-search-documents)
@@ -155,31 +158,6 @@ SearchIndex searchIndex = new(indexName)
 
 ---
 
-## Migrate from preview versions
-
-If your semantic ranking code is using preview APIs, this section explains how to migrate to stable versions. You can check the change logs for verification of general availability:
-
-+ [2024-07-01 (REST)](/rest/api/searchservice/indexes/create-or-update?view=rest-searchservice-2024-07-01&preserve-view=true)
-+ [Azure SDK for .NET (11.5) change log](https://github.com/Azure/azure-sdk-for-net/blob/Azure.Search.Documents_11.5.1/sdk/search/Azure.Search.Documents/CHANGELOG.md#1150-2023-11-10)
-+ [Azure SDK for Python (11.4) change log](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/search/azure-search-documents/CHANGELOG.md#1140-2023-10-13)
-+ [Azure SDK for Java (11.6) change log](https://github.com/Azure/azure-sdk-for-java/blob/azure-search-documents_11.6.1/sdk/search/azure-search-documents/CHANGELOG.md#1160-2023-11-13)
-+ [Azure SDK for JavaScript (12.0) change log](https://github.com/Azure/azure-sdk-for-js/blob/%40azure/search-documents_12.0.0/sdk/search/search-documents/CHANGELOG.md#1200-2023-11-13)
-
-
-### queryLanguage for semantic ranker
-
-As of July 14, 2023, semantic ranker is language agnostic. It can rerank results composed of multilingual content, with no bias towards a specific language. In preview versions, semantic ranking would deprioritize results differing from the language specified by the field analyzer.
-
-Stop using `queryLanguage` in your code if you were using it for semantic ranking. The `queryLanguage` property is still applicable to features such as [spell correction](speller-how-to-add.md), but not to semantic ranking.
-
-### searchFields for semantic ranker
-
-For the REST API and all SDK packages targeting version `2021-04-30-Preview` and later, the `searchFields` property is no longer used for semantic ranking.
-
-Instead, use the `semanticConfiguration` property (in a search index) to determine which search fields are used in semantic ranking. To specify field prioritization, add a `semanticConfiguration` to in an index schema following the [instructions in this article](#add-a-semantic-configuration).
-
-You can keep `searchFields` in query requests if you're using it to limit full text search to the list of named fields. 
-
 ## Next steps
 
 Test your semantic configuration by running a semantic query.

articles/search/semantic-how-to-enable-disable.md

@@ -10,12 +10,12 @@ ms.service: azure-ai-search
 ms.custom:
   - ignite-2023
 ms.topic: how-to
-ms.date: 10/28/2024
+ms.date: 12/10/2024
 ---
 
 # Enable or disable semantic ranker
 
-Semantic ranker is a premium feature billed by usage. By default, semantic ranker is turned off when you create a new search service, but anyone with *Contributor* permissions can enable it. If you don't want anyone enabling it inadvertently, you can [disable it using the REST API](#disable-semantic-ranker-using-the-rest-api).
+Semantic ranker is a premium feature billed by usage. By default, semantic ranker is turned off when you create a new search service, but anyone with *Contributor* permissions can enable it. If you don't want anyone enabling it inadvertently, you can [disable it service-wide using the management REST API](#disable-semantic-ranker-using-the-rest-api).
 
 ## Check availability
 

articles/search/semantic-how-to-query-request.md

@@ -11,14 +11,14 @@ ms.custom:
   - ignite-2023
   - ignite-2024
 ms.topic: how-to
-ms.date: 11/19/2024
+ms.date: 12/10/2024
 ---
 
 # Add semantic ranking to queries in Azure AI Search
 
 You can apply semantic ranking to text queries, hybrid queries, and vector queries if your search documents contain string fields and the [vector query has a text representation](vector-search-how-to-query.md#query-with-integrated-vectorization) in the search document.
 
-This article explains how to invoke the semantic ranker on queries. 
+This article explains how to invoke the semantic ranker on queries. It assumes you're using the most recent stable or preview APIs. For help with older versions, see [Migrate semantic ranking code](semantic-code-migration.md).
 
 ## Prerequisites
 

articles/search/semantic-search-overview.md

@@ -10,7 +10,7 @@ ms.service: azure-ai-search
 ms.custom:
   - ignite-2023
 ms.topic: concept-article
-ms.date: 09/24/2024
+ms.date: 12/10/2024
 ---
 
 # Semantic ranking in Azure AI Search
@@ -20,13 +20,13 @@ In Azure AI Search, *semantic ranker* is a feature that measurably improves sear
 Semantic ranker is a premium feature, billed by usage. We recommend this article for background, but if you'd rather get started, [follow these steps](#how-to-get-started-with-semantic-ranker).
 
 > [!NOTE]
-> Semantic ranker doesn't use generative AI or vectors. If you're looking for vector support and similarity search, see [Vector search in Azure AI Search](vector-search-overview.md) for details.
+> Semantic ranker doesn't use generative AI or vectors. If you're looking for vectors and similarity search, see [Vector search in Azure AI Search](vector-search-overview.md) for details.
 
 ## What is semantic ranking?
 
-Semantic ranker is a collection of query-side capabilities that improve the quality of an initial [BM25-ranked](index-similarity-and-scoring.md) or [RRF-ranked](hybrid-search-ranking.md) search result for text-based queries, vector queries, and hybrid queries. When you enable it on your search service, semantic ranking extends the query execution pipeline in two ways: 
+Semantic ranker is a collection of query-side capabilities that improve the quality of an initial [BM25-ranked](index-similarity-and-scoring.md) or [RRF-ranked](hybrid-search-ranking.md) search result for text-based queries, vector queries, and hybrid queries. When you enable it on your search service, semantic ranking extends the query execution pipeline in two ways:
 
-* First, it adds secondary ranking over an initial result set that was scored using BM25 or Reciprocal Rank Fusion (RRF). This secondary ranking uses multi-lingual, deep learning models adapted from Microsoft Bing to promote the most semantically relevant results. 
+* First, it adds secondary ranking over an initial result set that was scored using BM25 or Reciprocal Rank Fusion (RRF). This secondary ranking uses multi-lingual, deep learning models adapted from Microsoft Bing to promote the most semantically relevant results.
 
 * Second, it extracts and returns captions and answers in the response, which you can render on a search page to improve the user's search experience.
 
@@ -72,11 +72,11 @@ In semantic ranking, the query subsystem passes search results as an input to su
 
 1. Summarization output is a summary string for each document, composed of the most relevant information from each field. Summary strings are sent to the ranker for scoring, and to machine reading comprehension models for captions and answers.
 
-   The maximum length of each generated summary string passed to the semantic ranker is 256 tokens. 
+   As of November 2024, the maximum length of each generated summary string passed to the semantic ranker is 2,048 tokens. Previously, it was 256 tokens.
 
 ### How ranking is scored
 
-Scoring is done over the caption, and any other content from the summary string that fills out the 256 token length.
+Scoring is done over the caption, and any other content from the summary string that fills out the 2,048 token length.
 
 1. Captions are evaluated for conceptual and semantic relevance, relative to the query provided.
 
@@ -131,7 +131,7 @@ Semantic ranker is available on search services at the Basic and higher tiers, s
 
 When you enable semantic ranker, choose a pricing plan for the feature:
 
-* At lower query volumes (under 1000 monthly), semantic ranking is free.
+* At lower query volumes (under 1,000 monthly), semantic ranking is free.
 * At higher query volumes, choose the standard pricing plan.
 
 The [Azure AI Search pricing page](https://azure.microsoft.com/pricing/details/search/) shows you the billing rate for different currencies and intervals.