Merge pull request #1905 from HeidiSteen/heidist-freshness

Update index content, port material from old REST API to concept docs

Author: AnnaMHuff

articles/search/search-how-to-define-index-projections.md

@@ -292,8 +292,7 @@ For data sources that provide change tracking and deletion detection, an indexer
 
 If you add new content to your data source, new chunks or child documents are added to the index on the next indexer run.
 
-If you modify existing content in the data source, chunks are updated incrementally in the search index if the data source you're using supports change tracking and deletion detection. For exammple, if a word or sentence changes in a document, the chunk in the target index that contains that word or sentence is updated on the next indexer run. Other types of updates, such as changing a field type and some attributions, aren't supported for existing fields. For more information about allowed updates, see [
-Change an index schema](search-howto-reindex.md#change-an-index-schema).
+If you modify existing content in the data source, chunks are updated incrementally in the search index if the data source you're using supports change tracking and deletion detection. For exammple, if a word or sentence changes in a document, the chunk in the target index that contains that word or sentence is updated on the next indexer run. Other types of updates, such as changing a field type and some attributions, aren't supported for existing fields. For more information about allowed updates, see [Update an index schema](search-howto-reindex.md#update-an-index-schema).
 
 Some data sources like [Azure Storage](search-howto-index-changed-deleted-blobs.md) support change and deletion tracking by default, based on the timestamp. Other data sources such as [OneLake](search-how-to-index-onelake-files.md), [Azure SQL](search-how-to-index-sql-database.md), or [Azure Cosmos DB](search-howto-index-cosmosdb.md) must be configured for change tracking.
 

articles/search/search-howto-reindex.md

@@ -11,7 +11,7 @@ ms.service: azure-ai-search
 ms.custom:
   - ignite-2024
 ms.topic: how-to
-ms.date: 08/14/2024
+ms.date: 12/09/2024
 ---
 
 # Update or rebuild an index in Azure AI Search
@@ -20,28 +20,126 @@ This article explains how to update an existing index in Azure AI Search with sc
 
 During active development, it's common to drop and rebuild indexes when you're iterating over index design. Most developers work with a small representative sample of their data so that reindexing goes faster.
 
-For schema changes on applications already in production, we recommend creating and testing a new index that runs side by side an existing index. Use an [index alias](search-how-to-alias.md) to swap in the new index while avoiding changes your application code.
+For schema changes on applications already in production, we recommend creating and testing a new index that runs side by side an existing index. Use an [index alias](search-how-to-alias.md) to swap in the new index so that you can avoid changes your application code.
 
 ## Update content
 
-Incremental indexing and synchronizing an index against changes in source data is fundamental to most search applications. This section explains the workflow for updating field contents in a search index.
+Incremental indexing and synchronizing an index against changes in source data is fundamental to most search applications. This section explains the workflow for updating field contents in a search index through the REST API, but the Azure SDKs provide equivalent functionality.
 
-1. Use the same techniques for loading documents: [Documents - Index (REST)](/rest/api/searchservice/documents) or an equivalent API in the Azure SDKs. For more information about indexing techniques, see [Load documents](search-how-to-load-search-index.md).
+The body of the request contains one or more documents to be indexed. Documents are identified by a unique case-sensitive key. Each document is associated with an action: "upload", "delete", "merge", or "mergeOrUpload". Upload requests must include the document data as a set of key/value pairs.
 
-1. Set the `@search.action` parameter to determine the effect on existing documents:
+```json
+{  
+  "value": [  
+    {  
+      "@search.action": "upload (default) | merge | mergeOrUpload | delete",  
+      "key_field_name": "unique_key_of_document", (key/value pair for key field from index schema)  
+      "field_name": field_value (key/value pairs matching index schema)  
+        ...  
+    },  
+    ...  
+  ]  
+}
+```
+
++ First, use the APIs for loading documents, such as [Documents - Index (REST)](/rest/api/searchservice/documents) or an equivalent API in the Azure SDKs. For more information about indexing techniques, see [Load documents](search-how-to-load-search-index.md).
+
++ For a large update, batching (up to 1,000 documents per batch, or about 16 MB per batch, whichever limit comes first) is recommended and significantly improves indexing performance.
+
++ Set the `@search.action` parameter on the API to determine the effect on existing documents.
 
    | Action | Effect |
    |--------|--------|
    | delete | Removes the entire document from the index. If you want to remove an individual field, use merge instead, setting the field in question to null. Deleted documents and fields don't immediately free up space in the index. Every few minutes, a background process performs the physical deletion. Whether you use the Azure portal or an API to return index statistics, you can expect a small delay before the deletion is reflected in the Azure portal and through APIs. |
-   | merge | Updates a document that already exists, and fails a document that can't be found. Merge replaces existing values. For this reason, be sure to check for collection fields that contain multiple values, such as fields of type `Collection(Edm.String)`. For example, if a `tags` field starts with a value of `["budget"]` and you execute a merge with `["economy", "pool"]`, the final value of the `tags` field is `["economy", "pool"]`. It won't be `["budget", "economy", "pool"]`. |
+   | merge | Updates a document that already exists, and fails a document that can't be found. Merge replaces existing values. For this reason, be sure to check for collection fields that contain multiple values, such as fields of type `Collection(Edm.String)`. For example, if a `tags` field starts with a value of `["budget"]` and you execute a merge with `["economy", "pool"]`, the final value of the `tags` field is `["economy", "pool"]`. It won't be `["budget", "economy", "pool"]`. <br><br>The same behavior applies to complex collections. If the document contains a complex collection field named Rooms with a value of `[{ "Type": "Budget Room", "BaseRate": 75.0 }]`, and you execute a merge with a value of `[{ "Type": "Standard Room" }, { "Type": "Budget Room", "BaseRate": 60.5 }]`, the final value of the Rooms field will be `[{ "Type": "Standard Room" }, { "Type": "Budget Room", "BaseRate": 60.5 }]`. It won't append or merge new and existing values. |
    | mergeOrUpload | Behaves like merge if the document exists, and upload if the document is new. This is the most common action for incremental updates. |
    | upload | Similar to an "upsert" where the document is inserted if it's new, and updated or replaced if it exists. If the document is missing values that the index requires, the document field's value is set to null. |
 
-1. Post the update.
+Queries continue to run during indexing, but if you're updating or removing existing fields, you can expect mixed results and a higher incidence of throttling.
+
+> [!NOTE]
+> There are no ordering guarantees for which action in the request body is executed first. It's not recommended to have multiple "merge" actions associated with the same document in a single request body. If there are multiple "merge" actions required for the same document, perform the merging client-side before updating the document in the search index.
+
+### Responses
+
+Status code 200 is returned for a successful response, meaning that all items have been stored durably and will start to be indexed. Indexing runs in the background and makes new documents available (that is, queryable and searchable) a few seconds after the indexing operation completed. The specific delay depends on the load on the service.
+
+Successful indexing is indicated by the status property being set to true for all items, as well as the `statusCode` property being set to either 201 (for newly uploaded documents) or 200 (for merged or deleted documents):
+
+```json
+{
+  "value": [
+    {
+      "key": "unique_key_of_new_document",
+      "status": true,
+      "errorMessage": null,
+      "statusCode": 201
+    },
+    {
+      "key": "unique_key_of_merged_document",
+      "status": true,
+      "errorMessage": null,
+      "statusCode": 200
+    },
+    {
+      "key": "unique_key_of_deleted_document",
+      "status": true,
+      "errorMessage": null,
+      "statusCode": 200
+    }
+  ]
+}
+```
+
+Status code 207 is returned when at least one item wasn't successfully indexed. Items that haven't been indexed have the status field set to false. The `errorMessage` and `statusCode` properties indicate the reason for the indexing error:
 
-Queries continue to run, but if you're updating or removing existing fields, you can expect mixed results and a higher incidence of throttling.
+```json
+{
+  "value": [
+    {
+      "key": "unique_key_of_document_1",
+      "status": false,
+      "errorMessage": "The search service is too busy to process this document. Please try again later.",
+      "statusCode": 503
+    },
+    {
+      "key": "unique_key_of_document_2",
+      "status": false,
+      "errorMessage": "Document not found.",
+      "statusCode": 404
+    },
+    {
+      "key": "unique_key_of_document_3",
+      "status": false,
+      "errorMessage": "Index is temporarily unavailable because it was updated with the 'allowIndexDowntime' flag set to 'true'. Please try again later.",
+      "statusCode": 422
+    }
+  ]
+}  
+```
+
+The `errorMessage` property indicates the reason for the indexing error if possible.
+
+The following table explains the various per-document status codes that can be returned in the response. Some status codes indicate problems with the request itself, while others indicate temporary error conditions. The latter you should retry after a delay.
+
+| Status code | Meaning | Retryable | Notes |
+|-------------|---------|-----------|-------|
+| 200 | Document was successfully modified or deleted. | n/a | Delete operations are idempotent. That is, even if a document key doesn't exist in the index, attempting a delete operation with that key results in a 200 status code. |
+| 201 | Document was successfully created. | n/a |  |
+| 400 | There was an error in the document that prevented it from being indexed. | No | The error message in the response indicates what is wrong with the document.|
+| 404 | The document couldn't be merged because the given key doesn't exist in the index. | No | This error doesn't occur for uploads since they create new documents, and it doesn't occur for deletes because they're idempotent. |
+| 409 | A version conflict was detected when attempting to index a document.| Yes | This can happen when you're trying to index the same document more than once concurrently. |
+| 422 | The index is temporarily unavailable because it was updated with the 'allowIndexDowntime' flag set to 'true'. | Yes | |
+| 503 | Your search service is temporarily unavailable, possibly due to heavy load. | Yes | Your code should wait before retrying in this case or you risk prolonging the service unavailability.|
 
-## Tips for incremental indexing
+If your client code frequently encounters a 207 response, one possible reason is that the system is under load. You can confirm this by checking the statusCode property for 503. If this is the case, we recommend throttling indexing requests. Otherwise, if indexing traffic doesn't subside, the system could start rejecting all requests with 503 errors.
+
+Status code 429 indicates that you have exceeded your quota on the number of documents per index. You must either create a new index or upgrade for higher capacity limits.
+
+> [!NOTE]
+> When you upload `DateTimeOffset` values with time zone information to your index, Azure AI Search normalizes these values to UTC. For example, 2024-01-13T14:03:00-08:00 is stored as 2024-01-13T22:03:00Z. If you need to store time zone information, add an extra column to your index for this data point.
+
+### Tips for incremental indexing
 
 + [Indexers automate incremental indexing](search-indexer-overview.md). If you can use an indexer, and if the data source supports change tracking, you can run the indexer on a recurring schedule to add, update, or overwrite searchable content so that it's synchronized to your external data.
 
@@ -88,18 +186,22 @@ GET  {{baseUrl}}/indexes/hotels-vector-quickstart/docs('1')?api-version=2024-07-
     api-key: {{apiKey}}
 ```
 
-## Change an index schema
+## Update an index schema
+
+The index schema defines the physical data structures created on the search service, so there aren't many schema changes that you can make without incurring a full rebuild.
 
-The index schema defines the physical data structures created on the search service, so there aren't many schema changes that you can make without incurring a full rebuild. The following list enumerates the schema changes that can be introduced seamlessly into an existing index. Generally, the list includes new fields and functionality used during query execution.
+### Updates with no rebuild
+
+The following list enumerates the schema changes that can be introduced seamlessly into an existing index. Generally, the list includes new fields and functionality used during query execution.
 
 + Add a new field
 + Set the `retrievable` attribute on an existing field
 + Update `searchAnalyzer` on a field having an existing `indexAnalyzer`
-+ Add a new analyzer definition in an index (which can be applied to new fields)
-+ Add, update, or delete scoring profiles
++ Add a new [analyzer definition](index-add-custom-analyzers.md) in an index (which can be applied to new fields)
++ Add, update, or delete [scoring profiles](index-add-scoring-profiles.md)
++ Add, update, or delete [synonymMaps](search-synonyms.md)
++ Add, update, or delete [semantic configurations](semantic-how-to-configure.md)
 + Add, update, or delete CORS settings
-+ Add, update, or delete synonymMaps
-+ Add, update, or delete semantic configurations
 
 The order of operations is:
 
@@ -115,7 +217,7 @@ When you update an index schema to include a new field, existing documents in th
 
 There should be no query disruptions during the updates, but query results will vary as the updates take effect.
 
-## Drop and rebuild an index
+### Updates requiring a rebuild
 
 Some modifications require an index drop and rebuild, replacing a current index with a new one.
 
@@ -144,6 +246,8 @@ The order of operations is:
 
 When you create the index, physical storage is allocated for each field in the index schema, with an inverted index created for each searchable field and a vector index created for each vector field. Fields that aren't searchable can be used in filters or expressions, but don't have inverted indexes and aren't full-text or fuzzy searchable. On an index rebuild, these inverted indexes and vector indexes are deleted and recreated based on the index schema you provide.
 
+To minimize disruption to application code, consider [creating an index alias](search-how-to-alias.md). Application code references the alias, but you can update the name of the index that the alias points to.
+
 ## Balancing workloads
 
 Indexing doesn't run in the background, but the search service will balance any indexing jobs against ongoing queries. During indexing, you can [monitor query requests](search-monitor-queries.md) in the Azure portal to ensure queries are completing in a timely manner.
@@ -156,7 +260,13 @@ You can begin querying an index as soon as the first document is loaded. If you
 
 You can use [Search Explorer](search-explorer.md) or a [REST client](search-get-started-rest.md) to check for updated content.
 
-If you added or renamed a field, use [$select](search-query-odata-select.md) to return that field: `search=*&$select=document-id,my-new-field,some-old-field&$count=true`.
+If you added or renamed a field, use [select](search-query-odata-select.md) to return that field: 
+
+```json
+"search": "*",
+"select": "document-id, my-new-field, some-old-field",
+"count": true
+```
 
 The Azure portal provides index size and vector index size. You can check these values after updating an index, but remember to expect a small delay as the service processes the change and to account for portal refresh rates, which can be a few minutes.
 

articles/search/search-limits-quotas-capacity.md

@@ -198,17 +198,18 @@ Static rate request limits for operations related to a service:
 
 + Service Statistics (GET /servicestats): 4 per second per search unit
 
-### Semantic Ranker Throttling limits
+### Semantic ranker throttling limits
 
 [Semantic ranker](search-get-started-semantic.md) uses a queuing system to manage concurrent requests. This system allows search services get the highest number of queries per second possible. When the limit of concurrent requests is reached, additional requests are placed in a queue. If the queue is full, further requests are rejected and must be retried.
 
 Total semantic ranker queries per second varies based on the following factors:
-+ The SKU of the search service. Both queue capacity and concurrent request limits vary by SKU.
+
++ The tier of the search service. Both queue capacity and concurrent request limits vary by tier.
 + The number of search units in the search service. The simplest way to increase the maximum number of concurrent semantic ranker queries is to [add more search units to your search service](search-capacity-planning.md#how-to-change-capacity).
 + The total available semantic ranker capacity in the region.
 + The amount of time it takes to serve a query using semantic ranker. This varies based on how busy the search service is.
 
-The following table describes the semantic ranker throttling limits by SKU. Subject to available capacity in the region, contact support to request a limit increase.
+The following table describes the semantic ranker throttling limits by tier, subject to available capacity in the region. You can contact Microsoft support to request a limit increase.
 
 | Resource | Basic | S1 | S2 | S3 | S3-HD | L1 | L2 |
 |----------|-------|----|----|----|-------|----|----|