Merge pull request #262141 from HeidiSteen/heidist-fix

JamesJBarnett · web-flow · commit 51f39cff35e2 · 2024-01-02T20:17:13.000-07:00
[azure search] GH issue (storage limits)
diff --git a/articles/search/search-limits-quotas-capacity.md b/articles/search/search-limits-quotas-capacity.md
@@ -8,7 +8,7 @@ author: HeidiSteen
 ms.author: heidist
 ms.service: cognitive-search
 ms.topic: conceptual
-ms.date: 11/16/2023
+ms.date: 01/02/2024
 ms.custom:
   - references_regions
   - ignite-2023
@@ -18,7 +18,7 @@ ms.custom:
 
 Maximum limits on storage, workloads, and quantities of indexes and other objects depend on whether you [provision Azure AI Search](search-create-service-portal.md) at **Free**, **Basic**, **Standard**, or **Storage Optimized** pricing tiers.
 
-+ **Free** is a multi-tenant shared service that comes with your Azure subscription. 
++ **Free** is a multitenant shared service that comes with your Azure subscription. 
 
 + **Basic** provides dedicated computing resources for production workloads at a smaller scale, but shares some networking infrastructure with other tenants.
 
@@ -29,7 +29,7 @@ Maximum limits on storage, workloads, and quantities of indexes and other object
 ## Subscription limits
 [!INCLUDE [azure-search-limits-per-subscription](../../includes/azure-search-limits-per-subscription.md)]
 
-## Storage limits
+## Service limits
 [!INCLUDE [azure-search-limits-per-service](../../includes/azure-search-limits-per-service.md)]
 
 <a name="index-limits"></a>
@@ -59,7 +59,7 @@ You might find some variation in maximum limits if your service happens to be pr
 
 ## Document limits 
 
-There are no longer any document limits per service in Azure AI Search, however, there's a limit of approximately 24 billion documents per index on Basic, S1, S2, S3, L1, and L2 search services. For S3 HD, the limit is 2 billion documents per index. Each element of a complex collection counts as a separate document in terms of these limits.
+You can have approximately 24 billion documents per index on Basic, S1, S2, S3, L1, and L2 search services. For S3 HD, the limit is 2 billion documents per index. Each instance of a complex collection counts as a separate document in terms of these limits.
 
 ### Document size limits per API call
 
@@ -73,11 +73,11 @@ When estimating document size, remember to consider only those fields that can b
 
 When you index documents with vector fields, Azure AI Search constructs internal vector indexes using the algorithm parameters you provide. The size of these vector indexes is restricted by the memory reserved for vector search for your service's tier (or SKU).
 
-The service enforces a vector index size quota **for every partition** in your search service. Each extra partition increases the available vector index size quota. This quota is a hard limit to ensure your service remains healthy, which means that further indexing attempts once the limit is exceeded results in failure. You may resume indexing once you free up available quota by either deleting some vector documents or by scaling up in partitions.
+The service enforces a vector index size quota **for every partition** in your search service. Each extra partition increases the available vector index size quota. This quota is a hard limit to ensure your service remains healthy, which means that further indexing attempts once the limit is exceeded results in failure. You can resume indexing once you free up available quota by either deleting some vector documents or by scaling up in partitions.
 
 The table describes the vector index size quota per partition across the service tiers (or SKU). For context, it includes:
 
-+ [Storage limits](#storage-limits) for each tier, repeated here for context.
++ [Partition storage limits](#service-limits) for each tier, repeated here for context.
 + Amount of each partition (in GB) available for vector indexes (created when you add vector fields to an index).
 + Approximate number of embeddings (floating point values) per partition.
 
@@ -125,8 +125,8 @@ Maximum running times exist to provide balance and stability to the service as a
 | Maximum skillsets <sup>4</sup> |3 |5 or 15 |50 |200 |200 |N/A |10 |10 |
 | Maximum indexing load per invocation |10,000 documents |Limited only by maximum documents |Limited only by maximum documents |Limited only by maximum documents |Limited only by maximum documents |N/A |No limit |No limit |
 | Minimum schedule | 5 minutes |5 minutes |5 minutes |5 minutes |5 minutes |5 minutes |5 minutes | 5 minutes |
-| Maximum running time <sup>6</sup>| 1-3 minutes |2 or 24 hours |2 or 24 hours |2 or 24 hours |2 or 24 hours |N/A  |2 or 24 hours |2 or 24 hours |
-| Maximum running time for indexers with a skillset <sup>5</sup> | 3-10 minutes |2 hours |2 hours |2 hours |2 hours |N/A  |2 hours |2 hours |
+| Maximum running time <sup>5</sup>| 1-3 minutes |2 or 24 hours |2 or 24 hours |2 or 24 hours |2 or 24 hours |N/A  |2 or 24 hours |2 or 24 hours |
+| Maximum running time for indexers with a skillset <sup>6</sup> | 3-10 minutes |2 hours |2 hours |2 hours |2 hours |N/A  |2 hours |2 hours |
 | Blob indexer: maximum blob size, MB |16 |16 |128 |256 |256 |N/A  |256 |256 |
 | Blob indexer: maximum characters of content extracted from a blob |32,000 |64,000 |4&nbsp;million |8&nbsp;million |16&nbsp;million |N/A |4&nbsp;million |4&nbsp;million |
 
@@ -138,9 +138,9 @@ Maximum running times exist to provide balance and stability to the service as a
 
 <sup>4</sup> Maximum of 30 skills per skillset.
 
-<sup>5</sup> AI enrichment and image analysis are computationally intensive and consume disproportionate amounts of available processing power. Running time for these workloads has been shortened to give other jobs in the queue more opportunity to run.
+<sup>5</sup> Regarding the 2 or 24 hour maximum duration for indexers: a 2-hour maximum is the most common and it's what you should plan for. The 24-hour limit is from an older indexer implementation. If you have unscheduled indexers that run continuously for 24 hours, it's because those indexers couldn't be migrated to the newer runtime behavior. For extra large data sets, indexers can be made to run longer than maximum limits if you put them on a [2-hour run time schedule](search-howto-schedule-indexers.md). When the first 2-hour interval is complete, the indexer picks up where it left off to start the next 2-hour interval.
 
-<sup>6</sup> Indexer execution and combined indexer-skillset execution is subject to a 2-hour maximum duration.  Currently, some indexers have a longer 24-hour maximum execution window, but that behavior isn't the norm. The longer window only applies if a service or its indexers can't be internally migrated to the newer runtime behavior. If more than 2 hours are needed to complete an indexer or indexer-skillset process, [schedule the indexer](search-howto-schedule-indexers.md) to run at 2-hour intervals.
+<sup>6</sup> Skillset execution, and image analysis in particular, are computationally intensive and consume disproportionate amounts of available processing power. Running time for these workloads has been shortened to give other jobs in the queue more opportunity to run.
 
 > [!NOTE]
 > As stated in the [Index limits](#index-limits), indexers will also enforce the upper limit of 3000 elements across all complex collections per document starting with the latest GA API version that supports complex types (`2019-05-06`) onwards. This means that if you've created your indexer with a prior API version, you will not be subject to this limit. To preserve maximum compatibility, an indexer that was created with a prior API version and then updated with an API version `2019-05-06` or later, will still be **excluded** from the limits. Customers should be aware of the adverse impact of having very large complex collections (as stated previously) and we highly recommend creating any new indexers with the latest GA API version.
@@ -156,7 +156,7 @@ Indexers can access other Azure resources [over private endpoints](search-indexe
 | Maximum private endpoints | N/A | 10 or 30 | 100 | 400 | 400 | N/A | 20 | 20 |
 | Maximum distinct resource types<sup>2</sup> | N/A | 4 | 7 | 15 | 15 | N/A | 4 | 4 |
 
-<sup>1</sup> AI enrichment and image analysis are computationally intensive and consume disproportionate amounts of available processing power. For this reason, private connections are disabled on lower tiers to avoid an adverse impact on the performance and stability of the search service itself.
+<sup>1</sup> AI enrichment and image analysis are computationally intensive and consume disproportionate amounts of available processing power. For this reason, private connections are disabled on lower tiers to ensure the performance and stability of the search service itself.
 
 <sup>2</sup> The number of distinct resource types are computed as the number of unique `groupId` values used across all shared private link resources for a given search service, irrespective of the status of the resource.
 
@@ -179,7 +179,7 @@ Maximum number of [index aliases](search-how-to-alias.md) varies by tier. In all
 
 ## Data limits (AI enrichment)
 
-An [AI enrichment pipeline](cognitive-search-concept-intro.md) that makes calls to Azure AI Language resource for [entity recognition](cognitive-search-skill-entity-recognition-v3.md), [entity linking](cognitive-search-skill-entity-linking-v3.md), [key phrase extraction](cognitive-search-skill-keyphrases.md), [sentiment analysis](cognitive-search-skill-sentiment-v3.md), [language detection](cognitive-search-skill-language-detection.md), and [personal-information detection](cognitive-search-skill-pii-detection.md) is subject to data limits. The maximum size of a record should be 50,000 characters as measured by [`String.Length`](/dotnet/api/system.string.length). If you need to break up your data before sending it to the sentiment analyzer, use the [Text Split skill](cognitive-search-skill-textsplit.md).
+An [AI enrichment pipeline](cognitive-search-concept-intro.md) that makes calls to an Azure AI Language resource for [entity recognition](cognitive-search-skill-entity-recognition-v3.md), [entity linking](cognitive-search-skill-entity-linking-v3.md), [key phrase extraction](cognitive-search-skill-keyphrases.md), [sentiment analysis](cognitive-search-skill-sentiment-v3.md), [language detection](cognitive-search-skill-language-detection.md), and [personal-information detection](cognitive-search-skill-pii-detection.md) is subject to data limits. The maximum size of a record should be 50,000 characters as measured by [`String.Length`](/dotnet/api/system.string.length). If you need to break up your data before sending it to the sentiment analyzer, use the [Text Split skill](cognitive-search-skill-textsplit.md).
 
 ## Throttling limits
 
@@ -198,6 +198,7 @@ Static rate request limits for operations related to a service:
 + Service Statistics (GET /servicestats): 4 per second per search unit
 
 ## API request limits
+
 * Maximum of 16 MB per request <sup>1</sup>
 * Maximum 8 KB URL length
 * Maximum 1000 documents per batch of index uploads, merges, or deletes
@@ -213,10 +214,12 @@ Static rate request limits for operations related to a service:
 Limits on query size and composition exist because unbounded queries can destabilize your search service. Typically, such queries are created programmatically. If your application generates search queries programmatically, we recommend designing it in such a way that it doesn't generate queries of unbounded size.
 
 ## API response limits
+
 * Maximum 1000 documents returned per page of search results
 * Maximum 100 suggestions returned per Suggest API request
 
 ## API key limits
+
 API keys are used for service authentication. There are two types. Admin keys are specified in the request header and grant full read-write access to the service. Query keys are read-only, specified on the URL, and typically distributed to client applications.
 
 * Maximum of 2 admin keys per service
diff --git a/articles/search/vector-search-index-size.md b/articles/search/vector-search-index-size.md
@@ -114,7 +114,7 @@ The storage size of one vector is determined by its dimensionality. Multiply the
 
 For `Edm.Single`, the size of the data type is 4 bytes.
 
-### Memory Overhead from the Selected Algorithm  
+### Memory overhead from the selected algorithm  
   
 Every approximate nearest neighbor (ANN) algorithm generates extra data structures in memory to enable efficient searching. These structures consume extra space within memory.  
   
@@ -161,4 +161,4 @@ Disk storage overhead of vector data is roughly three times the size of vector i
 
 ### Storage vs. vector index size quotas
 
-The storage and vector index size quotas aren't separate quotas. Vector indexes contribute to the [storage quota for the search service](search-limits-quotas-capacity.md#storage-limits) as a whole. For example, if your storage quota is exhausted but there's remaining vector quota, you can't index any more documents, regardless if they're vector documents, until you scale up in partitions to increase storage quota or delete documents (either text or vector) to reduce storage usage. Similarly, if vector quota is exhausted but there's remaining storage quota, further indexing attempts fail until vector quota is freed, either by deleting some vector documents or by scaling up in partitions.
+The storage and vector index size quotas aren't separate quotas. Vector indexes contribute to the [storage quota for the search service](search-limits-quotas-capacity.md#service-limits) as a whole. For example, if your storage quota is exhausted but there's remaining vector quota, you can't index any more documents, regardless if they're vector documents, until you scale up in partitions to increase storage quota or delete documents (either text or vector) to reduce storage usage. Similarly, if vector quota is exhausted but there's remaining storage quota, further indexing attempts fail until vector quota is freed, either by deleting some vector documents or by scaling up in partitions.
diff --git a/includes/azure-search-limits-per-service.md b/includes/azure-search-limits-per-service.md
@@ -5,21 +5,22 @@
  author: HeidiSteen
  ms.service: cognitive-search
  ms.topic: include
- ms.date: 05/11/2020
+ ms.date: 01/02/2024
  ms.author: heidist
  ms.custom: include file
 ---
 
-A search service is constrained by disk space or by a hard limit on the maximum number of indexes or indexers, whichever comes first. The following table documents storage limits. For maximum object limits, see [Limits by resource](../articles/search/search-limits-quotas-capacity.md#index-limits).
+A search service is subject to a maximum storage limit (partition size multiplied by the number of partitions) or by a hard limit on the [maximum number of indexes](../articles/search/search-limits-quotas-capacity.md#index-limits) or [indexers](../articles/search/search-limits-quotas-capacity.md#indexer-limits), whichever comes first. 
 
-| Resource | Free | Basic<sup>1</sup> | S1 | S2 | S3 | S3&nbsp;HD | L1 | L2 |
+| Resource | Free <sup>1</sup> | Basic <sup>1</sup> | S1 | S2 | S3 | S3&nbsp;HD | L1 | L2 |
 | -------- | --- | --- | --- | --- | --- | --- | --- | --- |
-| Service level agreement (SLA)<sup>2</sup>  |No |Yes |Yes |Yes |Yes |Yes |Yes |Yes |
-| Storage per partition |50 MB |2 GB |25 GB |100 GB |200 GB |200 GB |1 TB |2 TB |
-| Partitions per service |N/A |1 |12 |12 |12 |3 |12 |12 |
-| Partition size |N/A |2 GB |25 GB |100 GB |200 GB |200 GB |1 TB |2 TB |
-| Replicas |N/A |3 |12 |12 |12 |12 |12 |12 |
+| Service level agreement (SLA) <sup>2</sup>  |No |Yes |Yes |Yes |Yes |Yes |Yes |Yes |
+| Storage (partition size) |50 MB <sup>3</sup> |2 GB |25 GB |100 GB |200 GB |200 GB |1 TB |2 TB |
+| Partitions | N/A |1 |12 |12 |12 |3 |12 |12 |
+| Replicas | N/A |3 |12 |12 |12 |12 |12 |12 |
 
-<sup>1</sup> Basic has one fixed partition. Additional search units can be used to add replicas for larger query volumes.
+<sup>1</sup> Basic has one fixed partition. You can specify up to 3 more search units to add replicas for larger query volumes and high availability.
 
-<sup>2</sup> Service level agreements are in effect for billable services on dedicated resources. Free services and preview features have no SLA. For billable services, SLAs take effect when you provision sufficient redundancy for your service. Two or more replicas are required for query (read) SLAs. Three or more replicas are required for query and indexing (read-write) SLAs. The number of partitions isn't an SLA consideration. 
+<sup>2</sup> Service level agreements apply to billable services having dedicated resources. Free services and preview features have no SLA. For billable services, SLAs take effect when you provision sufficient redundancy for your service. Two or more replicas are required for query (read) SLAs. Three or more replicas are required for query and indexing (read-write) SLAs. The number of partitions isn't an SLA consideration. See [Reliability in Azure AI Search](/azure/search/search-reliability#high-availability) to learn more about replicas and high availability.
+
+<sup>3</sup> Free services don't have a dedicated partition. The 50-MB storage limit refers to the maximum space allocated to a free search service on infrastructure shared with other customers.
diff --git a/includes/azure-search-limits-per-subscription.md b/includes/azure-search-limits-per-subscription.md
@@ -4,22 +4,22 @@
  author: HeidiSteen
  ms.service: cognitive-search
  ms.topic: include
- ms.date: 07/17/2023
+ ms.date: 01/02/2024
  ms.author: heidist
 ms.custom:
   - include file
   - ignite-2023
 ---
 
-You can create multiple *billable* search services (Basic and higher), limited only by the number of services allowed at each tier. For example, you could create up to 16 services at the Basic tier and another 16 services at the S1 tier within the same subscription. For more information about tiers, see [Choose an SKU or tier for Azure AI Search](../articles/search/search-sku-tier.md).
+You can create multiple *billable* search services (Basic and higher), up to the maximum number of services allowed at each tier. For example, you could create up to 16 services at the Basic tier and another 16 services at the S1 tier within the same subscription. For more information about tiers, see [Choose a tier (or SKU) for Azure AI Search](../articles/search/search-sku-tier.md).
 
 Maximum service limits can be raised upon request. If you need more services within the same subscription, [file a support request](../articles/search/search-create-service-portal.md#add-more-services-to-a-subscription).
 
-| Resource            | Free<sup>1</sup> | Basic | S1  | S2 | S3 | S3&nbsp;HD | L1 | L2 |
-| ------------------- | ---- | ----- | --- | -- | -- | ----- | -- | -- |
+| Resource            | Free <sup>1</sup> | Basic | S1  | S2 | S3 | S3&nbsp;HD | L1 | L2 |
+| ------------------- | ---- | ----- | --- |--|--| ----- |--|--|
 | Maximum services    |1     | 16    | 16  | 8  | 6  | 6     | 6  | 6  |
-| Maximum scale in search units (SU)<sup>2</sup> |N/A |3 SU |36 SU |36 SU |36 SU |36 SU |36 SU |36 SU |
+| Maximum search units (SU)<sup>2</sup> |N/A |3 SU |36 SU |36 SU |36 SU |36 SU |36 SU |36 SU |
 
-<sup>1</sup> You can have one free search service per Azure subscription. The free tier is based on infrastructure that's shared with other customers. Because the hardware isn't dedicated, scale-up isn't supported, and storage is limited to 50 MB.
+<sup>1</sup> You can have one free search service per Azure subscription. The free tier is based on infrastructure shared with other customers. Because the hardware isn't dedicated, scale-up isn't supported, and storage is limited to 50 MB.
 
-<sup>2</sup> Search units are billing units, allocated as either a *replica* or a *partition*. You need both resources for storage, indexing, and query operations. To learn more about SU computations, see [Scale resource levels for query and index workloads](../articles/search/search-capacity-planning.md). 
+<sup>2</sup> Search units (SU) are billing units, allocated as either a *replica* or a *partition*. You need both. To learn more about SU combinations, see [Estimate and manage capacity of a search service](../articles/search/search-capacity-planning.md). 
