Skip to content

Commit 85e9273

Browse files
aahillaahill
committed
moving content
1 parent b08a26b commit 85e9273

File tree

2 files changed

+59
-68
lines changed

2 files changed

+59
-68
lines changed

‎articles/ai-services/openai/concepts/provisioned-throughput.md

Lines changed: 2 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ recommendations: false
1616
> The Azure OpenAI Provisioned offering received significant updates on August 12, 2024, including aligning the purchase model with Azure standards and moving to model-independent quota. It is highly recommended that customers onboarded before this date read the Azure [OpenAI provisioned August update](./provisioned-migration.md) to learn more about these changes.
1717
1818

19-
The provisioned throughput offring is a model deployment type that allows you to specify the amount of throughput you require in a model deployment. The Azure OpenAI service then allocates the necessary model processing capacity and ensures it's ready for you. Provisioned throughput provides:
19+
The provisioned throughput offering is a model deployment type that allows you to specify the amount of throughput you require in a model deployment. The Azure OpenAI service then allocates the necessary model processing capacity and ensures it's ready for you. Provisioned throughput provides:
2020

2121
- **Predictable performance:** stable max latency and throughput for uniform workloads.
2222
- **Allocated processing capacity:** A deployment configures the amount of throughput. Once deployed, the throughput is available whether used or not.
@@ -71,63 +71,6 @@ az cognitiveservices account deployment create \
7171
--sku-name GlobalProvisionedManaged
7272
```
7373

74-
### Quota
75-
76-
#### Provisioned throughput units
77-
78-
Provisioned throughput units (PTUs) are generic units of model processing capacity that you can use to size provisioned deployments to achieve the required throughput for processing prompts and generating completions. Provisioned throughput units are granted to a subscription as quota. Each quota is specific to a region and defines the maximum number of PTUs that can be assigned to deployments in that subscription and region.
79-
80-
#### Model independent quota
81-
82-
Unlike the Tokens Per Minute (TPM) quota used by other Azure OpenAI offerings, PTUs are model-independent. The PTUs might be used to deploy any supported model/version in the region.
83-
84-
:::image type="content" source="../media/provisioned/model-independent-quota.png" alt-text="Diagram of model independent quota with one pool of PTUs available to multiple Azure OpenAI models." lightbox="../media/provisioned/model-independent-quota.png":::
85-
86-
Quota for provisioned deployments shows up in Azure AI Foundry as the following:
87-
88-
> * Provisioned throughput is available as the following deployment types: [global provisioned](../how-to/deployment-types.md#global-provisioned), [data zone provisioned](../how-to/deployment-types.md#data-zone-provisioned) and [standard provisioned](../how-to/deployment-types.md#provisioned).
89-
90-
|deployment type |Quota name |
91-
|---------|---------|
92-
|[provisioned](../how-to/deployment-types.md#provisioned) | Provisioned Managed Throughput Unit |
93-
|[global provisioned](../how-to/deployment-types.md#global-provisioned) | Global Provisioned Managed Throughput Unit |
94-
|[data zone provisioned](../how-to/deployment-types.md#data-zone-provisioned) | Data Zone Provisioned Managed Throughput Unit |
95-
96-
:::image type="content" source="../media/provisioned/ptu-quota-page.png" alt-text="Screenshot of quota UI for Azure OpenAI provisioned." lightbox="../media/provisioned/ptu-quota-page.png":::
97-
98-
99-
## How much throughput per PTU you get for each model
100-
The amount of throughput (measured in tokens per minute or TPM) a deployment gets per PTU is a function of the input and output tokens in a given minute.
101-
102-
Generating output tokens requires more processing than input tokens. For the models specified in the table below, 1 output token counts as 3 input tokens towards your TPM-per-PTU limit. The service dynamically balances the input & output costs, so users do not have to set specific input and output limits. This approach means your deployment is resilient to fluctuations in the workload.
103-
104-
To help with simplifying the sizing effort, the following table outlines the TPM-per-PTU for the specified models. To understand the impact of output tokens on the TPM-per-PTU limit, use the 3 input token to 1 output token ratio.
105-
106-
For a detailed understanding of how different ratios of input and output tokens impact the throughput your workload needs, see the [Azure OpenAI capacity calculator](https://oai.azure.com/portal/calculator). The table also shows Service Level Agreement (SLA) Latency Target Values per model. For more information about the SLA for Azure OpenAI Service, see the [Service Level Agreements (SLA) for Online Services page](https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services?lang=1)
107-
108-
|Topic| **gpt-4o** | **gpt-4o-mini** | **o1**|
109-
| --- | --- | --- | --- |
110-
|Global & data zone provisioned minimum deployment|15|15|15|
111-
|Global & data zone provisioned scale increment|5|5|5|
112-
|Regional provisioned minimum deployment|50|25|50|
113-
|Regional provisioned scale increment|50|25|50|
114-
|Input TPM per PTU |2,500|37,000|230|
115-
|Latency Target Value |25 Tokens Per Second|33 Tokens Per Second|25 Tokens Per Second|
116-
117-
For a full list see the [Azure OpenAI Service in Azure AI Foundry portal calculator](https://oai.azure.com/portal/calculator).
118-
119-
120-
> [!NOTE]
121-
> Global provisioned and data zone provisioned deployments are only supported for gpt-4o and gpt-4o-mini models at this time. For more information on model availability, review the [models documentation](./models.md).
122-
123-
#### Obtaining PTU quota
124-
125-
PTU quota is available by default in many regions. If more quota is required, customers can request quota via the Request Quota link. This link can be found to the right of the designated provisioned deployment type quota tabs in Azure AI Foundry The form allows the customer to request an increase in the specified PTU quota for a given region. The customer receives an email at the included address once the request is approved, typically within two business days.
126-
127-
#### Per-Model PTU minimums
128-
129-
The minimum PTU deployment, increments, and processing capacity associated with each unit varies by model type & version.
130-
13174
### Capacity transparency
13275

13376
Azure OpenAI is a highly sought-after service where customer demand might exceed service GPU capacity. Microsoft strives to provide capacity for all in-demand regions and models, but selling out a region is always a possibility. This constraint can limit some customers' ability to create a deployment of their desired model, version, or number of PTUs in a desired region - even if they have quota available in that region. Generally speaking:
@@ -181,7 +124,7 @@ For provisioned deployments, we use a variation of the leaky bucket algorithm to
181124

182125
a. When the current utilization is above 100%, the service returns a 429 code with the `retry-after-ms` header set to the time until utilization is below 100%
183126

184-
b. Otherwise, the service estimates the incremental change to utilization required to serve the request by combining the prompt tokens, less any cached tokens, and the specified `max_tokens` in the call. A customer can receive up to a 100% discount on their prompt tokens depending on the size of their cached tokens. If the `max_tokens` parameter is not specified, the service estimates a value. This estimation can lead to lower concurrency than expected when the number of actual generated tokens is small. For highest concurrency, ensure that the `max_tokens` value is as close as possible to the true generation size.
127+
b. Otherwise, the service estimates the incremental change to utilization required to serve the request by combining the prompt tokens, less any cached tokens, and the specified `max_tokens` in the call. A customer can receive up to a 100% discount on their prompt tokens depending on the size of their cached tokens. If the `max_tokens` parameter is not specified, the service estimates a value. This estimation can lead to lower concurrency than expected when the number of actual generated tokens is small. For highest concurrency, ensure that the `max_tokens` value is as close as possible to the true generation size.
185128

186129
1. When a request finishes, we now know the actual compute cost for the call. To ensure an accurate accounting, we correct the utilization using the following logic:
187130

‎articles/ai-services/openai/how-to/provisioned-throughput-onboarding.md

Lines changed: 57 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -12,26 +12,24 @@ recommendations: false
1212

1313
# Understanding costs associated with provisioned throughput units (PTU)
1414

15-
Use this article to learn about calculating and understanding costs associated with PTU. For an overview of PTU, see [What is provisioned throughput?](../concepts/provisioned-throughput.md). When you're ready to sign up for PTU, see the [getting started guide](./provisioned-get-started.md).
16-
17-
<!--
18-
## When to use provisioned throughput units (PTU)
19-
20-
You should consider switching from standard deployments to provisioned deployments when you have well-defined, predictable throughput and latency requirements. Typically, this occurs when the application is ready for production or has already been deployed in production and there's an understanding of the expected traffic. This allows users to accurately forecast the required capacity and avoid unexpected billing.
21-
-->
15+
Use this article to learn about calculating and understanding costs associated with PTU. For an overview of the provisioned throughput offering, see [What is provisioned throughput?](../concepts/provisioned-throughput.md). When you're ready to sign up for the provisioned throughput offering, see the [getting started guide](./provisioned-get-started.md).
2216

2317
> [!NOTE]
2418
> In function calling and agent use cases, token usage can be variable. You should understand your expected Tokens Per Minute (TPM) usage in detail prior to migrating workloads to PTU.
2519
20+
#### Provisioned throughput units
21+
22+
Provisioned throughput units (PTUs) are generic units of model processing capacity that you can use to size provisioned deployments to achieve the required throughput for processing prompts and generating completions. Provisioned throughput units are granted to a subscription as quota. Each quota is specific to a region and defines the maximum number of PTUs that can be assigned to deployments in that subscription and region.
2623

2724
## Understanding the provisioned throughput purchase model
2825

29-
Azure OpenAI [Provisioned](../how-to/deployment-types.md#provisioned), [Data Zone Provisioned](../how-to/deployment-types.md#data-zone-provisioned), and [Global Provisioned](../how-to/deployment-types.md#global-provisioned) are purchased on-demand at an hourly basis based on the number of deployed PTUs, with substantial term discount available via the purchase of Azure Reservations.
26+
Azure OpenAI [Provisioned](../how-to/deployment-types.md#provisioned), [Data Zone Provisioned](../how-to/deployment-types.md#data-zone-provisioned), and [Global Provisioned](../how-to/deployment-types.md#global-provisioned) are purchased on-demand at an hourly basis based on the number of deployed PTUs, with substantial term discount available via the purchase of [Azure Reservations](#azure-reservations-for-azure-openai-provisioned-deployments).
3027

3128
The hourly model is useful for short-term deployment needs, such as validating new models or acquiring capacity for a hackathon.  However, the discounts provided by the Azure Reservation for Azure OpenAI Provisioned, Data Zone Provisioned, and Global Provisioned are considerable and most customers with consistent long-term usage will find a reserved model to be a better value proposition.
3229

3330
> [!NOTE]
3431
> Azure OpenAI Provisioned customers onboarded prior to the August self-service update use a purchase model called the Commitment model. These customers can continue to use this older purchase model alongside the Hourly/reservation purchase model. The Commitment model is not available for new customers or new models introduced after August 2024. For details on the Commitment purchase model and options for coexistence and migration, please see the [Azure OpenAI Provisioned August Update](../concepts/provisioned-migration.md).
32+
3533
## Hourly usage
3634

3735
Provisioned, Data Zone Provisioned, and Global Provisioned deployments are charged an hourly rate ($/PTU/hr) on the number of PTUs that have been deployed.  For example, a 300 PTU deployment will be charged the hourly rate times 300.  All Azure OpenAI pricing is available in the Azure Pricing Calculator.
@@ -52,6 +50,27 @@ Customers that require long-term usage of provisioned, data zoned provisioned, a
5250
> * Having unused provisioned quota (PTUs) does not guarantee that capacity will be available to support an increase in the size of the deployment when required. Quota limits the maximum number of PTUs that can be deployed, but it is not a capacity guarantee. Provisioned capacity for each region and model dynamically changes throughout the day and might not be available when required. As a result, it is recommended to maintain a permanent deployment to cover your traffic needs (paid for via a reservation).
5351
> Charges for deployments on a deleted resource will continue until the resource is purged. To prevent this, delete a resource’s deployment before deleting the resource. For more information, see [Recover or purge deleted Azure AI services resources](../../recover-purge-resources.md).
5452
53+
## How much throughput per PTU you get for each model
54+
The amount of throughput (measured in tokens per minute or TPM) a deployment gets per PTU is a function of the input and output tokens in a given minute.
55+
56+
Generating output tokens requires more processing than input tokens. For the models specified in the table below, 1 output token counts as 3 input tokens towards your TPM-per-PTU limit. The service dynamically balances the input & output costs, so users do not have to set specific input and output limits. This approach means your deployment is resilient to fluctuations in the workload.
57+
58+
To help with simplifying the sizing effort, the following table outlines the TPM-per-PTU for the specified models. To understand the impact of output tokens on the TPM-per-PTU limit, use the 3 input token to 1 output token ratio.
59+
60+
For a detailed understanding of how different ratios of input and output tokens impact the throughput your workload needs, see the [Azure OpenAI capacity calculator](https://oai.azure.com/portal/calculator). The table also shows Service Level Agreement (SLA) Latency Target Values per model. For more information about the SLA for Azure OpenAI Service, see the [Service Level Agreements (SLA) for Online Services page](https://www.microsoft.com/licensing/docs/view/Service-Level-Agreements-SLA-for-Online-Services?lang=1)
61+
62+
63+
|Topic| **gpt-4o** | **gpt-4o-mini** | **o1**|
64+
| --- | --- | --- | --- |
65+
|Global & data zone provisioned minimum deployment|15|15|15|
66+
|Global & data zone provisioned scale increment|5|5|5|
67+
|Regional provisioned minimum deployment|50|25|50|
68+
|Regional provisioned scale increment|50|25|50|
69+
|Input TPM per PTU |2,500|37,000|230|
70+
|Latency Target Value |25 Tokens Per Second|33 Tokens Per Second|25 Tokens Per Second|
71+
72+
For a full list see the [Azure OpenAI Service in Azure AI Foundry portal calculator](https://oai.azure.com/portal/calculator).
73+
5574
## Determining the number of PTUs needed for a workload
5675

5776
Determining the right amount of provisioned throughput, or PTUs, you require for your workload is an essential step to optimizing performance and cost.
@@ -62,7 +81,36 @@ A few high-level considerations:
6281
- Generations require more capacity than prompts
6382
- For GPT-4o and later models, the TPM per PTU is set for input and output tokens separately. For older models, larger calls are progressively more expensive to compute. For example, 100 calls of with a 1000 token prompt size requires less capacity than one call with 100,000 tokens in the prompt. This tiering means that the distribution of these call shapes is important in overall throughput. Traffic patterns with a wide distribution that includes some large calls might experience lower throughput per PTU than a narrower distribution with the same average prompt & completion token sizes.
6483

65-
### Estimate provisioned throughput units and cost
84+
85+
### Model independent quota
86+
87+
Unlike the Tokens Per Minute (TPM) quota used by other Azure OpenAI offerings, PTUs are model-independent. The PTUs might be used to deploy any supported model/version in the region.
88+
89+
:::image type="content" source="../media/provisioned/model-independent-quota.png" alt-text="Diagram of model independent quota with one pool of PTUs available to multiple Azure OpenAI models." lightbox="../media/provisioned/model-independent-quota.png":::
90+
91+
Quota for provisioned deployments shows up in Azure AI Foundry as the following deployment types: [global provisioned](../how-to/deployment-types.md#global-provisioned), [data zone provisioned](../how-to/deployment-types.md#data-zone-provisioned) and [standard provisioned](../how-to/deployment-types.md#provisioned).
92+
93+
|deployment type |Quota name |
94+
|---------|---------|
95+
|[provisioned](../how-to/deployment-types.md#provisioned) | Provisioned Managed Throughput Unit |
96+
|[global provisioned](../how-to/deployment-types.md#global-provisioned) | Global Provisioned Managed Throughput Unit |
97+
|[data zone provisioned](../how-to/deployment-types.md#data-zone-provisioned) | Data Zone Provisioned Managed Throughput Unit |
98+
99+
:::image type="content" source="../media/provisioned/ptu-quota-page.png" alt-text="Screenshot of quota UI for Azure OpenAI provisioned." lightbox="../media/provisioned/ptu-quota-page.png":::
100+
101+
102+
> [!NOTE]
103+
> Global provisioned and data zone provisioned deployments are only supported for gpt-4o and gpt-4o-mini models at this time. For more information on model availability, review the [models documentation](./models.md).
104+
105+
### Obtaining PTU quota
106+
107+
PTU quota is available by default in many regions. If more quota is required, customers can request quota via the Request Quota link. This link can be found to the right of the designated provisioned deployment type quota tabs in Azure AI Foundry The form allows the customer to request an increase in the specified PTU quota for a given region. The customer receives an email at the included address once the request is approved, typically within two business days.
108+
109+
### Per-Model PTU minimums
110+
111+
The minimum PTU deployment, increments, and processing capacity associated with each unit varies by model type & version.
112+
113+
## Estimate provisioned throughput units and cost
66114

67115
To get a quick estimate for your workload using input and output TPM, leverage the built-in capacity planner in the deployment details section of the deployment dialogue screen. The built-in capacity planner is part of the deployment workflow to help streamline the sizing and allocation of quota to a PTU deployment for a given workload. For more information on how to identify and estimate TPM data, review the recommendations in our [performance and latency documentation](./latency.md).
68116

0 commit comments

Comments
 (0)