Merge branch 'main' of https://github.com/MicrosoftDocs/azure-docs-pr into WI196311-export-to-SIEM

Author: ElazarK

articles/ai-services/openai/concepts/provisioned-throughput.md

@@ -3,7 +3,7 @@ title: Azure OpenAI Service provisioned throughput
 description: Learn about provisioned throughput and Azure OpenAI. 
 ms.service: azure-ai-openai
 ms.topic: conceptual 
-ms.date: 11/20/2023
+ms.date: 1/16/2024
 ms.custom: 
 manager: nitinme
 author: mrbullwinkle #ChrisHMSFT
@@ -14,44 +14,96 @@ keywords:
 
 # What is provisioned throughput?
 
-The provisioned throughput capability allows you to specify the amount of throughput you require for your application. The service then provisions the necessary compute and ensures it is ready for you. Throughput is defined in terms of provisioned throughput units (PTU) which is a normalized way of representing an amount of throughput for your deployment. Each model-versions pair requires different amounts of PTU to deploy and provide different amounts of throughput per PTU.
+The provisioned throughput capability allows you to specify the amount of throughput you require in a deployment. The service then allocates the necessary model processing capacity and ensures it's ready for you. Throughput is defined in terms of provisioned throughput units (PTU) which is a normalized way of representing the throughput for your deployment. Each model-version pair requires different amounts of PTU to deploy and provide different amounts of throughput per PTU.
 
 ## What does the provisioned deployment type provide?
 
-- **Predictable performance:** stable max latency and throughput for uniform workloads.
+- **Predictable performance:** stable max latency and throughput for uniform workloads. 
 - **Reserved processing capacity:** A deployment configures the amount of throughput. Once deployed, the throughput is available whether used or not.
-- **Cost savings:** High throughput workloads may provide cost savings vs token-based consumption.
+- **Cost savings:** High throughput workloads might provide cost savings vs token-based consumption.
 
-An Azure OpenAI Deployment is a unit of management for a specific OpenAI Model. A deployment provides customer access to a model for inference and integrates additional features like Content Moderation ([See content moderation documentation](content-filter.md)).
+An Azure OpenAI Deployment is a unit of management for a specific OpenAI Model. A deployment provides customer access to a model for inference and integrates more features like Content Moderation ([See content moderation documentation](content-filter.md)).
 
 > [!NOTE]
-> Provisioned throughput units (PTU) are different from standard quota in Azure OpenAI and are not available by default. To learn more about this offering contact your Microsoft Account Team.
+> Provisioned throughput unit (PTU) quota is different from standard quota in Azure OpenAI and is not available by default. To learn more about this offering contact your Microsoft Account Team.
 
 ## What do you get?
 
-|Topic | Provisioned|
+| Topic | Provisioned|
 |---|---|
-| What is it? | Provides guaranteed throughput at smaller increments than the existing provisioned offer. Deployments will have a consistent max latency for a given model-version |
+| What is it? | Provides guaranteed throughput at smaller increments than the existing provisioned offer. Deployments have a consistent max latency for a given model-version. |
 | Who is it for? | Customers who want guaranteed throughput with minimal latency variance. |
-| Quota | Provisioned-managed throughput Units |
-| Latency | Max latency constrained |
-| Utilization | Provisioned-managed Utilization measure provided in Azure Monitor |
-| Estimating size | Provided calculator in the studio & load test script |
+| Quota | Provisioned-managed throughput Units for a given model. |
+| Latency | Max latency constrained from the model. Overall latency is a factor of call shape.  |
+| Utilization | Provisioned-managed Utilization measure provided in Azure Monitor. |
+| Estimating size | Provided calculator in the studio & benchmarking script. |
 
 ## Key concepts
 
 ### Provisioned throughput units
 
-Provisioned throughput Units (PTU) are units of model processing capacity that customers you can reserve and deploy for processing prompts and generating completions. The minimum PTU deployment, increments, and processing capacity associated with each unit varies by model type & version.
+Provisioned throughput units (PTU) are units of model processing capacity that customers you can reserve and deploy for processing prompts and generating completions. The minimum PTU deployment, increments, and processing capacity associated with each unit varies by model type & version.
 
 ### Deployment types
 
-We introduced a new deployment type called **ProvisionedManaged** which provides smaller increments of PTU per deployment. Both types have their own quota, and you will only see the options you have been enabled for.
+When deploying a model in Azure OpenAI, you need to set the `sku-name` to be Provisioned-Managed. The `sku-capacity` specifies the number of PTUs assigned to the deployment. 
+
+```azurecli
+az cognitiveservices account deployment create \
+--name <myResourceName> \
+--resource-group  <myResourceGroupName> \
+--deployment-name MyDeployment \
+--model-name GPT-4 \
+--model-version 0613  \
+--model-format OpenAI \
+--sku-capacity 100 \
+--sku-name ProvisionedManaged 
+```
 
 ### Quota
 
-Provisioned throughput quota represents a specific amount of total throughput you can deploy. Quota in the Azure OpenAI Service is managed at the subscription level meaning that it can be consumed by different resources within that subscription.
+Provisioned throughput quota represents a specific amount of total throughput you can deploy. Quota in the Azure OpenAI Service is managed at the subscription level. All Azure OpenAI resources within the subscription share this quota. 
+
+Quota is specific to a (deployment type, model, region) triplet and isn't interchangeable. Meaning you can't use quota for GPT-4 to deploy GPT-35-turbo. You can raise a support request to move quota across deployment types, models, or regions but the swap isn't guaranteed.
+
+While we make every attempt to ensure that quota is deployable, quota doesn't represent a guarantee that the underlying capacity is available. The service assigns capacity during the deployment operation and if capacity is unavailable the deployment fails with an out of capacity error.
+
+
+### How utilization enforcement works
+Provisioned deployments provide you with an allocated amount of model processing capacity to run a given model. The `Provisioned-Managed Utilization` metric in Azure Monitor measures a given deployments utilization on 1-minute increments. Provisioned-Managed deployments are optimized to ensure that accepted calls are processed with a consistent model processing time (actual end-to-end latency is dependent on a call's characteristics). When the workload exceeds the allocated PTU capacity, the service returns a 429 HTTP status code until the utilization drops down below 100%. 
+
+
+#### What should  I do when I receive a 429 response?
+The 429 response isn't an error, but instead part of the design for telling users that a given deployment is fully utilized at a point in time. By providing a fast-fail response, you have control over how to handle these situations in a way that best fits your application requirements.
+
+The  `retry-after-ms` and `retry-after` headers in the response tell you the time to wait before the next call will be accepted. How you choose to handle this response depends on your application requirements. Here are some considerations:
+-	You can consider redirecting the traffic to other models, deployments or experiences. This option is the lowest-latency solution because the action can be taken as soon as you receive the 429 signal.
+-	If you're okay with longer per-call latencies, implement client-side retry logic. This option gives you the highest amount of throughput per PTU. The Azure OpenAI client libraries include built-in capabilities for handling retries.
+
+#### How does the service decide when to send a 429?
+We use a variation of the leaky bucket algorithm to maintain utilization below 100% while allowing some burstiness in the traffic. The high-level logic is as follows:
+1.	Each customer has a set amount of capacity they can utilize on a deployment
+2.	When a request is made:
+
+    a.	When the current utilization is above 100%, the service returns a 429 code with the `retry-after-ms` header set to the time until utilization is below 100%
+     
+    b.	Otherwise, the service estimates the incremental change to utilization required to serve the request by combining prompt tokens and the specified max_tokens in the call.
+
+3.	When a request finishes, we now know the actual compute cost for the call. To ensure an accurate accounting, we correct the utilization using the following logic:
+
+    a.	If the actual > estimated, then the difference is added to the deployment's utilization
+    b.	If the actual < estimated, then the difference is subtracted. 
+
+4.	The overall utilization is decremented down at a continuous rate based on the number of PTUs deployed. 
+
+Since calls are accepted until utilization reaches 100%, you're allowed to burst over 100% utilization when first increasing traffic. For sizeable calls and small sized deployments, you might then be over 100% utilization for up to several minutes.
+
+
+:::image type="content" source="../media/provisioned/utilization.jpg" alt-text="Diagram showing how subsequent calls are added to the utilization." lightbox="../media/provisioned/utilization.jpg":::
+
+
 
-Quota is specific to a (deployment type, model, region) triplet and isn't interchangeable. Meaning you can't use quota for GPT-4 to deploy GPT-35-turbo. Customers can raise a support request to move the quota across deployment types, models, or regions but we can't guarantee that it will be possible.
+## Next steps
 
-While we make every attempt to ensure that quota is always deployable, quota does not represent a guarantee that the underlying capacity is available for the customer to use. The service assigns capacity to the customer at deployment time and if capacity is unavailable the deployment will fail with an out of capacity error.
+- [Learn about the onboarding steps for provisioned deployments](../how-to/provisioned-throughput-onboarding.md)
+- [Provisioned Throughput Units (PTU) getting started guide](../how-to//provisioned-get-started.md) 

articles/ai-services/openai/how-to/integrate-synapseml.md

@@ -35,9 +35,9 @@ This tutorial shows how to apply large language models at a distributed scale by
    - To install SynapseML for your Apache Spark cluster, see [Install SynapseML](#install-synapseml).
 
 > [!NOTE]
-> This article is designed to work with the [Azure OpenAI Service legacy models](/azure/ai-services/openai/concepts/legacy-models) like `Text-Davinci-003`, which support prompt-based completions. Newer models like the current `GPT-3.5 Turbo` and `GPT-4` model series are designed to work with the new chat completion API that expects a specially formatted array of messages as input. 
+> The `OpenAICompletion()` transformer is designed to work with the [Azure OpenAI Service legacy models](/azure/ai-services/openai/concepts/legacy-models) like `Text-Davinci-003`, which supports prompt-based completions. Newer models like the current `GPT-3.5 Turbo` and `GPT-4` model series are designed to work with the new chat completion API that expects a specially formatted array of messages as input. If you working with embeddings or chat completion models, please check the [Chat Completion](#chat-completion) and [Generating Text Embeddings](#generating-text-embeddings) sections bellow.
 > 
-> The Azure OpenAI SynapseML integration supports the latest models via the [OpenAIChatCompletion()](https://github.com/microsoft/SynapseML/blob/0836e40efd9c48424e91aa10c8aa3fbf0de39f31/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/openai/OpenAIChatCompletion.scala#L24) transformer, which isn't demonstrated in this article. After the [release of the GPT-3.5 Turbo Instruct model](https://techcommunity.microsoft.com/t5/azure-ai-services-blog/announcing-updates-to-azure-openai-service-models/ba-p/3866757), the newer model will be the preferred model to use with this article.
+> The Azure OpenAI SynapseML integration supports the latest models via the [OpenAIChatCompletion()](https://github.com/microsoft/SynapseML/blob/0836e40efd9c48424e91aa10c8aa3fbf0de39f31/cognitive/src/main/scala/com/microsoft/azure/synapse/ml/cognitive/openai/OpenAIChatCompletion.scala#L24) transformer.
 
 We recommend that you [create an Azure Synapse workspace](../../../synapse-analytics/get-started-create-workspace.md). However, you can also use Azure Databricks, Azure HDInsight, Spark on Kubernetes, or the Python environment with the `pyspark` package.
 
@@ -187,15 +187,87 @@ The following image shows example output with completions in Azure Synapse Analy
 
 Here are some other use cases for working with Azure OpenAI Service and large datasets.
 
-### Improve throughput with request batching
+### Generating Text Embeddings
+
+In addition to completing text, we can also embed text for use in downstream algorithms or vector retrieval architectures. Creating embeddings allows you to search and retrieve documents from large collections and can be used when prompt engineering isn't sufficient for the task. For more information on using [OpenAIEmbedding](https://mmlspark.blob.core.windows.net/docs/0.11.1/pyspark/_modules/synapse/ml/cognitive/openai/OpenAIEmbedding.html), see our [embedding guide](https://microsoft.github.io/SynapseML/docs/Explore%20Algorithms/OpenAI/Quickstart%20-%20OpenAI%20Embedding/).
+
+from synapse.ml.services.openai import OpenAIEmbedding
+
+```python
+embedding = (
+    OpenAIEmbedding()
+    .setSubscriptionKey(key)
+    .setDeploymentName(deployment_name_embeddings)
+    .setCustomServiceName(service_name)
+    .setTextCol("prompt")
+    .setErrorCol("error")
+    .setOutputCol("embeddings")
+)
+
+display(embedding.transform(df))
+```
+
+### Chat Completion
+Models such as ChatGPT and GPT-4 are capable of understanding chats instead of single prompts. The [OpenAIChatCompletion](https://mmlspark.blob.core.windows.net/docs/0.11.1/pyspark/_modules/synapse/ml/cognitive/openai/OpenAIChatCompletion.html) transformer exposes this functionality at scale.
+
+```python
+from synapse.ml.services.openai import OpenAIChatCompletion
+from pyspark.sql import Row
+from pyspark.sql.types import *
+
+
+def make_message(role, content):
+    return Row(role=role, content=content, name=role)
+
+
+chat_df = spark.createDataFrame(
+    [
+        (
+            [
+                make_message(
+                    "system", "You are an AI chatbot with red as your favorite color"
+                ),
+                make_message("user", "Whats your favorite color"),
+            ],
+        ),
+        (
+            [
+                make_message("system", "You are very excited"),
+                make_message("user", "How are you today"),
+            ],
+        ),
+    ]
+).toDF("messages")
+
+chat_completion = (
+    OpenAIChatCompletion()
+    .setSubscriptionKey(key)
+    .setDeploymentName(deployment_name)
+    .setCustomServiceName(service_name)
+    .setMessagesCol("messages")
+    .setErrorCol("error")
+    .setOutputCol("chat_completions")
+)
+
+display(
+    chat_completion.transform(chat_df).select(
+        "messages", "chat_completions.choices.message.content"
+    )
+)
+```
+
+### Improve throughput with request batching from OpenAICompletion
 
 You can use Azure OpenAI Service with large datasets to improve throughput with request batching. In the previous example, you make several requests to the service, one for each prompt. To complete multiple prompts in a single request, you can use batch mode.
 
-In the `OpenAICompletion` object definition, you specify the `"batchPrompt"` value to configure the dataframe to use a **batchPrompt** column. Create the dataframe with a list of prompts for each row.
+In the [OpenAItCompletion](https://mmlspark.blob.core.windows.net/docs/0.11.1/pyspark/_modules/synapse/ml/cognitive/openai/OpenAICompletion.html) object definition, you specify the `"batchPrompt"` value to configure the dataframe to use a **batchPrompt** column. Create the dataframe with a list of prompts for each row.
 
 > [!NOTE]
 > There's currently a limit of 20 prompts in a single request and a limit of 2048 tokens, or approximately 1500 words.
 
+> [!NOTE]
+> Currently, request batching is not supported by the `OpenAIChatCompletion()` transformer.
+
 ```python
 batch_df = spark.createDataFrame(
     [
@@ -227,9 +299,6 @@ completed_batch_df = batch_completion.transform(batch_df).cache()
 display(completed_batch_df)
 ```
 
-> [!NOTE]
-> There's currently a limit of 20 prompts in a single request and a limit of 2048 tokens, or approximately 1500 words.
-
 ### Use an automatic mini-batcher
 
 You can use Azure OpenAI Service with large datasets to transpose the data format. If your data is in column format, you can transpose it to row format by using the SynapseML `FixedMiniBatcherTransformer` object.