Merge pull request #1084 from MicrosoftDocs/main

10/28/2024 PM Publish

Author: Taojunshen

articles/ai-services/translator/text-sdk-overview.md

@@ -236,8 +236,8 @@ const {TextTranslationClient } = require("@azure-rest/ai-translation-text").defa
 ### [Python](#tab/python)
 
 ```python
-from azure.core.credentials import TextTranslationClient
-from azure-ai-translation-text import TextTranslationClient
+from azure.core.credentials import AzureKeyCredential
+from azure.ai.translation.text import TextTranslationClient
 ```
 
 ---

articles/ai-studio/ai-services/concepts/deployment-types.md

@@ -0,0 +1,52 @@
+---
+title: Understanding deployment types in Azure AI model inference
+titleSuffix: Azure AI services
+description: Learn how to use deployment types in Azure AI model deployments
+author: sdgilley
+manager: scottpolly
+ms.service: azure-ai-studio
+ms.topic: conceptual
+ms.date: 10/24/2024
+ms.author: fasantia
+ms.reviewer: fasantia
+ms.custom: github-universe-2024
+---
+
+# Deployment types in Azure AI model inference
+
+Azure AI model inference in Azure AI services provides customers with choices on the hosting structure that fits their business and usage patterns. The service offers two main types of deployment: **standard** and **provisioned**. Standard is offered with a global deployment option, routing traffic globally to provide higher throughput. Provisioned is also offered with a global deployment option, allowing customers to purchase and deploy provisioned throughput units across Azure global infrastructure.
+
+All deployments can perform the exact same inference operations, however the billing, scale, and performance are substantially different. As part of your solution design, you  need to make two key decisions:
+
+- **Data residency needs**: global vs. regional resources  
+- **Call volume**: standard vs. provisioned
+
+Deployment types support varies by model and model provider. 
+
+## Global versus regional deployment types
+
+For standard and provisioned deployments, you have an option of two types of configurations within your resource – **global** or **regional**. Global standard is the recommended starting point. 
+
+Global deployments use Azure's global infrastructure, dynamically route customer traffic to the data center with best availability for the customer’s inference requests. This means you get the highest initial throughput limits and best model availability with Global while still providing our uptime SLA and low latency. For high volume workloads above the specified usage tiers on standard and global standard, you may experience increased latency variation. For customers that require the lower latency variance at large workload usage, we recommend purchasing provisioned throughput.
+
+Our global deployments are the first location for all new models and features. Customers with very large throughput requirements should consider our provisioned deployment offering.
+
+## Standard
+
+Standard deployments provide a pay-per-call billing model on the chosen model. Provides the fastest way to get started as you only pay for what you consume. Models available in each region as well as throughput may be limited.  
+
+Standard deployments are optimized for low to medium volume workloads with high burstiness. Customers with high consistent volume may experience greater latency variability.
+
+Only Azure OpenAI models support this deployment type.
+
+## Global standard
+
+Global deployments are available in the same Azure AI services resources as nonglobal deployment types but allow you to use Azure's global infrastructure to dynamically route traffic to the data center with best availability for each request.  Global standard provides the highest default quota and eliminates the need to load balance across multiple resources.  
+
+Customers with high consistent volume may experience greater latency variability. The threshold is set per model.  For applications that require the lower latency variance at large workload usage, we recommend purchasing provisioned throughput if available.
+
+## Global provisioned
+
+Global deployments are available in the same Azure AI services resources as nonglobal deployment types but allow you to leverage Azure's global infrastructure to dynamically route traffic to the data center with best availability for each request. Global provisioned deployments provide reserved model processing capacity for high and predictable throughput using Azure global infrastructure.
+
+Only Azure OpenAI models support this deployment type.

articles/ai-studio/ai-services/concepts/endpoints.md

@@ -0,0 +1,99 @@
+---
+title: Use the Azure AI model inference endpoint
+titleSuffix: Azure AI studio
+description: Learn about to use the Azure AI model inference endpoint and how to configure it.
+ms.service: azure-ai-studio
+ms.topic: conceptual
+author: sdgilley
+manager: scottpolly
+ms.date: 10/24/2024
+ms.author: sgilley
+ms.reviewer: fasantia
+ms.custom: github-universe-2024
+---
+
+# Use the Azure AI model inference endpoint
+
+Azure AI inference service in Azure AI services allows customers to consume the most powerful models from flagship model providers using a single endpoint and credentials. This means that you can switch between models and consume them from your application without changing a single line of code.
+
+The article explains how models are organized inside of the service and how to use the inference endpoint to invoke them.
+
+## Deployments
+
+Azure AI model inference service makes models available using the **deployment** concept. **Deployments** are a way to give a model a name under certain configurations. Then, you can invoke such model configuration by indicating its name on your requests.
+
+Deployments capture:
+
+> [!div class="checklist"]
+> * A model name
+> * A model version
+> * A provisioning/capacity type<sup>1</sup>
+> * A content filtering configuration<sup>1</sup>
+> * A rate limiting configuration<sup>1</sup>
+
+<sup>1</sup> Configurations may vary depending on the model you have selected.
+
+An Azure AI services resource can have as many model deployments as needed and they don't incur in cost unless inference is performed for those models. Deployments are Azure resources and hence they are subject to Azure policies.
+
+To learn more about how to create deployments see [Add and configure model deployments](../how-to/create-model-deployments.md).
+
+## Azure AI inference endpoint
+
+The Azure AI inference endpoint allows customers to use a single endpoint with the same authentication and schema to generate inference for the deployed models in the resource. This endpoint follows the [Azure AI model inference API](../../reference/reference-model-inference-api.md) which is supported by all the models in Azure AI model inference service.
+
+You can see the endpoint URL and credentials in the **Overview** section. The endpoint usually has the form `https://<resource-name>.services.ai.azure.com/models`:
+
+:::image type="content" source="../../media/ai-services/overview/overview-endpoint-and-key.png" alt-text="A screenshot showing how to get the URL and key associated with the resource." lightbox="../../media/ai-services/overview/overview-endpoint-and-key.png":::
+
+### Routing
+
+The inference endpoint routes requests to a given deployment by matching the parameter `name` inside of the request to the name of the deployment. This means that *deployments work as an alias of a given model under certain configurations*. This flexibility allows you to deploy a given model multiple times in the service but under different configurations if needed.
+
+:::image type="content" source="../../media/ai-services/endpoint/endpoint-routing.png" alt-text="An illustration showing how routing works for a Meta-llama-3.2-8b-instruct model by indicating such name in the parameter 'model' inside of the payload request." lightbox="../../media/ai-services/endpoint/endpoint-routing.png":::
+
+For example, if you create a deployment named `Mistral-large`, then such deployment can be invoked as:
+
+[!INCLUDE [code-create-chat-completion](../../includes/ai-services/code-create-chat-completion.md)]
+
+> [!TIP]
+> Deployment routing is not case sensitive.
+
+### Supported languages and SDKs
+
+All models deployed in Azure AI model inference service support the [Azure AI model inference API](https://aka.ms/aistudio/modelinference) and its associated family of SDKs, which are available in the following languages:
+
+| Language   | Documentation | Package | Examples |
+|------------|---------|-----|-------|
+| C#         | [Reference](https://aka.ms/azsdk/azure-ai-inference/csharp/reference) | [azure-ai-inference (NuGet)](https://www.nuget.org/packages/Azure.AI.Inference/) | [C# examples](https://aka.ms/azsdk/azure-ai-inference/csharp/samples)       |
+| Java       | [Reference](https://aka.ms/azsdk/azure-ai-inference/java/reference) | [azure-ai-inference (Maven)](https://central.sonatype.com/artifact/com.azure/azure-ai-inference/) | [Java examples](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-inference/src/samples) |
+| JavaScript | [Reference](https://aka.ms/AAp1kxa) | [@azure/ai-inference (npm)](https://www.npmjs.com/package/@azure/ai-inference) | [JavaScript examples](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/ai/ai-inference-rest/samples) |
+| Python     | [Reference](https://aka.ms/azsdk/azure-ai-inference/python/reference) | [azure-ai-inference (PyPi)](https://pypi.org/project/azure-ai-inference/) | [Python examples](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-inference/samples) |
+
+## Azure OpenAI inference endpoint
+
+Azure OpenAI models also support the Azure OpenAI API. This API exposes the full capabilities of OpenAI models and supports additional features like assistants, threads, files, and batch inference.
+
+Each OpenAI model deployment has its own URL associated with such deployment under the Azure OpenAI inference endpoint. However, the same authentication mechanism can be used to consume it. URLs are usually in the form of `https://<resource-name>.openai.azure.com/openai/deployments/<model-deployment-name>`. Learn more in the reference page for [Azure OpenAI API](../../../ai-services/openai/reference.md)
+
+:::image type="content" source="../../media/ai-services/endpoint/endpoint-openai.png" alt-text="An illustration showing how Azure OpenAI deployments contain a single URL for each deployment." lightbox="../../media/ai-services/endpoint/endpoint-openai.png":::
+
+Each deployment has a URL that is the concatenations of the **Azure OpenAI** base URL and the route `/deployments/<model-deployment-name>`.
+
+> [!IMPORTANT]
+> There is no routing mechanism for the Azure OpenAI endpoint, as each URL is exclusive for each model deployment.
+
+### Supported languages and SDKs
+
+The Azure OpenAI endpoint is supported by the **OpenAI SDK (`AzureOpenAI` class)** and **Azure OpenAI SDKs**, which are available in multiple languages:
+
+| Language   | Source code | Package | Examples |
+|------------|---------|-----|-------|
+| C#         | [Source code](https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/openai/Azure.AI.OpenAI) | [Azure.AI.OpenAI (NuGet)](https://www.nuget.org/packages/Azure.AI.OpenAI/) | [C# examples](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/openai/Azure.AI.OpenAI/tests/Samples)       |
+| Go         | [Source code](https://github.com/Azure/azure-sdk-for-go/tree/main/sdk/ai/azopenai) | [azopenai (Go)](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/ai/azopenai)| [Go examples](https://pkg.go.dev/github.com/Azure/azure-sdk-for-go/sdk/ai/azopenai#pkg-examples) |
+| Java       | [Source code](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/openai/azure-ai-openai) | [azure-ai-openai (Maven)](https://central.sonatype.com/artifact/com.azure/azure-ai-openai/) | [Java examples](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/openai/azure-ai-openai/src/samples) |
+| JavaScript | [Source code](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/openai/openai) | [@azure/openai (npm)](https://www.npmjs.com/package/@azure/openai) | [JavaScript examples](https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/openai/openai/samples/) |
+| Python     | [Source code](https://github.com/openai/openai-python) | [openai (PyPi)](https://pypi.org/project/openai/) | [Python examples](https://github.com/openai/openai-cookbook) |
+
+## Next steps
+
+- [Deployment types](deployment-types.md)

articles/ai-studio/ai-services/concepts/quotas-limits.md

@@ -0,0 +1,68 @@
+---
+title: Azure AI model inference quotas and limits
+titleSuffix: Azure AI services
+description: Quick reference, detailed description, and best practices on the quotas and limits for the Azure AI models service in Azure AI services.
+ms.service: azure-ai-studio
+ms.custom: github-universe-2024
+ms.topic: conceptual
+author: sdgilley
+manager: scottpolly
+ms.date: 10/24/2024
+ms.author: sgilley
+ms.reviewer: fasantia
+---
+
+# Azure AI model inference quotas and limits
+
+This article contains a quick reference and a detailed description of the quotas and limits for Azure AI model's inference in Azure AI services. For quotas and limits specific to the Azure OpenAI Service, see [Quota and limits in the Azure OpenAI service](../../../ai-services/openai/quotas-limits.md).
+
+## Quotas and limits reference
+
+The following sections provide you with a quick guide to the default quotas and limits that apply to Azure AI model's inference service in Azure AI services:
+
+### Resource limits
+
+| Limit name | Limit value |
+|--|--|
+| Azure AI Services resources per region per Azure subscription | 30 |
+| Max model deployments per resource | 32 | 
+
+### Rate limits
+
+| Limit name | Limit value |
+| ---------- | ----------- |
+| Tokens per minute (Azure OpenAI models)   | Varies per model and SKU. See [limits for Azure OpenAI](../../../ai-services/openai/quotas-limits.md). |
+| Tokens per minute (rest of models)        | 200.000 |
+| Requests per minute (Azure OpenAI models) | Varies per model and SKU. See [limits for Azure OpenAI](../../../ai-services/openai/quotas-limits.md). |
+| Requests per minute (rest of models)      | 1.000   |
+
+### Other limits
+
+| Limit name | Limit value |
+|--|--|
+| Max number of custom headers in API requests<sup>1</sup> | 10 |
+
+<sup>1</sup> Our current APIs allow up to 10 custom headers, which are passed through the pipeline, and returned. We have noticed some customers now exceed this header count resulting in HTTP 431 errors. There is no solution for this error, other than to reduce header volume. **In future API versions we will no longer pass through custom headers**. We recommend customers not depend on custom headers in future system architectures.
+
+## Usage tiers
+
+Global Standard deployments use Azure's global infrastructure, dynamically routing customer traffic to the data center with best availability for the customer’s inference requests. This enables more consistent latency for customers with low to medium levels of traffic. Customers with high sustained levels of usage might see more variability in response latency.
+
+The Usage Limit determines the level of usage above which customers might see larger variability in response latency. A customer’s usage is defined per model and is the total tokens consumed across all deployments in all subscriptions in all regions for a given tenant.
+
+## General best practices to remain within rate limits
+
+To minimize issues related to rate limits, it's a good idea to use the following techniques:
+
+- Implement retry logic in your application.
+- Avoid sharp changes in the workload. Increase the workload gradually.
+- Test different load increase patterns.
+- Increase the quota assigned to your deployment. Move quota from another deployment, if necessary.
+
+### Request increases to the default quotas and limits
+
+Quota increase requests can be submitted and evaluated per request. [Submit a service request](../../../ai-services/cognitive-services-support-options.md?context=/azure/ai-studio/context/context).
+
+## Next steps
+
+* Learn more about the [Azure AI model inference service](../model-inference.md)