Merge pull request #299427 from MicrosoftDocs/main

5/6/2025 PM Publish

Author: Taojunshen

articles/api-management/azure-openai-api-from-specification.md

@@ -5,7 +5,7 @@ ms.service: azure-api-management
 author: dlepow
 ms.author: danlep
 ms.topic: how-to
-ms.date: 04/01/2025
+ms.date: 04/30/2025
 ms.collection: ce-skilling-ai-copilot
 ms.custom: template-how-to, build-2024
 ---
@@ -65,11 +65,12 @@ To import an Azure OpenAI API to API Management:
 
         For example, if your API Management gateway endpoint is `https://contoso.azure-api.net`, set a **Base URL** similar to `https://contoso.azure-api.net/my-openai-api/openai`.
     1. Optionally select one or more products to associate with the API. Select **Next**.
-1. On the **Policies** tab, optionally enable policies to monitor and manage Azure OpenAI API token consumption. You can also set or edit policies later.
+1. On the **Policies** tab, optionally enable policies to help monitor and manage the API. You can also set or edit policies later.
 
     If selected, enter settings or accept defaults that define the following policies (see linked articles for prerequisites and configuration details):
     * [Manage token consumption](azure-openai-token-limit-policy.md)
     * [Track token usage](azure-openai-emit-token-metric-policy.md) 
+    * [Enable semantic caching of responses](azure-openai-enable-semantic-caching.md)
 
     Select **Review + Create**.
 1. After settings are validated, select **Create**. 

articles/api-management/azure-openai-enable-semantic-caching.md

@@ -18,7 +18,7 @@ ms.collection: ce-skilling-ai-copilot
 Enable semantic caching of responses to Azure OpenAI API requests to reduce bandwidth and processing requirements imposed on the backend APIs and lower latency perceived by API consumers. With semantic caching, you can return cached responses for identical prompts and also for prompts that are similar in meaning, even if the text isn't the same. For background, see [Tutorial: Use Azure Cache for Redis as a semantic cache](../redis/tutorial-semantic-cache.md).
 
 > [!NOTE]
-> The configuration steps in this article enable semantic caching for Azure OpenAI APIs. These steps can be generalized to enable semantic caching for corresponding large language model (LLM) APIs available through the [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api). 
+> The configuration steps in this article enable semantic caching for Azure OpenAI APIs. These steps can be generalized to enable semantic caching for corresponding large language model (LLM) APIs available through the [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api) or with OpenAI-compatible models served through third-party inference providers. 
 
 ## Prerequisites
 

articles/api-management/azure-openai-semantic-cache-lookup-policy.md

@@ -9,7 +9,7 @@ ms.collection: ce-skilling-ai-copilot
 ms.custom:
   - build-2024
 ms.topic: reference
-ms.date: 12/13/2024
+ms.date: 04/29/2025
 ms.author: danlep
 ---
 
@@ -22,7 +22,7 @@ Use the `azure-openai-semantic-cache-lookup` policy to perform cache lookup of r
 > [!NOTE]
 > * This policy must have a corresponding [Cache responses to Azure OpenAI API requests](azure-openai-semantic-cache-store-policy.md) policy. 
 > * For prerequisites and steps to enable semantic caching, see [Enable semantic caching for Azure OpenAI APIs in Azure API Management](azure-openai-enable-semantic-caching.md).
-> * Currently, this policy is in preview.
+
 
 [!INCLUDE [api-management-policy-generic-alert](../../includes/api-management-policy-generic-alert.md)]
 
@@ -45,10 +45,10 @@ Use the `azure-openai-semantic-cache-lookup` policy to perform cache lookup of r
 
 | Attribute         | Description                                            | Required | Default |
 | ----------------- | ------------------------------------------------------ | -------- | ------- |
-| score-threshold	| Similarity score threshold used to determine whether to return a cached response to a prompt. Value is a decimal between 0.0 and 1.0. [Learn more](../redis/tutorial-semantic-cache.md#change-the-similarity-threshold). | Yes |	N/A |
+| score-threshold	| Similarity score threshold used to determine whether to return a cached response to a prompt. Value is a decimal between 0.0 and 1.0. Smaller values represent greater semantic similarity. [Learn more](../redis/tutorial-semantic-cache.md#change-the-similarity-threshold). | Yes |	N/A |
 | embeddings-backend-id | [Backend](backends.md) ID for OpenAI embeddings API call. |	Yes |	N/A |
 | embeddings-backend-auth | Authentication used for Azure OpenAI embeddings API backend. | Yes. Must be set to `system-assigned`. | N/A |
-| ignore-system-messages | Boolean. If set to `true`, removes system messages from a GPT chat completion prompt before assessing cache similarity. | No | false |
+| ignore-system-messages | Boolean. When set to `true` (recommended), removes system messages from a GPT chat completion prompt before assessing cache similarity. | No | false |
 | max-message-count | If specified, number of remaining dialog messages after which caching is skipped. | No | N/A |
                                              
 ## Elements
@@ -67,7 +67,8 @@ Use the `azure-openai-semantic-cache-lookup` policy to perform cache lookup of r
 ### Usage notes
 
 - This policy can only be used once in a policy section.
-
+- Fine-tune the value of `score-threshold` based on your application to ensure that the right sensitivity is used when determining which queries to cache. Start with a low value such as 0.05 and adjust to optimize the ratio of cache hits to misses.
+- The embeddings model should have enough capacity and sufficient context size to accommodate the prompt volume and prompts.
 
 ## Examples
 

articles/api-management/genai-gateway-capabilities.md

@@ -7,7 +7,7 @@ author: dlepow
 ms.service: azure-api-management
 ms.collection: ce-skilling-ai-copilot
 ms.topic: concept-article
-ms.date: 02/05/2025
+ms.date: 04/29/2025
 ms.author: danlep
 ---
 
@@ -18,7 +18,7 @@ ms.author: danlep
 This article introduces capabilities in Azure API Management to help you manage generative AI APIs, such as those provided by [Azure OpenAI Service](/azure/ai-services/openai/overview). Azure API Management provides a range of policies, metrics, and other features to enhance security, performance, and reliability for the APIs serving your intelligent apps. Collectively, these features are called *AI gateway capabilities* for your generative AI APIs.
 
 > [!NOTE]
-> * This article focuses on capabilities to manage APIs exposed by Azure OpenAI Service. Many of the AI gateway capabilities apply to other large language model (LLM) APIs, including those available through [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api).
+> * Use AI gateway capabilities to manage APIs exposed by Azure OpenAI Service, available through [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api), or with OpenAI-compatible models served through third-party inference providers.
 > * AI gateway capabilities are features of API Management's existing API gateway, not a separate API gateway. For more information on API Management, see [Azure API Management overview](api-management-key-concepts.md).
 
 ## Challenges in managing generative AI APIs
@@ -36,7 +36,7 @@ The rest of this article describes how Azure API Management can help you address
 
 ## Import Azure OpenAI Service resource as an API
 
-[Import an API from an Azure OpenAI Service endpoint](azure-openai-api-from-specification.md) to Azure API management using a single-click experience. API Management streamlines the onboarding process by automatically importing the OpenAPI schema for the Azure OpenAI API and sets up authentication to the Azure OpenAI endpoint using managed identity, removing the need for manual configuration. Within the same user-friendly experience, you can preconfigure policies for [token limits](#token-limit-policy) and [emitting token metrics](#emit-token-metric-policy). 
+[Import an API from an Azure OpenAI Service endpoint](azure-openai-api-from-specification.md) to Azure API management using a single-click experience. API Management streamlines the onboarding process by automatically importing the OpenAPI schema for the Azure OpenAI API and sets up authentication to the Azure OpenAI endpoint using managed identity, removing the need for manual configuration. Within the same user-friendly experience, you can preconfigure policies for [token limits](#token-limit-policy), [emitting token metrics](#emit-token-metric-policy), and [semantic caching](#semantic-caching-policy). 
 
 :::image type="content" source="media/azure-openai-api-from-specification/azure-openai-api.png" alt-text="Screenshot of Azure OpenAI API tile in the portal.":::
 
@@ -57,7 +57,7 @@ The following basic example demonstrates how to set a TPM limit of 500 per subsc
 ```
 
 > [!TIP]
-> To manage and enforce token limits for LLM APIs available through the Azure AI Model Inference API, API Management provides the equivalent [llm-token-limit](llm-token-limit-policy.md) policy.
+> To manage and enforce token limits for other LLM APIs, API Management provides the equivalent [llm-token-limit](llm-token-limit-policy.md) policy.
 
 
 ## Emit token metric policy 
@@ -79,7 +79,7 @@ For example, the following policy sends metrics to Application Insights split by
 ```
 
 > [!TIP]
-> To send metrics for LLM APIs available through the Azure AI Model Inference API, API Management provides the equivalent [llm-emit-token-metric](llm-emit-token-metric-policy.md) policy.
+> To send metrics for other LLM APIs, API Management provides the equivalent [llm-emit-token-metric](llm-emit-token-metric-policy.md) policy.
 
 ## Backend load balancer and circuit breaker
 
@@ -99,12 +99,18 @@ Configure [Azure OpenAI semantic caching](azure-openai-enable-semantic-caching.m
 
 :::image type="content" source="media/genai-gateway-capabilities/semantic-caching.png" alt-text="Diagram of semantic caching in API Management.":::
 
-In API Management, enable semantic caching by using Azure Redis Enterprise or another [external cache](api-management-howto-cache-external.md) compatible with RediSearch and onboarded to Azure API Management. By using the Azure OpenAI Service Embeddings API, the [azure-openai-semantic-cache-store](azure-openai-semantic-cache-store-policy.md) and [azure-openai-semantic-cache-lookup](azure-openai-semantic-cache-lookup-policy.md) policies store and retrieve semantically similar prompt completions from the cache. This approach ensures completions reuse, resulting in reduced token consumption and improved response performance. 
+In API Management, enable semantic caching by using Azure Redis Enterprise, Azure Managed Redis, or another [external cache](api-management-howto-cache-external.md) compatible with RediSearch and onboarded to Azure API Management. By using the Azure OpenAI Service Embeddings API, the [azure-openai-semantic-cache-store](azure-openai-semantic-cache-store-policy.md) and [azure-openai-semantic-cache-lookup](azure-openai-semantic-cache-lookup-policy.md) policies store and retrieve semantically similar prompt completions from the cache. This approach ensures completions reuse, resulting in reduced token consumption and improved response performance. 
 
 > [!TIP]
-> To enable semantic caching for LLM APIs available through the Azure AI Model Inference API, API Management provides the equivalent [llm-semantic-cache-store-policy](llm-semantic-cache-store-policy.md) and [llm-semantic-cache-lookup-policy](llm-semantic-cache-lookup-policy.md) policies.
+> To enable semantic caching for other LLM APIs, API Management provides the equivalent [llm-semantic-cache-store-policy](llm-semantic-cache-store-policy.md) and [llm-semantic-cache-lookup-policy](llm-semantic-cache-lookup-policy.md) policies.
 
 
+## Content safety policy
+
+To help safeguard users from harmful, offensive, or misleading content, you can automatically moderate all incoming requests to an LLM API by configuring the [llm-content-safety](llm-content-safety-policy.md) policy. The policy enforces content safety checks on LLM prompts by transmitting them first to the [Azure AI Content Safety](/azure/ai-services/content-safety/overview) service before sending to the backend LLM API. 
+
+:::image type="content" source="media/genai-gateway-capabilities/content-safety.png" alt-text="Diagram of moderating prompts by Azure AI Content Safety in an API Management policy.":::
+
 ## Labs and samples
 
 * [Labs for the AI gateway capabilities of Azure API Management](https://github.com/Azure-Samples/ai-gateway)

articles/api-management/llm-emit-token-metric-policy.md

@@ -18,9 +18,6 @@ ms.custom:
 
 The `llm-emit-token-metric` policy sends custom metrics to Application Insights about consumption of large language model (LLM) tokens through LLM APIs. Token count metrics include: Total Tokens, Prompt Tokens, and Completion Tokens. 
 
-> [!NOTE]
-> Currently, this policy is in preview.
-
 [!INCLUDE [api-management-policy-generic-alert](../../includes/api-management-policy-generic-alert.md)]
 
 [!INCLUDE [api-management-llm-models](../../includes/api-management-llm-models.md)]

articles/api-management/llm-semantic-cache-lookup-policy.md

@@ -9,7 +9,7 @@ ms.collection: ce-skilling-ai-copilot
 ms.custom:
   - build-2024
 ms.topic: reference
-ms.date: 12/13/2024
+ms.date: 04/29/2025
 ms.author: danlep
 ---
 
@@ -22,7 +22,6 @@ Use the `llm-semantic-cache-lookup` policy to perform cache lookup of responses
 > [!NOTE]
 > * This policy must have a corresponding [Cache responses to large language model API requests](llm-semantic-cache-store-policy.md) policy. 
 > * For prerequisites and steps to enable semantic caching, see [Enable semantic caching for Azure OpenAI APIs in Azure API Management](azure-openai-enable-semantic-caching.md).
-> * Currently, this policy is in preview.
 
 [!INCLUDE [api-management-policy-generic-alert](../../includes/api-management-policy-generic-alert.md)]
 
@@ -45,10 +44,10 @@ Use the `llm-semantic-cache-lookup` policy to perform cache lookup of responses
 
 | Attribute         | Description                                            | Required | Default |
 | ----------------- | ------------------------------------------------------ | -------- | ------- |
-| score-threshold	| Similarity score threshold used to determine whether to return a cached response to a prompt. Value is a decimal between 0.0 and 1.0. [Learn more](../redis/tutorial-semantic-cache.md#change-the-similarity-threshold). | Yes |	N/A |
+| score-threshold	| Similarity score threshold used to determine whether to return a cached response to a prompt. Value is a decimal between 0.0 and 1.0. Smaller values represent greater semantic similarity. [Learn more](../redis/tutorial-semantic-cache.md#change-the-similarity-threshold). | Yes |	N/A |
 | embeddings-backend-id | [Backend](backends.md) ID for OpenAI embeddings API call. |	Yes |	N/A |
 | embeddings-backend-auth | Authentication used for Azure OpenAI embeddings API backend. | Yes. Must be set to `system-assigned`. | N/A |
-| ignore-system-messages | Boolean. If set to `true`, removes system messages from a GPT chat completion prompt before assessing cache similarity. | No | false |
+| ignore-system-messages | Boolean. When set to `true` (recommended), removes system messages from a GPT chat completion prompt before assessing cache similarity. | No | false |
 | max-message-count | If specified, number of remaining dialog messages after which caching is skipped. | No | N/A |
                                              
 ## Elements
@@ -67,6 +66,8 @@ Use the `llm-semantic-cache-lookup` policy to perform cache lookup of responses
 ### Usage notes
 
 - This policy can only be used once in a policy section.
+- Fine-tune the value of `score-threshold` based on your application to ensure that the right sensitivity is used when determining which queries to cache. Start with a low value such as 0.05 and adjust to optimize the ratio of cache hits to misses.
+- The embeddings model should have enough capacity and sufficient context size to accommodate the prompt volume and prompts.
 
 
 ## Examples

articles/api-management/llm-semantic-cache-store-policy.md

@@ -21,7 +21,6 @@ The `llm-semantic-cache-store` policy caches responses to chat completion API re
 > [!NOTE]
 > * This policy must have a corresponding [Get cached responses to large language model API requests](llm-semantic-cache-lookup-policy.md) policy. 
 > * For prerequisites and steps to enable semantic caching, see [Enable semantic caching for Azure OpenAI APIs in Azure API Management](azure-openai-enable-semantic-caching.md). 
-> * Currently, this policy is in preview.
 
 [!INCLUDE [api-management-policy-generic-alert](../../includes/api-management-policy-generic-alert.md)]
 

articles/api-management/llm-token-limit-policy.md

@@ -20,9 +20,6 @@ The `llm-token-limit` policy prevents large language model (LLM) API usage spike
 
 By relying on token usage metrics returned from the LLM endpoint, the policy can accurately monitor and enforce limits in real time. The policy also enables precalculation of prompt tokens by API Management, minimizing unnecessary requests to the LLM backend if the limit is already exceeded.
 
-> [!NOTE]
-> Currently, this policy is in preview.
-
 [!INCLUDE [api-management-policy-generic-alert](../../includes/api-management-policy-generic-alert.md)]
 
 [!INCLUDE [api-management-llm-models](../../includes/api-management-llm-models.md)]