Skip to content

Commit 5680c3c

Browse files
Merge pull request #4719 from PatrickFarley/foundry-build
Foundry build
2 parents dd7e9b6 + f14a97a commit 5680c3c

39 files changed

+94
-95
lines changed

articles/ai-foundry/concepts/ai-red-teaming-agent.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,11 @@ author: lgayhardt
1717

1818
The AI Red Teaming Agent (preview) is a powerful tool designed to help organizations proactively find safety risks associated with generative AI systems during design and development of generative AI models and applications.
1919

20-
Traditional red teaming involves exploiting the cyber kill chain and describes the process by which a system is tested for security vulnerabilities. However, with the rise of generative AI, the term AI red teaming has been coined to describe probing for novel risks (both content safety and security related) that these systems present and refers to simulating the behavior of an adversarial user who is trying to cause your AI system to misbehave in a particular way.
20+
Traditional red teaming involves exploiting the cyber kill chain and describes the process by which a system is tested for security vulnerabilities. However, with the rise of generative AI, the term AI red teaming has been coined to describe probing for novel risks (both content and security related) that these systems present and refers to simulating the behavior of an adversarial user who is trying to cause your AI system to misbehave in a particular way.
2121

2222
The AI Red Teaming Agent leverages Microsoft's open-source framework for Python Risk Identification Tool's ([PyRIT](https://github.com/Azure/PyRIT)) AI red teaming capabilities along with Azure AI Foundry's [Risk and Safety Evaluations](./evaluation-metrics-built-in.md#risk-and-safety-evaluators) to help you automatically assess safety issues in three ways:
2323

24-
- **Automated scans for content safety risks:** Firstly, you can automatically scan your model and application endpoints for safety risks by simulating adversarial probing.
24+
- **Automated scans for content risks:** Firstly, you can automatically scan your model and application endpoints for safety risks by simulating adversarial probing.
2525
- **Evaluate probing success:** Next, you can evaluate and score each attack-response pair to generate insightful metrics such as Attack Success Rate (ASR).
2626
- **Reporting and logging** Finally, you can generate a score card of the attack probing techniques and risk categories to help you decide if the system is ready for deployment. Findings can be logged, monitored, and tracked over time directly in Azure AI Foundry, ensuring compliance and continuous risk mitigation.
2727

articles/ai-foundry/concepts/content-filtering.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ author: PatrickFarley
2626

2727
The content filtering system is powered by [Azure AI Content Safety](../../ai-services/content-safety/overview.md), and it works by running both the prompt input and completion output through a set of classification models designed to detect and prevent the output of harmful content. Variations in API configurations and application design might affect completions and thus filtering behavior.
2828

29-
With Azure OpenAI model deployments, you can use the default content filter or create your own content filter (described later on). Models available through **serverless APIs** have content filtering enabled by default. To learn more about the default content filter enabled for serverless APIs, see [Content safety for models curated by Azure AI in the model catalog](model-catalog-content-safety.md).
29+
With Azure OpenAI model deployments, you can use the default content filter or create your own content filter (described later on). Models available through **serverless APIs** have content filtering enabled by default. To learn more about the default content filter enabled for serverless APIs, see [Guardrails & controls for models curated by Azure AI in the model catalog](model-catalog-content-safety.md).
3030

3131
## Language support
3232

articles/ai-foundry/concepts/deployments-overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ Azure AI Foundry offers four different deployment options:
2727

2828
|Name | Azure OpenAI | Azure AI model inference | Standard deployment | Managed compute |
2929
|-------------------------------|----------------------|-------------------|----------------|-----------------|
30-
| Which models can be deployed? | [Azure OpenAI models](../../ai-services/openai/concepts/models.md) | [Azure OpenAI models and Standard deployment](../../ai-foundry/model-inference/concepts/models.md) | [Standard deployment](../how-to/model-catalog-overview.md#content-safety-for-models-deployed-via-serverless-apis) | [Open and custom models](../how-to/model-catalog-overview.md#availability-of-models-for-deployment-as-managed-compute) |
30+
| Which models can be deployed? | [Azure OpenAI models](../../ai-services/openai/concepts/models.md) | [Azure OpenAI models and Standard deployment](../../ai-foundry/model-inference/concepts/models.md) | [Standard deployment](../how-to/model-catalog-overview.md) | [Open and custom models](../how-to/model-catalog-overview.md#availability-of-models-for-deployment-as-managed-compute) |
3131
| Deployment resource | Azure OpenAI resource | Azure AI services resource | AI project resource | AI project resource |
3232
| Requires Hubs/Projects | No | No | Yes | Yes |
3333
| Data processing options | Regional <br /> Data-zone <br /> Global | Global | Regional | Regional |

articles/ai-foundry/concepts/evaluation-metrics-built-in.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -197,12 +197,12 @@ We support evaluating vulnerability towards the following types of jailbreak att
197197
- **Direct attack jailbreak** (also known as UPIA or User Prompt Injected Attack) injects prompts in the user role turn of conversations or queries to generative AI applications. Jailbreaks are when a model response bypasses the restrictions placed on it. Jailbreak also happens where an LLM deviates from the intended task or topic.
198198
- **Indirect attack jailbreak** (also known as XPIA or cross domain prompt injected attack) injects prompts in the returned documents or context of the user's query to generative AI applications.
199199

200-
*Evaluating direct attack* is a comparative measurement using the content safety evaluators as a control. It isn't its own AI-assisted evaluator. Run `ContentSafetyEvaluator` on two different, red-teamed datasets:
200+
*Evaluating direct attack* is a comparative measurement using the Azure AI Content Safety evaluators as a control. It isn't its own AI-assisted evaluator. Run `ContentSafetyEvaluator` on two different, red-teamed datasets:
201201

202202
- Baseline adversarial test dataset.
203203
- Adversarial test dataset with direct attack jailbreak injections in the first turn.
204204

205-
You can do this with functionality and attack datasets generated with the [direct attack simulator](../how-to/develop/simulator-interaction-data.md#simulating-jailbreak-attacks) with the same randomization seed. Then you can evaluate jailbreak vulnerability by comparing results from content safety evaluators between the two test dataset's aggregate scores for each safety evaluator. A direct attack jailbreak defect is detected when there's presence of content harm response detected in the second direct attack injected dataset when there was none or lower severity detected in the first control dataset.
205+
You can do this with functionality and attack datasets generated with the [direct attack simulator](../how-to/develop/simulator-interaction-data.md#simulating-jailbreak-attacks) with the same randomization seed. Then you can evaluate jailbreak vulnerability by comparing results from Azure AI Content Safety evaluators between the two test dataset's aggregate scores for each safety evaluator. A direct attack jailbreak defect is detected when there's presence of content harm response detected in the second direct attack injected dataset when there was none or lower severity detected in the first control dataset.
206206

207207
### Indirect attack definition and label
208208

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
---
2-
title: Content safety for models curated by Azure AI in the model catalog
2+
title: Guardrails & controls for models curated by Azure AI in the model catalog
33
titleSuffix: Azure AI Foundry
4-
description: Learn about content safety for models deployed using serverless APIs, using Azure AI Foundry.
4+
description: Learn about Guardrails & controls for models deployed using serverless APIs, using Azure AI Foundry.
55
manager: scottpolly
66
ms.service: azure-ai-foundry
77
ms.topic: conceptual
@@ -13,28 +13,28 @@ reviewer: ositanachi
1313
ms.custom:
1414
---
1515

16-
# Content safety for models curated by Azure AI in the model catalog
16+
# Guardrails & controls for models curated by Azure AI in the model catalog
1717

1818
[!INCLUDE [feature-preview](../includes/feature-preview.md)]
1919

20-
In this article, learn about content safety capabilities for models from the model catalog deployed using serverless APIs.
20+
In this article, learn about Guardrails & controls capabilities for models from the model catalog deployed using serverless APIs.
2121

2222

2323
## Content filter defaults
2424

25-
Azure AI uses a default configuration of [Azure AI Content Safety](/azure/ai-services/content-safety/overview) content filters to detect harmful content across four categories including hate and fairness, self-harm, sexual, and violence for models deployed via serverless APIs. To learn more about content filtering (preview), see [Understand harm categories](#understand-harm-categories).
25+
Azure AI uses a default configuration of [Azure AI Content Safety](/azure/ai-services/content-safety/overview) content filters to detect harmful content across four categories including hate and fairness, self-harm, sexual, and violence for models deployed via serverless APIs. To learn more about content filtering, see [Understand harm categories](#understand-harm-categories).
2626

27-
The default content filtering configuration for text models is set to filter at the medium severity threshold, filtering any detected content at this level or higher. For image models, the default content filtering configuration is set at the low configuration threshold, filtering at this level or higher. For models deployed using the [Azure AI model inference service](../../ai-foundry/model-inference/how-to/configure-content-filters.md), you can create configurable filters by selecting the **Content filters** tab within the **Safety + security** page of the Azure AI Foundry portal.
27+
The default content filtering configuration for text models is set to filter at the medium severity threshold, filtering any detected content at this level or higher. For image models, the default content filtering configuration is set at the low configuration threshold, filtering at this level or higher. For models deployed using the [Azure AI model inference service](../../ai-foundry/model-inference/how-to/configure-content-filters.md), you can create configurable filters by selecting the **Content filters** tab within the **Guardrails & controls** page of the Azure AI Foundry portal.
2828

2929
> [!TIP]
30-
> Content filtering (preview) isn't available for certain model types that are deployed via serverless APIs. These model types include embedding models and time series models.
30+
> Content filtering isn't available for certain model types that are deployed via serverless APIs. These model types include embedding models and time series models.
3131
32-
Content filtering (preview) occurs synchronously as the service processes prompts to generate content. You might be billed separately according to [Azure AI Content Safety pricing](https://azure.microsoft.com/pricing/details/cognitive-services/content-safety/) for such use. You can disable content filtering (preview) for individual serverless endpoints either:
32+
Content filtering occurs synchronously as the service processes prompts to generate content. You might be billed separately according to [Azure AI Content Safety pricing](https://azure.microsoft.com/pricing/details/cognitive-services/content-safety/) for such use. You can disable content filtering for individual serverless endpoints either:
3333

3434
- When you first deploy a language model
3535
- Later, by selecting the content filtering toggle on the deployment details page
3636

37-
Suppose you decide to use an API other than the [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api) to work with a model that is deployed via a serverless API. In such a situation, content filtering (preview) isn't enabled unless you implement it separately by using Azure AI Content Safety. To get started with Azure AI Content Safety, see [Quickstart: Analyze text content](/azure/ai-services/content-safety/quickstart-text). You run a higher risk of exposing users to harmful content if you don't use content filtering (preview) when working with models that are deployed via serverless APIs.
37+
Suppose you decide to use an API other than the [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api) to work with a model that is deployed via a serverless API. In such a situation, content filtering isn't enabled unless you implement it separately by using Azure AI Content Safety. To get started with Azure AI Content Safety, see [Quickstart: Analyze text content](/azure/ai-services/content-safety/quickstart-text). You run a higher risk of exposing users to harmful content if you don't use content filtering when working with models that are deployed via serverless APIs.
3838

3939
[!INCLUDE [content-safety-harm-categories](../includes/content-safety-harm-categories.md)]
4040

@@ -44,6 +44,6 @@ Pricing details are viewable at [Azure AI Content Safety pricing](https://azure.
4444

4545
## Related content
4646

47-
- [How to configure content filters (preview) for models in Azure AI services](../../ai-foundry/model-inference/how-to/configure-content-filters.md)
47+
- [How to configure content filters for models in Azure AI services](../../ai-foundry/model-inference/how-to/configure-content-filters.md)
4848
- [What is Azure AI Content Safety?](../../ai-services/content-safety/overview.md)
4949
- [Model catalog and collections in Azure AI Foundry portal](../how-to/model-catalog-overview.md)

articles/ai-foundry/concepts/models-featured.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -364,5 +364,5 @@ For examples of how to use Stability AI models, see the following examples:
364364
- [Deploy models as serverless APIs](../how-to/deploy-models-serverless.md)
365365
- [Model catalog and collections in Azure AI Foundry portal](../how-to/model-catalog-overview.md)
366366
- [Region availability for models in serverless API endpoints](../how-to/deploy-models-serverless-availability.md)
367-
- [Content safety for models curated by Azure AI in the model catalog](model-catalog-content-safety.md)
367+
- [Guardrails & controls for models curated by Azure AI in the model catalog](model-catalog-content-safety.md)
368368

articles/ai-foundry/how-to/concept-data-privacy.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ The model processes your input prompts and generates outputs based on its functi
4747

4848
Microsoft acts as the data processor for prompts and outputs sent to, and generated by, a model deployed for pay-as-you-go inferencing (MaaS). Microsoft doesn't share these prompts and outputs with the model provider. Also, Microsoft doesn't use these prompts and outputs to train or improve Microsoft models, the model provider's models, or any third party's models.
4949

50-
Models are stateless, and they don't store any prompts or outputs. If content filtering (preview) is enabled, the Azure AI Content Safety service screens prompts and outputs for certain categories of harmful content in real time. [Learn more about how Azure AI Content Safety processes data](/legal/cognitive-services/content-safety/data-privacy).
50+
Models are stateless, and they don't store any prompts or outputs. If content filtering is enabled, the Azure AI Content Safety service screens prompts and outputs for certain categories of harmful content in real time. [Learn more about how Azure AI Content Safety processes data](/legal/cognitive-services/content-safety/data-privacy).
5151

5252
Prompts and outputs are processed within the geography specified during deployment, but they might be processed between regions within the geography for operational purposes. Operational purposes include performance and capacity management.
5353

articles/ai-foundry/how-to/deploy-models-gretel-navigator.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -233,11 +233,11 @@ result = client.complete(
233233
```
234234

235235

236-
### Apply content safety
236+
### Apply Guardrails and controls
237237

238-
The Azure AI model inference API supports [Azure AI content safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AI content safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering (preview) system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
238+
The Azure AI model inference API supports [Azure AI Content Safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AI Content Safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
239239

240-
The following example shows how to handle events when the model detects harmful content in the input prompt and content safety is enabled.
240+
The following example shows how to handle events when the model detects harmful content in the input prompt and the filter is enabled.
241241

242242

243243
```python
@@ -477,11 +477,11 @@ The following example request shows other parameters that you can specify in the
477477
}
478478
```
479479

480-
### Apply content safety
480+
### Apply Guardrails & controls
481481

482-
The Azure AI model inference API supports [Azure AI content safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AI content safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering (preview) system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
482+
The Azure AI model inference API supports [Azure AI Content Safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AI Content Safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
483483

484-
The following example shows how to handle events when the model detects harmful content in the input prompt and content safety is enabled.
484+
The following example shows how to handle events when the model detects harmful content in the input prompt.
485485

486486

487487
```json

articles/ai-foundry/how-to/deploy-models-serverless.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -332,7 +332,7 @@ In this section, you create an endpoint with the name **meta-llama3-8b-qwerty**.
332332
333333
:::image type="content" source="../media/deploy-monitor/serverless/deployment-name.png" alt-text="A screenshot showing how to specify the name of the deployment you want to create." lightbox="../media/deploy-monitor/serverless/deployment-name.png":::
334334
> [!TIP]
335-
> The **Content filter (preview)** option is enabled by default. Leave the default setting for the service to detect harmful content such as hate, self-harm, sexual, and violent content. For more information about content filtering (preview), see [Content filtering in Azure AI Foundry portal](../concepts/content-filtering.md).
335+
> The **Content filter** option is enabled by default. Leave the default setting for the service to detect harmful content such as hate, self-harm, sexual, and violent content. For more information about content filtering, see [Content filtering in Azure AI Foundry portal](../concepts/content-filtering.md).
336336
337337
1. Select **Deploy**. Wait until the deployment is ready and you're redirected to the Deployments page.
338338

0 commit comments

Comments
 (0)