You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/ai-foundry/concepts/ai-red-teaming-agent.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -17,11 +17,11 @@ author: lgayhardt
17
17
18
18
The AI Red Teaming Agent (preview) is a powerful tool designed to help organizations proactively find safety risks associated with generative AI systems during design and development of generative AI models and applications.
19
19
20
-
Traditional red teaming involves exploiting the cyber kill chain and describes the process by which a system is tested for security vulnerabilities. However, with the rise of generative AI, the term AI red teaming has been coined to describe probing for novel risks (both content safety and security related) that these systems present and refers to simulating the behavior of an adversarial user who is trying to cause your AI system to misbehave in a particular way.
20
+
Traditional red teaming involves exploiting the cyber kill chain and describes the process by which a system is tested for security vulnerabilities. However, with the rise of generative AI, the term AI red teaming has been coined to describe probing for novel risks (both content and security related) that these systems present and refers to simulating the behavior of an adversarial user who is trying to cause your AI system to misbehave in a particular way.
21
21
22
22
The AI Red Teaming Agent leverages Microsoft's open-source framework for Python Risk Identification Tool's ([PyRIT](https://github.com/Azure/PyRIT)) AI red teaming capabilities along with Azure AI Foundry's [Risk and Safety Evaluations](./evaluation-metrics-built-in.md#risk-and-safety-evaluators) to help you automatically assess safety issues in three ways:
23
23
24
-
-**Automated scans for content safety risks:** Firstly, you can automatically scan your model and application endpoints for safety risks by simulating adversarial probing.
24
+
-**Automated scans for content risks:** Firstly, you can automatically scan your model and application endpoints for safety risks by simulating adversarial probing.
25
25
-**Evaluate probing success:** Next, you can evaluate and score each attack-response pair to generate insightful metrics such as Attack Success Rate (ASR).
26
26
-**Reporting and logging** Finally, you can generate a score card of the attack probing techniques and risk categories to help you decide if the system is ready for deployment. Findings can be logged, monitored, and tracked over time directly in Azure AI Foundry, ensuring compliance and continuous risk mitigation.
Copy file name to clipboardExpand all lines: articles/ai-foundry/concepts/content-filtering.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,7 +26,7 @@ author: PatrickFarley
26
26
27
27
The content filtering system is powered by [Azure AI Content Safety](../../ai-services/content-safety/overview.md), and it works by running both the prompt input and completion output through a set of classification models designed to detect and prevent the output of harmful content. Variations in API configurations and application design might affect completions and thus filtering behavior.
28
28
29
-
With Azure OpenAI model deployments, you can use the default content filter or create your own content filter (described later on). Models available through **serverless APIs** have content filtering enabled by default. To learn more about the default content filter enabled for serverless APIs, see [Content safety for models curated by Azure AI in the model catalog](model-catalog-content-safety.md).
29
+
With Azure OpenAI model deployments, you can use the default content filter or create your own content filter (described later on). Models available through **serverless APIs** have content filtering enabled by default. To learn more about the default content filter enabled for serverless APIs, see [Guardrails & controls for models curated by Azure AI in the model catalog](model-catalog-content-safety.md).
| Which models can be deployed? |[Azure OpenAI models](../../ai-services/openai/concepts/models.md)|[Azure OpenAI models and Standard deployment](../../ai-foundry/model-inference/concepts/models.md)|[Standard deployment](../how-to/model-catalog-overview.md#content-safety-for-models-deployed-via-serverless-apis)|[Open and custom models](../how-to/model-catalog-overview.md#availability-of-models-for-deployment-as-managed-compute)|
30
+
| Which models can be deployed? |[Azure OpenAI models](../../ai-services/openai/concepts/models.md)|[Azure OpenAI models and Standard deployment](../../ai-foundry/model-inference/concepts/models.md)|[Standard deployment](../how-to/model-catalog-overview.md)|[Open and custom models](../how-to/model-catalog-overview.md#availability-of-models-for-deployment-as-managed-compute)|
31
31
| Deployment resource | Azure OpenAI resource | Azure AI services resource | AI project resource | AI project resource |
32
32
| Requires Hubs/Projects | No | No | Yes | Yes |
33
33
| Data processing options | Regional <br /> Data-zone <br /> Global | Global | Regional | Regional |
Copy file name to clipboardExpand all lines: articles/ai-foundry/concepts/evaluation-metrics-built-in.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -197,12 +197,12 @@ We support evaluating vulnerability towards the following types of jailbreak att
197
197
-**Direct attack jailbreak** (also known as UPIA or User Prompt Injected Attack) injects prompts in the user role turn of conversations or queries to generative AI applications. Jailbreaks are when a model response bypasses the restrictions placed on it. Jailbreak also happens where an LLM deviates from the intended task or topic.
198
198
-**Indirect attack jailbreak** (also known as XPIA or cross domain prompt injected attack) injects prompts in the returned documents or context of the user's query to generative AI applications.
199
199
200
-
*Evaluating direct attack* is a comparative measurement using the content safety evaluators as a control. It isn't its own AI-assisted evaluator. Run `ContentSafetyEvaluator` on two different, red-teamed datasets:
200
+
*Evaluating direct attack* is a comparative measurement using the Azure AI Content Safety evaluators as a control. It isn't its own AI-assisted evaluator. Run `ContentSafetyEvaluator` on two different, red-teamed datasets:
201
201
202
202
- Baseline adversarial test dataset.
203
203
- Adversarial test dataset with direct attack jailbreak injections in the first turn.
204
204
205
-
You can do this with functionality and attack datasets generated with the [direct attack simulator](../how-to/develop/simulator-interaction-data.md#simulating-jailbreak-attacks) with the same randomization seed. Then you can evaluate jailbreak vulnerability by comparing results from content safety evaluators between the two test dataset's aggregate scores for each safety evaluator. A direct attack jailbreak defect is detected when there's presence of content harm response detected in the second direct attack injected dataset when there was none or lower severity detected in the first control dataset.
205
+
You can do this with functionality and attack datasets generated with the [direct attack simulator](../how-to/develop/simulator-interaction-data.md#simulating-jailbreak-attacks) with the same randomization seed. Then you can evaluate jailbreak vulnerability by comparing results from Azure AI Content Safety evaluators between the two test dataset's aggregate scores for each safety evaluator. A direct attack jailbreak defect is detected when there's presence of content harm response detected in the second direct attack injected dataset when there was none or lower severity detected in the first control dataset.
In this article, learn about content safety capabilities for models from the model catalog deployed using serverless APIs.
20
+
In this article, learn about Guardrails & controls capabilities for models from the model catalog deployed using serverless APIs.
21
21
22
22
23
23
## Content filter defaults
24
24
25
-
Azure AI uses a default configuration of [Azure AI Content Safety](/azure/ai-services/content-safety/overview) content filters to detect harmful content across four categories including hate and fairness, self-harm, sexual, and violence for models deployed via serverless APIs. To learn more about content filtering (preview), see [Understand harm categories](#understand-harm-categories).
25
+
Azure AI uses a default configuration of [Azure AI Content Safety](/azure/ai-services/content-safety/overview) content filters to detect harmful content across four categories including hate and fairness, self-harm, sexual, and violence for models deployed via serverless APIs. To learn more about content filtering, see [Understand harm categories](#understand-harm-categories).
26
26
27
-
The default content filtering configuration for text models is set to filter at the medium severity threshold, filtering any detected content at this level or higher. For image models, the default content filtering configuration is set at the low configuration threshold, filtering at this level or higher. For models deployed using the [Azure AI model inference service](../../ai-foundry/model-inference/how-to/configure-content-filters.md), you can create configurable filters by selecting the **Content filters** tab within the **Safety + security** page of the Azure AI Foundry portal.
27
+
The default content filtering configuration for text models is set to filter at the medium severity threshold, filtering any detected content at this level or higher. For image models, the default content filtering configuration is set at the low configuration threshold, filtering at this level or higher. For models deployed using the [Azure AI model inference service](../../ai-foundry/model-inference/how-to/configure-content-filters.md), you can create configurable filters by selecting the **Content filters** tab within the **Guardrails & controls** page of the Azure AI Foundry portal.
28
28
29
29
> [!TIP]
30
-
> Content filtering (preview) isn't available for certain model types that are deployed via serverless APIs. These model types include embedding models and time series models.
30
+
> Content filtering isn't available for certain model types that are deployed via serverless APIs. These model types include embedding models and time series models.
31
31
32
-
Content filtering (preview) occurs synchronously as the service processes prompts to generate content. You might be billed separately according to [Azure AI Content Safety pricing](https://azure.microsoft.com/pricing/details/cognitive-services/content-safety/) for such use. You can disable content filtering (preview) for individual serverless endpoints either:
32
+
Content filtering occurs synchronously as the service processes prompts to generate content. You might be billed separately according to [Azure AI Content Safety pricing](https://azure.microsoft.com/pricing/details/cognitive-services/content-safety/) for such use. You can disable content filtering for individual serverless endpoints either:
33
33
34
34
- When you first deploy a language model
35
35
- Later, by selecting the content filtering toggle on the deployment details page
36
36
37
-
Suppose you decide to use an API other than the [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api) to work with a model that is deployed via a serverless API. In such a situation, content filtering (preview) isn't enabled unless you implement it separately by using Azure AI Content Safety. To get started with Azure AI Content Safety, see [Quickstart: Analyze text content](/azure/ai-services/content-safety/quickstart-text). You run a higher risk of exposing users to harmful content if you don't use content filtering (preview) when working with models that are deployed via serverless APIs.
37
+
Suppose you decide to use an API other than the [Azure AI Model Inference API](/azure/ai-studio/reference/reference-model-inference-api) to work with a model that is deployed via a serverless API. In such a situation, content filtering isn't enabled unless you implement it separately by using Azure AI Content Safety. To get started with Azure AI Content Safety, see [Quickstart: Analyze text content](/azure/ai-services/content-safety/quickstart-text). You run a higher risk of exposing users to harmful content if you don't use content filtering when working with models that are deployed via serverless APIs.
Copy file name to clipboardExpand all lines: articles/ai-foundry/how-to/concept-data-privacy.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -47,7 +47,7 @@ The model processes your input prompts and generates outputs based on its functi
47
47
48
48
Microsoft acts as the data processor for prompts and outputs sent to, and generated by, a model deployed for pay-as-you-go inferencing (MaaS). Microsoft doesn't share these prompts and outputs with the model provider. Also, Microsoft doesn't use these prompts and outputs to train or improve Microsoft models, the model provider's models, or any third party's models.
49
49
50
-
Models are stateless, and they don't store any prompts or outputs. If content filtering (preview) is enabled, the Azure AI Content Safety service screens prompts and outputs for certain categories of harmful content in real time. [Learn more about how Azure AI Content Safety processes data](/legal/cognitive-services/content-safety/data-privacy).
50
+
Models are stateless, and they don't store any prompts or outputs. If content filtering is enabled, the Azure AI Content Safety service screens prompts and outputs for certain categories of harmful content in real time. [Learn more about how Azure AI Content Safety processes data](/legal/cognitive-services/content-safety/data-privacy).
51
51
52
52
Prompts and outputs are processed within the geography specified during deployment, but they might be processed between regions within the geography for operational purposes. Operational purposes include performance and capacity management.
Copy file name to clipboardExpand all lines: articles/ai-foundry/how-to/deploy-models-gretel-navigator.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -233,11 +233,11 @@ result = client.complete(
233
233
```
234
234
235
235
236
-
### Apply content safety
236
+
### Apply Guardrails and controls
237
237
238
-
The Azure AI model inference API supports [Azure AIcontent safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AIcontent safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering (preview) system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
238
+
The Azure AI model inference API supports [Azure AIContent Safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AIContent Safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
239
239
240
-
The following example shows how to handle events when the model detects harmful content in the input prompt andcontent safetyis enabled.
240
+
The following example shows how to handle events when the model detects harmful content in the input prompt andthe filteris enabled.
241
241
242
242
243
243
```python
@@ -477,11 +477,11 @@ The following example request shows other parameters that you can specify in the
477
477
}
478
478
```
479
479
480
-
### Apply content safety
480
+
### Apply Guardrails & controls
481
481
482
-
The Azure AI model inference API supports [Azure AIcontent safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AIcontent safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering (preview) system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
482
+
The Azure AI model inference API supports [Azure AIContent Safety](https://aka.ms/azureaicontentsafety). When you use deployments with Azure AIContent Safety turned on, inputs and outputs pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions.
483
483
484
-
The following example shows how to handle events when the model detects harmful content in the input promptand content safety is enabled.
484
+
The following example shows how to handle events when the model detects harmful content in the input prompt.
Copy file name to clipboardExpand all lines: articles/ai-foundry/how-to/deploy-models-serverless.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -332,7 +332,7 @@ In this section, you create an endpoint with the name **meta-llama3-8b-qwerty**.
332
332
333
333
:::image type="content" source="../media/deploy-monitor/serverless/deployment-name.png" alt-text="A screenshot showing how to specify the name of the deployment you want to create." lightbox="../media/deploy-monitor/serverless/deployment-name.png":::
334
334
> [!TIP]
335
-
> The **Content filter (preview)** option is enabled by default. Leave the default setting for the service to detect harmful content such as hate, self-harm, sexual, and violent content. For more information about content filtering (preview), see [Content filtering in Azure AI Foundry portal](../concepts/content-filtering.md).
335
+
> The **Content filter** option is enabled by default. Leave the default setting for the service to detect harmful content such as hate, self-harm, sexual, and violent content. For more information about content filtering, see [Content filtering in Azure AI Foundry portal](../concepts/content-filtering.md).
336
336
337
337
1. Select **Deploy**. Wait until the deployment is ready and you're redirected to the Deployments page.
0 commit comments