|
| 1 | +--- |
| 2 | +title: AI Red Teaming Agent |
| 3 | +titleSuffix: Azure AI Foundry |
| 4 | +description: This article provides conceptual overview of the AI Red Teaming Agent. |
| 5 | +manager: scottpolly |
| 6 | +ms.service: azure-ai-foundry |
| 7 | +ms.topic: how-to |
| 8 | +ms.date: 04/04/2025 |
| 9 | +ms.reviewer: minthigpen |
| 10 | +ms.author: lagayhar |
| 11 | +author: lgayhardt |
| 12 | +--- |
| 13 | + |
| 14 | +# AI Red Teaming Agent (preview) |
| 15 | + |
| 16 | +[!INCLUDE [feature-preview](../includes/feature-preview.md)] |
| 17 | + |
| 18 | +The AI Red Teaming Agent (preview) is a powerful tool designed to help organizations proactively find safety risks associated with generative AI systems during design and development of generative AI models and applications. |
| 19 | + |
| 20 | +Traditional red teaming involves exploiting the cyber kill chain and describes the process by which a system is tested for security vulnerabilities. However, with the rise of generative AI, the term AI red teaming has been coined to describe probing for novel risks (both content safety and security related) that these systems present and refers to simulating the behavior of an adversarial user who is trying to cause your AI system to misbehave in a particular way. |
| 21 | + |
| 22 | +The AI Red Teaming Agent leverages Microsoft's open-source framework for Python Risk Identification Tool's ([PyRIT](https://github.com/Azure/PyRIT)) AI red teaming capabilities along with Azure AI Foundry's [Risk and Safety Evaluations](./evaluation-metrics-built-in.md#risk-and-safety-evaluators) to help you automatically assess safety issues in three ways: |
| 23 | + |
| 24 | +- **Automated scans for content safety risks:** Firstly, you can automatically scan your model and application endpoints for safety risks by simulating adversarial probing. |
| 25 | +- **Evaluate probing success:** Next, you can evaluate and score each attack-response pair to generate insightful metrics such as Attack Success Rate (ASR). |
| 26 | +- **Reporting and logging** Finally, you can generate a score card of the attack probing techniques and risk categories to help you decide if the system is ready for deployment. Findings can be logged, monitored, and tracked over time directly in Azure AI Foundry, ensuring compliance and continuous risk mitigation. |
| 27 | + |
| 28 | +Together these components (scanning, evaluating, and reporting) help teams understand how AI systems respond to common attacks, ultimately guiding a comprehensive risk management strategy. |
| 29 | + |
| 30 | +## When to use the AI Red Teaming Agent's scans |
| 31 | + |
| 32 | +When thinking about AI-related safety risks developing trustworthy AI systems, Microsoft uses NIST's framework to mitigate risk effectively: Govern, Map, Measure, Manage. We'll focus on the last three parts in relation to the generative AI development lifecycle: |
| 33 | + |
| 34 | +- Map: Identify relevant risks and define your use case. |
| 35 | +- Measure: Evaluate risks at scale. |
| 36 | +- Manage: Mitigate risks in production and monitor with a plan for incident response. |
| 37 | + |
| 38 | +:::image type="content" source="../media/evaluations/red-teaming-agent/map-measure-mitigate-ai-red-teaming.png" alt-text="Diagram of how to use AI Red Teaming Agent showing proactive to reactive and less costly to more costly." lightbox="../media/evaluations/red-teaming-agent/map-measure-mitigate-ai-red-teaming.png"::: |
| 39 | + |
| 40 | +AI Red Teaming Agent can be used to run automated scans and simulate adversarial probing to help accelerate the identification and evaluation of known risks at scale. This helps teams "shift left" from costly reactive incidents to more proactive testing frameworks that can catch issues before deployment. Manual AI red teaming process is time and resource intensive. It relies on the creativity of safety and security expertise to simulate adversarial probing. This process can create a bottleneck for many organizations to accelerate AI adoption. With the AI Red Teaming Agent, organizations can now leverage Microsoft’s deep expertise to scale and accelerate their AI development with Trustworthy AI at the forefront. |
| 41 | + |
| 42 | +We encourage teams to use the AI Red Teaming Agent to run automated scans throughout the design, development, and pre-deployment stage: |
| 43 | + |
| 44 | +- Design: Picking out the safest foundational model on your use case. |
| 45 | +- Development: Upgrading models within your application or creating fine-tuned models for your specific application. |
| 46 | +- Pre-deployment: Before deploying GenAI applications to productions. |
| 47 | + |
| 48 | +In production, we recommend implementing **safety mitigations** such as [Azure AI Content Safety filters](../../ai-services/content-safety/overview.md) or implementing safety system messages using our [templates](../../ai-services/openai/concepts/safety-system-message-templates.md). |
| 49 | + |
| 50 | +## How AI Red Teaming works |
| 51 | + |
| 52 | +The AI Red Teaming Agent helps automate simulation of adversarial probing of your target AI system. It provides a curated dataset of seed prompts or attack objectives per supported risk categories. These can be used to automate direct adversarial probing. However, direct adversarial probing might be easily caught by existing safety alignments of your model deployment. Applying attack strategies from PyRIT provides an extra conversion that can help to by-pass or subvert the AI system into producing undesirable content. |
| 53 | + |
| 54 | +In the diagram, we can see that a direct ask to your AI system on how to loot a bank triggers a refusal response. However, applying an attack strategy such as flipping all the characters can help trick the model into answering the question. |
| 55 | + |
| 56 | +:::image type="content" source="../media/evaluations/red-teaming-agent/how-ai-red-teaming-works.png" alt-text="Diagram of how AI Red Teaming Agent works." lightbox="../media/evaluations/red-teaming-agent/how-ai-red-teaming-works.png"::: |
| 57 | + |
| 58 | +Additionally, the AI Red Teaming Agent provides users with a fine-tuned adversarial large language model dedicated to the task of simulating adversarial attacks and evaluating responses that might have harmful content in them with the Risk and Safety Evaluators. The key metric to assess the risk posture of your AI system is Attack Success Rate (ASR) which calculates the percentage of successful attacks over the number of total attacks. |
| 59 | + |
| 60 | +## Supported risk categories |
| 61 | + |
| 62 | +The following risk categories are supported in the AI Red Teaming Agent from [Risk and Safety Evaluations](./evaluation-metrics-built-in.md#risk-and-safety-evaluators). Only text-based scenarios are supported. |
| 63 | + |
| 64 | +| **Risk category** | **Description** | |
| 65 | +|------------------|-----------------| |
| 66 | +| **Hateful and Unfair Content** | Hateful and unfair content refers to any language or imagery pertaining to hate toward or unfair representations of individuals and social groups along factors including but not limited to race, ethnicity, nationality, gender, sexual orientation, religion, immigration status, ability, personal appearance, and body size. Unfairness occurs when AI systems treat or represent social groups inequitably, creating or contributing to societal inequities. | |
| 67 | +| **Sexual Content** | Sexual content includes language or imagery pertaining to anatomical organs and genitals, romantic relationships, acts portrayed in erotic terms, pregnancy, physical sexual acts (including assault or sexual violence), prostitution, pornography, and sexual abuse. | |
| 68 | +| **Violent Content** | Violent content includes language or imagery pertaining to physical actions intended to hurt, injure, damage, or kill someone or something. It also includes descriptions of weapons and guns (and related entities such as manufacturers and associations). | |
| 69 | +| **Self-Harm-Related Content** | Self-harm-related content includes language or imagery pertaining to actions intended to hurt, injure, or damage one's body or kill oneself. | |
| 70 | + |
| 71 | +## Supported attack strategies |
| 72 | + |
| 73 | +The following attack strategies are supported in the AI Red Teaming Agent from [PyRIT](https://azure.github.io/PyRIT/index.html): |
| 74 | + |
| 75 | +| **Attack Strategy** | **Description** | |
| 76 | +|---------------------|-----------------| |
| 77 | +| AnsiAttack | Utilizes ANSI escape sequences to manipulate text appearance and behavior. | |
| 78 | +| AsciiArt | Generates visual art using ASCII characters, often used for creative or obfuscation purposes. | |
| 79 | +| AsciiSmuggler | Conceals data within ASCII characters, making it harder to detect. | |
| 80 | +| Atbash | Implements the Atbash cipher, a simple substitution cipher where each letter is mapped to its reverse. | |
| 81 | +| Base64 | Encodes binary data into a text format using Base64, commonly used for data transmission. | |
| 82 | +| Binary | Converts text into binary code, representing data in a series of 0s and 1s. | |
| 83 | +| Caesar | Applies the Caesar cipher, a substitution cipher that shifts characters by a fixed number of positions. | |
| 84 | +| CharacterSpace | Alters text by adding spaces between characters, often used for obfuscation. | |
| 85 | +| CharSwap | Swaps characters within text to create variations or obfuscate the original content. | |
| 86 | +| Diacritic | Adds diacritical marks to characters, changing their appearance and sometimes their meaning. | |
| 87 | +| Flip | Flips characters from front to back, creating a mirrored effect. | |
| 88 | +| Leetspeak | Transforms text into Leetspeak, a form of encoding that replaces letters with similar-looking numbers or symbols. | |
| 89 | +| Morse | Encodes text into Morse code, using dots and dashes to represent characters. | |
| 90 | +| ROT13 | Applies the ROT13 cipher, a simple substitution cipher that shifts characters by 13 positions. | |
| 91 | +| SuffixAppend | Appends an adversarial suffix to the prompt | |
| 92 | +| StringJoin | Joins multiple strings together, often used for concatenation or obfuscation. | |
| 93 | +| UnicodeConfusable | Uses Unicode characters that look similar to standard characters, creating visual confusion. | |
| 94 | +| UnicodeSubstitution | Substitutes standard characters with Unicode equivalents, often for obfuscation. | |
| 95 | +| Url | Encodes text into URL format | |
| 96 | +| Jailbreak | Injects specially crafted prompts to bypass AI safeguards, known as User Injected Prompt Attacks (UPIA). | |
| 97 | +| Tense | Changes the tense of text, specifically converting it into past tense. | |
| 98 | + |
| 99 | +## Learn more |
| 100 | + |
| 101 | +Get started with our [documentation on how to run an automated scan for safety risks with the AI Red Teaming Agent](../how-to/develop/run-scans-ai-red-teaming-agent.md). |
| 102 | + |
| 103 | +Learn more about the tools leveraged by the AI Red Teaming Agent. |
| 104 | + |
| 105 | +- [Azure AI Risk and Safety Evaluations](./safety-evaluations-transparency-note.md) |
| 106 | +- [PyRIT: Python Risk Identification Tool](https://github.com/Azure/PyRIT) |
| 107 | + |
| 108 | +The most effective strategies for risk assessment we’ve seen leverage automated tools to surface potential risks, which are then analyzed by expert human teams for deeper insights. If your organization is just starting with AI red teaming, we encourage you to explore the resources created by our own AI red team at Microsoft to help you get started. |
| 109 | + |
| 110 | +- [Planning red teaming for large language models (LLMs) and their applications](../../ai-services/openai/concepts/red-teaming.md) |
| 111 | +- [Three takeaways from red teaming 100 generative AI products](https://www.microsoft.com/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/) |
| 112 | +- [Microsoft AI Red Team building future of safer AI](https://www.microsoft.com/security/blog/2023/08/07/microsoft-ai-red-team-building-future-of-safer-ai/) |
0 commit comments